System with image capturing means

Abstract

A system, in particular an auto-commissioning system comprising an image capturing means, such as a camera, arranged to capture an image of an area, and a processing means configured to perform the steps of controlling the image capturing means so as to capture a sequence of images over time, processing said captured sequence of images over time to detect at least one scene element in said area, selecting at least one application from a list of applications, such as a traffic monitoring application or a parking monitoring application, based on a result of the processing and configuring the at least one application based on the captured sequence.

Description

[0001] SYSTEM WITH IMAGE CAPTURING MEANS

[0002] FIELD OF INVENTION

[0003] The field of the invention relates to the field of systems with an image capturing means, in particular auto-commissioning systems with an image capturing means, and more in particular luminaire systems, such as outdoor luminaire systems, or traffic light systems, including an image capturing means. Such systems are preferably part of a network, and in particular a luminaire network such as an outdoor luminaire network.

[0004] BACKGROUND

[0005] Luminaires, especially outdoor luminaires, have long been present worldwide to provide the lighting of cities, industrial plants, private or public parking lots, etc. More and more, these luminaires become smarter, in that they are able to operate in networks which are capable of automatically exchanging information between the luminaires and / or with a remote entity. These smart luminaires are already present today in densely populated areas, such as streets, roads, paths, parking lots, parks, campuses, train stations, airports, harbors, beaches, etc., of cities around the world, from small towns to metropoles.

[0006] Luminaires may be mounted on long poles or placed at a significant height such as to illuminate wider portions of an area such as a road or a parking space. More and more, luminaires are provided with additional sensing devices, such as cameras, that take advantage of the height of the luminaire and the pole they are often mounted on. This allows for example to capture images from an elevated point of view and to access information otherwise not easily obtained at ground level. Information captured by such cameras and other image capturing means is valuable in a wide range of applications such as traffic monitoring, parking monitoring, crowd monitoring, etc. , and can be used to help road users, city planners, etc., and / or to increase the security of road users.

[0007] In order to be able to use the information captured by the camera, an application may be installed. For example, a camera aimed at the surface of a road will have a better view at the vehicles driving on the road surface than at parked vehicles on the parking spaces or at pedestrians on the sidewalk. Therefore, a road traffic monitoring application may be installed. In other examples, the camera may be aimed at a curbside and a people traffic monitoring may be installed. Furthermore, the application itself also often needs to be configured. SUMMARY

[0008] An object of embodiments of the invention is to provide a system, preferably an auto-commissioning system, comprising an image capturing means, allowing for an automatization of the use of the images captured by the image capturing means, preferably a kind of auto-commissioning of the system. The image capturing means may be part of a luminaire or traffic light and this autocommissioning may be part of the commissioning procedure of the entire luminaire or traffic light, but this is not required.

[0009] According to a first aspect of the invention, there is provided a system, preferably an autocommissioning system, with an image capturing means, such as a camera, arranged to capture an image of an area. The system further comprises a processing means configured to perform the steps of: controlling the image capturing means so as to capture a sequence of images over time; processing the captured sequence of images over time to detect at least one scene element in the area; selecting at least one application from a list of applications, such as a traffic monitoring application or a parking monitoring application, based on a result of the processing; configuring the at least one application based on the captured sequence.

[0010] In other words, the system takes a sequence of images of an area and processes these images to detect scene elements that are present in the images and hence in the area. For example, detected scene elements may include static or moving vehicles, pedestrians, parking spaces, occupied or not, a road crossing, a curb side, etc. Depending on the elements that are detected, one or more applications may be selected. The one or more applications may be pre-installed in the system or may be installed by the processing means. For example, if several moving vehicles are detected, a vehicle traffic monitoring, e.g., vehicle counting application or a speed monitoring application or a ghost driving application may be installed, and / or if several static vehicles are detected, a parking monitoring application may be installed. In another example, people may be detected on a curbside, and a people traffic monitoring, e.g., a people counting application, or a use type detection application, e.g., to detect the use of the curbside (such use as terrace, use as bicycle storage, etc.) may be selected. In a later step, the selected application is configured, based on information contained in the images of the captured sequence. For example, for a vehicle counting application, the position of one or more virtual counting barriers in the captured area may be set so as to improve the accuracy of the counting. For example, for a parking monitoring application, the boundaries of a parking space may be set. For example, for a use type application, the boundaries of the curb side may be set. In this way, the commissioning of the system is made automatic. The commissioning of the system is therefore faster and less expensive. Moreover, since the choice of installed application is determined based on information contained in the captured images, the application better corresponds to the characteristics of the image capturing means, e.g. the position, the field of view, the resolution of the camera and the environment around the camera. In addition, since the configuration of the at least one application is also based on the captured sequence, the applications can be performed in a more accurate manner. The system therefore provides improved information.

[0011] It is noted that the steps of controlling, processing, selecting and configuring may be done during commissioning but may also be done later, e.g. at regular intervals or upon request of an operator. In that manner, any changes to the environment may be taken into account.

[0012] In a preferred embodiment, the step of processing may comprise determining a position and / or a type of the at least one scene element. Based on the position of the at least one scene element and / or the type of the at least one scene element, it may be determined whether these elements allow an application to be performed in a good manner, and thus whether or not to select that application based on this determination. For example, it may be determined that there are parking spaces in the area, and that these spaces are found close to the borders of the image. This may be a sign that the accuracy of an application that would monitor these parking spaces will not be very good. On the contrary, it may be determined that a road surface is present in a middle portion of the image. This may be a sign that the information captured by the capturing means may be used for vehicle traffic monitoring. In such case, an application for vehicle traffic monitoring may be selected.

[0013] In a preferred embodiment, the step of processing may comprise determining whether a scene element of the at least one scene element is static or is moving, and optionally determining a moving direction of the scene element if it is moving. For example, it may be determined whether a car that has been detected is static, e.g., is parked in a parking space, or is moving, e.g., is moving on the surface of the road in a moving direction. In this way, different types of applications may be selected based on the result of the determination.

[0014] In a preferred embodiment, the processing may be performed using semantic segmentation. Semantic segmentation methods are deep learning methods, e.g., using convolutional neural networks, that associate a class with every pixel of an image and are used to recognize a collection of pixels that form distinct classes. For example, semantic segmentation may be used to identify vehicles, pedestrians, traffic signs, a pavement or curbside, scene elements on the pavement or curbside, and other road features.

[0015] In a preferred embodiment, the processing may comprise, for a predetermined type of scene element, determining a density of scene elements of the predetermined type in the area. For example, it may be determined that in a sequence of images, the type of scene element “moving vehicle” tends to be detected more often in the middle of the scene of the image than anywhere else on the scene of the image, i.e., the density of detected “moving vehicles” is larger in the middle of the image than at the borders. This may be due for example, to the fact that in the middle of the image, the vehicle appears in its entirety and / or undistorted and / or in a good resolution, while at the borders of the image, only parts of the vehicle appear, e.g. only the back of the vehicle, and / or the vehicle appears in a distorted form and / or the image has a bad resolution at the borders. In this example, the density may be used for the configuration of a vehicle counting application, e.g., virtual barriers used for the counting of moving vehicles may be placed at the location where the density of detected vehicles is the largest. The density of scene elements may therefore be valuable information for the commissioning of the device that may improve the quality of the information produced based on the images captured by the image capturing means.

[0016] In a preferred embodiment, the at least one scene element may comprise at least one of: an object such as a vehicle or a person or a tree, a parking lot, a parking space, a road, a bus stop, a pavement, a green area, a school, a hospital, a pedestrian crossing, a loading bay, a curbside. In this way, a wide variety of applications may be considered for the commissioning.

[0017] In a preferred embodiment, the at least one application comprises at least one of: a traffic monitoring application configured to monitor traffic (this may be vehicle and / or people traffic) in said area, such as a counting application, a ghost driver detection application, a traffic violation detection application; counting applications may be configured to monitor traffic of a road and / or a crowd, in said area. a parking monitoring application; parking applications may be configured to monitor a parking lot in said area. a use type detection application configured to detect one or more use types of said area, such as different types of curbside uses, for example use as a terrace, a bicycle parking use, a (un)loading area use, etc.-, a user type detection application configured to detect one or more user types in said area, such as a child, an animal, a wheelchair user, elderly people, etc. In a preferred embodiment, the step of processing further comprises determining, based on the captured sequence of images, of a number and / or a position and / or a direction of at least one of: lanes of a road and / or parking spots in the area. The step of configuring of the at least one application may then comprise setting the determined number and / or position and / or direction of at least one of: lanes of a road and / or parking spots. In this way, characteristics of the road and / or the parking spots in the area may be used, e.g., in the selecting and / or in the configuring of the application, such that information produced by the capturing means and by the application may be improved. For example, the direction of a lane may be relevant for a ghost driving application.

[0018] In a preferred embodiment, the step of processing further comprise the determining, based on the captured sequence of images, of one or more counting locations for counting an object, such as a vehicle or a person, in the area. The step of configuring may then comprise setting the determined one or more counting locations.

[0019] Counting locations may correspond to the location at which virtual barriers may be placed, in order to count moving objects, like moving vehicles on a road. As seen above, these locations may for example be determined by the density of detected objects.

[0020] Optionally, the processing means is further configured to install the at least one application, after having selected the at least one application, wherein the installing of the at least one application may comprise the downloading of the at least one application from a remote server and / or a cloud server.

[0021] It may be a waste of resources to pre-install or install applications that are not commonly used prior to the commissioning. However, such applications may still be valuable in particular situations. For example, a city planner of a city may decide to install luminaires or traffic lights with image capturing means in a given neighborhood, in which there is a stadium and a highway. It may be decided that most of these luminaires will be placed along the highway, and a few around the stadium. Having a crowd monitoring application for data produced by the capturing means of the luminaires along the highway may not be particularly relevant, as it is not expected to find pedestrians, let alone crowds, on a highway. However, for the few luminaires that are placed close to the stadium, a crowd monitoring application may well be relevant. By having the possibility of downloading applications, the system is more adaptable to the environment in which the devices are commissioned. This also allows for possible updates of the application.

[0022] In an embodiment, the system may comprise a non-transitory computer-readable storage medium on which one or more of the applications of the list of applications is pre-stored and / or preinstalled. For example, applications that are commonly used, such as vehicle counting applications or parking monitoring applications may be pre-stored and / or preinstalled on the storage medium, such that at the time of the commissioning, these applications are already available for the installing. In this way, some predetermined applications, e.g. common ones, may be readily available at a time previous to the installing, which may accelerate the commissioning of the device. This also allows to avoid having to fully install applications on the device prior to the commissioning without knowing if they will be useful, given the particular situation of the device.

[0023] The processing means may be part of any one of the following: the image capturing means, an edge device such as a luminaire or traffic light which also includes the image capturing means, a fog device in communication with the image capturing means, a central server in communication with the image capturing means.

[0024] In a preferred embodiment, the processing means may be configured to send to an operator a message indicating the selected application and / or a configuration parameter to be used in the selected application. Preferably, the processing means may also be configured to receive a confirmation message of the operator that the selected application is to be installed or used and / or that the configuration parameter is to be used to configure the selected application.

[0025] It may happen that the processing may deem some applications relevant given the environment of the system and / or the characteristics of the image capturing means, but that they are in fact not relevant for the use that is expected from them. For example, for a luminaire placed along a road with parking spaces, it may be found through the processing that traffic monitoring and parking monitoring are both relevant applications. However, the image capturing means may have been placed with the purpose of monitoring the parking spaces only. It may therefore be interesting to avoid the installing of the traffic monitoring application. Similarly, the processing may determine that the placement of three virtual barriers for the counting of vehicles may provide the best accuracy for a counting application. However, due to limitations in bandwidth and / or in processing power, such amount of virtual barriers and their position, comprised as configuration parameters, may not be ideal. By having an operator confirm the results of the processing, it may be avoided that unnecessary processing power and bandwidth is used for installing and / or configuring of a selected application that is not desired. The configuration parameters may also comprise information about a type of model to be used by the application, e.g. a type of neural network algorithm and / or architecture. According to a second aspect of the invention, there is provided a system, preferably an autocommissioning system comprising an image capturing means, such as a camera, arranged to capture an image of an area, and a processing means; The processing means is configured to perform the steps of: controlling the image capturing means so as to capture a sequence of images over time; processing said captured sequence of images over time to determine a position and type of scene elements in said area at consecutive moments in time; configuring at least one application which uses the image capturing means, based on the determined position and type of scene elements in said area at consecutive moments in time.

[0026] In other words, the system takes a sequence of images of an area and processes these images to determine the position and type of scene elements that are present in the images and hence in the area. For example, detected scene elements may include any type of objects such as static or moving vehicles, pedestrians, parking spaces, occupied or not, a road crossing, etc. On basis of the position and type of the scene elements that are detected, the applications that use the image capturing means may be configured by the processing means. For example, a traffic monitoring application, e.g. a vehicle counting application or a speed monitoring application, may use the images of the image capturing means. In that case, the detection of vehicles and the determining of their position is used to configure the application, which may improve the accuracy of the counting. For example, the configuration of vehicle counting applications may include the configuration of one or more virtual counting barriers in the captured area.

[0027] In this way, the configuring step of the commissioning of the system, such as a luminaire system, is made automatic. The configuring of the application and therefore the commissioning of the system as a whole, are therefore faster and less expensive. Moreover, since the configuring of the application is determined based on the position and type of the scene elements, i.e., on information contained in the captured sequence, the applications are more accurate in the information they can output. The application configuration therefore better corresponds to the characteristics of the image capturing means, e.g. the position, the field of view, the resolution of the image capturing means and the environment around the image capturing means. The system therefore provides improved information.

[0028] In a preferred embodiment, the configuring comprises the selecting of at least one first sub-area in the area where a same type of object is detected multiple times at different locations within the at least one first sub-area. The at least one first sub-area may be selected as at least one suitable subarea for use in the at least one application. In other words, within the area that is captured in a sequence of images, first sub-areas corresponding to regions of these captured images may be selected. Such one or more first sub-areas may include an area where a same type of object, e.g. a moving vehicle, has been detected multiple times at different locations within the first sub-area. For example, an image capturing means may capture a sequence of images that show one or more vehicles moving along a road. On these images, the processing may determine that the scene elements that are present in the images are moving vehicles and may determine their position. The location at which vehicles are detected will be dependent on the timing at which the images are captured. The position of the detected vehicles will therefore differ from one image to the other, indicating that the vehicles are moving. Similarly, in the case of a sequence of images comprising consecutive images, the position of a same vehicle advancing on the road will differ from one image to the other, since the vehicle is moving, and therefore its detected location will be different from one image to the other. On a given sequence of images, the detected locations may however be within a delimited region in the images, corresponding to a sub-area of the area, in which scene elements of the type “moving vehicles” may be detected. In the example discussed here, the first sub-area may correspond to a region of the image corresponding to the road surface, as it is not expected to detect vehicles on a sidewalk or in trees of the area. The at least one first sub-area comprising these detected locations may therefore be selected as at least one suitable sub-area for use in the at least one application, such as a traffic monitoring application.

[0029] In the example above, there may be sub-regions within the delimited region comprising the detected locations, where moving vehicles are more often detected. For example, there may exist sub-regions of this region in which the accuracy of the detection is better, e.g. in the middle of the scene of the image, since in the middle of the image, the vehicle tends to appear in its entirety and / or undistorted and / or in a good resolution, while at the borders of the image, only parts of the vehicle appear, e.g. only the back of the vehicle, and / or the vehicle appears in a distorted form and / or the image has a bad resolution in the borders. The at least one first sub-area may correspond to this sub-region and may therefore be selected as at least one suitable sub-area for use in the at least one application, such as a vehicle counting application or a traffic monitoring application.

[0030] In a preferred embodiment, the at least one first sub-area may comprise a plurality of first sub-areas. The configuring may then further comprise selecting one of the plurality of first sub-areas as the suitable sub-area for use in the at least one application.

[0031] Roads are often divided by a median strip and / or may be divided in multiple lanes. A plurality of first sub-areas may therefore be selected to correspond to the two sides of the road and / or to each of the lanes of the road. Also, depending on the lane in which the vehicle is driving, the resolution and / or the distortion of the image and / or the accuracy of the detection may vary. It may therefore be useful to determine a plurality of first sub-areas, in order to improve the reliability of the application and to be better adapted to the situation.

[0032] In a preferred embodiment, the selecting of the at least one suitable sub-area may be based on any one or more of: a density of the determined positions of scene elements, preferably per type of scene element, an accuracy of the determined positions and / or types of scene elements. In another preferred embodiment, the selecting of the at least one suitable sub-area is based on the number of detections of an object of a same type.

[0033] It may be determined that moving vehicles tend to be detected more often in the middle of the scene of the image than anywhere else on the scene of the image, i.e. the density of detected “moving vehicles” is larger in the middle of the image than at the borders. This may be due for example, to the fact that in the middle of the image, the vehicle appears in its entirety and / or undistorted and / or in a good resolution, while on the borders of the image, only parts of the vehicle appear, e.g. only the back of the vehicle, and / or the vehicle appears in a distorted form and / or the image has a bad resolution at the borders. The density may be used as a basis for the selecting of the at least one suitable sub-area, e.g. the placement of virtual barriers used for the counting of moving vehicles may be placed at the location where the density of detected vehicles is the largest. Similarly, the number of detections of an object of a same type may also be used for this selection. Additionally or alternatively, the selecting may be based on the accuracy of the determined positions and / or types, which may be based for example on the detected speed of the object, e.g. a vehicle, on the environment luminosity. In a situation in which vehicles have been detected on a sidewalk with low accuracy, e.g. false detections, these detections may not be taken into account in the selecting of the suitable sub-area.

[0034] In a preferred embodiment, a first application of the at least one application may be configured to count a number of moving scene elements of a predetermined type in the at least one first sub-area. For example, the area may comprise a road and the first application of the at least one application is configured to count a number of vehicles using the road in the at least one first sub-area. For example, the at least one first sub-area comprises one or more lanes of the road. The first application may for example be a moving vehicle counting application that specifically counts trucks, buses, cars, or bicycles using the bike lane.

[0035] In a preferred embodiment, the configuring may comprise the selecting in the area of at least one second sub-area where a same type of object is detected multiple times at a same location within the at least one second sub-area. The at least one second sub-area may be selected as a suitable sub-area for use in the at least one application.

[0036] In other words, within the area that is captured as a sequence of images, one or more second subareas corresponding to regions of these captured images may be selected. Such one or more second sub-area may be selected as an area where a same type of object, e.g. a static vehicle, has been detected multiple times at the same location within the second sub-area. For example, an image capturing means may capture a sequence of images that show one or more vehicles that are static on the side of road, e.g. in parking spaces along a road. The processing may determine that the scene elements that are present on the images are static vehicles and may determine their position. The location at which vehicles are detected should not be dependent on the timing at which the images are captured. The position of the detected static vehicles will therefore remain substantially the same from one image to the other, since the parking spaces do not move. On a given sequence of images, the detected locations may delimit a region on the images, corresponding to a second sub-area of the area, in which scene elements of the type “static vehicles” may be detected. In the example discussed here, this second sub-area may correspond to a region of the image corresponding to parking spaces of the road, as it is not expected to detect static vehicles on a sidewalk or in the lanes of the road. The at least one second sub-area comprising these detected locations may therefore be selected as at least one suitable sub-area for use in the at least one application, such as a parking monitoring application.

[0037] In a preferred embodiment, a second application of the at least one application may be configured to count a number of static objects of a predetermined type in the at least one second sub-area and / or to determine a position and / or presence of a static object in the at least one second sub-area. For example, the area may comprise one or more parking spaces and the second application of the at least one application may be configured to count a number of parked vehicles and / or empty parking spaces in the at least one second sub-area and / or to determine a position and / or presence of a parked vehicle in the at least one second sub-area.

[0038] The second application may for example be a parking monitoring application that monitors whether parking spaces are empty or occupied. This may help users to know where to park when parking spaces are busy. This may also be useful for monitoring whether the vehicles are parking illegally. For example, parking spaces may sometimes serve as delivery zones according to a given schedule. Determining that some vehicles, e.g. that are not delivery trucks, are parked at delivery times may trigger a warning for a user of the car, or to the authorities, e.g. such that the vehicle is removed or is fined. The step of determining whether a parking space is occupied or not may also be useful in the case of electric vehicle charging spaces. Some electric vehicles tend to remain parked in such charging spaces even though their vehicle is fully charged, thereby preventing other electric vehicles that may need to charge their vehicle to park there.

[0039] Parking monitoring applications may also be useful to adapt the pricing based on the type of user. For example, some drivers may have a reduced mobility, may belong to a diplomatic corps or to the police, may be domiciliated or work in the same zone as the parking space, etc. , which may influence the pricing of the parking there. The type of user may be known to the application through a user identification, e.g. entered on a cellphone through an app, e.g. a car license plate number, a category of vehicle such as compact vehicles, electric vehicles, SUV, pickups, trucks, etc. The type of user may influence the pricing, e.g. based on the ecological class of the vehicle. The pricing may be static or dynamic in time, to better adapt to the demand. For example, the pricing may also be based on the time of parking, e.g. cheaper during the night and more expensive during busy hours, and / or the pricing may be based on how busy the parking spaces are, e.g. the pricing may be cheaper when most of the parking spaces are unoccupied, and may increase as the parking spaces are more and more occupied. Also, a different pricing may be used depending on whether there is an event in the area, e.g. if there is a football match at the stadium nearby.

[0040] Below some further preferred features are discussed which may be used in embodiments of the first or second aspect.

[0041] The processing means may be integrated at least partly with the image capturing means or may be separate. For example, the processing means may be part of an edge device such as a luminaire, typically an outdoor luminaire, or a traffic light, which also includes the image capturing means. Also, the processing means may be part of a fog device which is configured to communicate with the image capturing means, e.g. through a mesh network. Further, the processing means may be part of a central server in communication with the image capturing means through a cellular network, optionally in combination with a mesh network. The processing means may also be distributed over multiple devices, e.g. over an edge device and a fog device.

[0042] Preferably, the at least one application uses one or more models, and the step of configuring comprises configuring the one or more models. The one or more models may comprise any one or more of the following: an object detection model, an object trajectory determining model, a model configure to determine if an object crosses a line, an object type determining model configured to determine a type of object. For example, a ghost driver detection application may use a vehicle detection model, a vehicle trajectory determining model and a model configured to determine if a vehicle crosses a line in a certain direction, wherein the latter will use the former for its determination.

[0043] The one or more models may be pre-trained models and / or the processing means may be configured to train the one or more models based on images captured by the image capturing means, wherein optionally the one or more models are one or more neural networks models. For example, first the processing means may operate the application in a training mode for training the one or more models, whereupon the application may be used in an operational mode for the intended application. In this way, by finalizing the training of the model in the environment in which the application will be used, the model may be better adjusted for the intended use of the application in this particular environment.

[0044] In an exemplary embodiment, the system is a luminaire system comprising a light source and a drive means for driving the light source. The system may further comprise a control means, optionally integrated with the processing means, configured to control the drive means in function of an output of the at least one application. For example, when the application is a counting application, the light intensity and / or the light distribution and / or the color of the light emitted by the light source may be changed depending on the number of vehicles or persons counted within a predetermined period of time. In another example, the application is a traffic violation application and the light source may be controlled to blink or otherwise attract attention when a traffic violation is being detected.

[0045] In an exemplary embodiment, the system further comprises at least one sensor configured to sense an environmental characteristic of the area, such as a sound sensor and / or a radar sensor, wherein the at least one application is configured to use an output of the at least one sensor as an input of the application. The step of configuring may then comprise configuring the application based on an output of the at least one sensor.

[0046] It is noted that the image capturing means and / or the at least one sensor may be added to different types of edge devices, such as any one or more of the following: a luminaire, a bin, a sensor device, a street furniture, a charging station, a payment terminal, a parking terminal, a street sign, a traffic light, a telecommunication cabinet, a traffic surveillance terminal, a safety surveillance terminal, a water management terminal, a weather station, an energy metering terminal, an access lid in a pavement. The image capturing means and the at least one sensor may be implemented in similar edge devices (e.g. a network of outdoor luminaires). Alternatively, the image capturing means and the at least one sensor may be implemented in different edge devices. Existing structures ubiquitously present in cities may be used for hosting networks, limiting in this way the aesthetic impact of installing such networks. Structures having already access to the power grid are particularly interesting, while luminaires having typically a suitable height to capture all kinds of valuable data from image capturing means and other sensors are further particularly suited as edge devices.

[0047] An edge device may be associated with the image capturing means and one or more sensors, e.g. a sensor configured for obtaining a set of environmental data related to an event in the vicinity of the edge device, preferably at least two sensors configured for obtaining at least two sets of environmental data related to an event in the vicinity of the edge device. Optionally, the edge device comprises a classification module configured to determine classification data of the event based on the captured images and the at least one set, preferably at least two sets, of environmental data, and the classification data may be used to select the at least one application.

[0048] According to an exemplary embodiment, the at least one sensor is selected from: a sound sensor, a radar such as a Doppler effect radar, a LIDAR, a humidity sensor, an air quality sensor, a temperature sensor, a motion sensor, an antenna such as a Bluetooth antenna for a Bluetooth sensor, an RF sensor, a metering device (e.g. a metering device for measuring the power consumption of a component of an edge device, more in particular a metering device for measuring the power consumption of a driver of a luminaire), a vibration sensor, a malfunctioning sensor e.g., a sensor for detecting the malfunctioning of a component of an edge device such as a current leakage detector for measuring current leaks in a driver of a luminaire), a measurement device for measuring a maintenance related parameter of a component of the edge device, an alarm device (e.g. a push button which a user can push in the event of an alarming situation). In this way, environmental data about an event in the vicinity of an edge device may be detected, e.g. characteristics (presence, absence, state, number, direction, speed, wearing mask or not) of objects like vehicles, street furniture, animals, persons, sub-parts of the edge device, or properties related to the environment (like weather (rain, fog, sun, wind), pollution, visibility, earth quake) or security related events (explosion, incident, gun shot, user alarm) in the vicinity of the edge device, maintenance related data or malfunctioning data of a component of the edge device. For example, in case of a crisis, the model may be configured to set one or more configuration parameters allowing to detect additional features like whether people are wearing mask or not, e.g. during a limited period of time.

[0049] According to an exemplary embodiment, the image capturing means and / or the at least one sensor may be mounted in a housing of an edge device, e.g. a luminaire, in an orientable manner. An example of a suitable mounting structure is disclosed in WO 2019 / 243331 Al, in WO 2022 / 122755 Al and WO 2022 / 189601 Al in the name of the applicant which are included herein by reference. Such mounting structure may be used for arranging e.g. an image capturing means in the housing of an edge device. Other suitable mounting structures for mounting the image capturing means and / or the at least one sensor are described in WO 2019 / 053259 Al, WO 2019 / 092273 Al, WO 2020 / 053342 Al, WO 2021 / 094612 Al, all of which are in the name of the applicant and included herein by reference. Although those patent specifications relate in particular to luminaire edge devices in which one or more sensors are provided, the skilled person understands that one or more sensors may be mounted in a similar way in another type of edge device.

[0050] According to an exemplary embodiment, the one or more edge devices are arranged in a mesh network. In this way edge devices can communicate directly with each other for a fast control. Reliability is also further improved due to redundant communication links. In particular, a fog device may also be part of the mesh network. Preferably, the edge communication means and optionally the fog communication means are configured to communicate through an IEEE 802.15.4 protocol.

[0051] Preferably, the edge devices are configured to transmit edge processed data to their associated fog device using a wireless personal area network (WPAN), preferably as defined in the IEEE 802.15.4 standard. According to another exemplary embodiment, the edge devices are configured to transmit edge processed data to their associated fog device through an LPWAN network, e.g., a LoRaWAN network or a SigFox network.

[0052] For example, the communication between the edge devices and its associated fog device may be based on a short-range protocol such as IEEE 802.15.4 (e.g. Zigbee) or on a long range communication protocol such as LoRa wireless data communication technology. The network may be managed by the fog device or by a separate segment controller. In such a solution the edge device may only be capable of communicating through the short-range communication protocol. However, it is also possible that some edge devices are capable of communicating both through a short-range and a long-range protocol. Also, the fog device may be integrated with one of the edge devices, e.g. one of the luminaires of a group of luminaires could function as the fog device for a group of edge devices comprising the group of luminaires and possibly also other edge devices.

[0053] Preferably, the fog devices are capable of communicating through a network comprising a cellular network with a central control system.

[0054] According to an exemplary embodiment, the processing means is configured to adjust measurement settings of a first sensor of the at least one sensor based on second processed data and / or to adjust measurement settings of a second sensor based on first processed data. In particular variables of operation of a sensor like the frequency of a sound sensor, the light exposure or view field of an image capturing means may be dynamically adjusted to benefit from the self-learning capabilities of the system. A measurement setting typically refers to a setting which affects the sensed environmental data obtained by the image capturing means or other sensor whilst the set of rules used to process the sensed environmental data will influence the processing of the sensed data which is done by the at least one application. Thus, according to exemplary embodiments not only the processing but also the measurements itself may be improved.

[0055] According to an exemplary embodiment, there is provided a communication means configured to receive data about an event in the one or more edge devices or in their vicinity from a central database and / or from the fog device. The processing means may then be configured to use said data for processing the data sensed by the image capturing means and / or the at least one sensor and / or for training the one or more models of the at least one selected application. In this way more depth in the classification / detection of scene elements may be attained using additional information from a database. For instance, special vehicles like ambulances may be further identified using data from a dedicated database. Preferably the data in a database includes any one or more of the following: weather related information for the area, traffic information for the area, geo-coordinates of the edge device, news and internet information, public transportation information, events (fairs, concerts, etc.) schedules, timing information (weekday, weekend, public holiday), public safety information, sanitary reports, security reports, road condition reports and cellphone data of cellphones in the vicinity of the subset. In this way, context data may be incorporated in the classification and / or training process for further accuracy.

[0056] According to an exemplary embodiment, the processing means is configured to adjust measurement settings of the image capturing means and / or the at least one sensor based on the received data. In this way, data from a database may be put to use for calibration of the image capturing means and / or the at least one sensor, increasing further the accuracy of the multi-sensor system.

[0057] A further object of embodiments of the invention is to provide an improved system for identifying and / or classifying objects having a higher accuracy compared to existing systems.

[0058] To that end, according to a further aspect, there is provided a system for identifying an object in an environment comprising: a first sensor configured to capture first data of the environment, said first data being represented in a first data space, a second sensor configured to capture second data of the environment, said second data being represented in a second data space and preferably including data related to a movement of an object or a sound made by an object, and a processing means. The processing means is configured to process said first sensed data using a first model to derive a first region of interest in said first data space related to an object in the environment and to process said second sensed data using a second model to derive a second region of interest in said second data space related to an object in the environment, to transform the first region of interest in said first data space into a first region of interest in said second data space or to transform said second region of interest in said second data space into a second region of interest in said first data space, to fuse the transformed first region of interest or the transformed second region of interest with the second region of interest in the second data space or with the first region of interest in the first data space, respectively, and to classify an object in the environment based on the fused data and / or to train the first and / or second model using the fused data.

[0059] By using sensed data represented in different data spaces, such as sensed data from a video camera and sensed data from a radar, and performing a transformation of the first sensed data (at least the region of interest of the first sensed data) from the first data space to the second data space or a transformation of the second sensed data (at least the region of interest of the second sensed data) from the second data space to the first data space, the data can be fused in an improved manner. Also, by using a first and second model to determine the regions of interest, upstream of the data fusion, the processing power can be used in an optimal manner. Further, the fused data may be used to train the first and second model, so that the quality of the determining of the regions of interest can be improved. The classifying of an object may be done either based on the fused data or based on the first or second sensed data using the trained first or second model, respectively.

[0060] Preferably, the processing means is configured to use a homographic transformation matrix defining a link between the first data space and the second data space, to perform the transforming. More in particular, the processing means may be configured to determine the homographic transformation matrix based on the first and second sensed data, preferably at predetermined moments in time or upon request.

[0061] Preferably, the first sensor is an imaging sensor and the first data is two-dimensional image data represented in a two-dimensional data space, for example a data space with cartesian coordinates.

[0062] Preferably, the second sensor is a radar, and the second data space is a three-dimensional data space comprising the polar coordinates: azimuth, elevation, velocity. However, the second sensor may also be an acoustic sensor or a seismic sensor. Also multiple different second sensors may be included. For example, second sensed data from a radar may be augmented with sensed data from a sound sensor and / or a seismic sensor. Preferably, when the first sensor is a video camera and the second sensor is a radar, the processing means is configured to transform the second region of interest represented in the second radar data space into a second region of interest represented in said first video data space, and to fuse the transformed second region of interest with the first region of interest in the first video data space. The reason for rather transforming the radar data into the video data space than the other way around, is that the visual representation in the video data space is more easy to read.

[0063] Preferably, the first data comprises multiple sets of frames taken at consecutive moments in time, and the fusing uses a velocity of an object derived from the second sensed data as well as a velocity of an object derived from the multiple frames.

[0064] Preferably, the first and second model are configured to detect and / or classify an object in the environment based on the sensed first and second data, respectively.

[0065] Preferably, the first region of interest is a bounding box comprising an object.

[0066] Preferably, the field-of-view of the first sensor overlaps with the field-of-view of the second sensor.

[0067] Preferably, the fusing is obtained using any one or more of the following types of processing techniques: averaging, voting, ensemble learning, stacking.

[0068] Preferably, the system comprises a network of luminaires, wherein the first and second sensor are preferably arranged on or in the same luminaire of the network. The processing means may be a centralized or distributed processing means located at a luminaire of the network and / or at a location remote from the luminaires of the network.

[0069] BRIEF DESCRIPTION OF THE FIGURES

[0070] This and other aspects of the present invention will now be described in more detail, with reference to the appended drawings showing exemplary embodiments of the invention. Like numbers refer to like features throughout the drawings.

[0071] Figures la and lb illustrate two embodiments of a system for auto-commissioning of a luminaire with an image capturing means present in a street with a plurality of luminaires. In Fig. la, the road is a one-way road with three lanes while in Fig. lb, the road is a two-way road with parking spaces. Figures 1c and Id illustrate two embodiments of a processing means according to an exemplary embodiment of the system.

[0072] Figure 2 illustrates an example of detection of scene elements as a result of the processing of a captured image along a road by a processing means according to an exemplary embodiment of the system.

[0073] Figures 3a, 3b, 3c and 3d illustrate an example of the configuring of a traffic monitoring application and / or a vehicle counting application on the basis of a sequence of images captured along a road, according to an exemplary embodiment of the system.

[0074] Figures 4a, 4b and 4c illustrate examples of the configuring of a parking application on the basis of a sequence of images captured along a road comprising parking spaces, according to an exemplary embodiment of the system.

[0075] Figures 5-8 illustrate an exemplary embodiment using an image capturing means and two sensors.

[0076] Figures 9-14 illustrate another exemplary embodiment of an architecture of a multimodal fusion process.

[0077] DESCRIPTION OF THE EMBODIMENTS

[0078] Figs, la and lb illustrate two embodiments of a system for auto-commissioning of a luminaire with an image capturing means present in a street with a plurality of luminaires.

[0079] In Fig. la, a one-way road with a road surface 200 comprising three lanes 200a-c, on which moving vehicles 300a-c are driving, is illustrated. The road is provided at its sides with luminaires 100, e.g. a network of luminaires. One of the luminaires 100 is provided with an image capturing means 101, e.g. a camera, that is pointed at an area Al comprising the road, and is arranged to capture an image of that area Al. The system comprises a processing means 102, configured to control the image capturing means 101 so as to capture a sequence of images over time.

[0080] The processing means may be a single unit, as illustrated in Fig. 1c, or may be distributed over multiple units, e.g. partly in a local unit of an edge device such as a luminaire and partly in a remote unit, such as a fog device or a remote central server, as illustrated in Fig. Id. Fig. lb illustrates a similar embodiment to the one of Fig. la, but where the road is a two-way road with a road surface 200 comprising one lane in each direction 200a and 200b instead. Instead of a third lane like for Fig. la, the road is provided with parking spaces 201 at one side thereof, where static vehicles 301 are parked. The image capturing means 101 of Fig. lb differs from the one of Fig. la at least in that the area A2 that it captures is focused on the parking spaces 201 rather than on the road surface 200.

[0081] The processing means is configured to process the captured sequence of images. For a system in accordance with the first aspect, the processing is configured to detect at least one scene element in the area Al, A2. For example, the image capturing means 101 of Fig. la may detect the road surface 200, the different lanes 200a-c, the moving vehicles 300a and 300c on the road surface 200 and / or some of the luminaires 100 that are in the field of view of the image capturing means 101. The image capturing means 101 of Fig. lb may detect the parking spaces 201, some of the parked vehicles 301 and some of the luminaires 100 that are in the area A2.

[0082] It should be noted that although only vehicles 300a-c, 301, a road surface 200, parking spaces 201 and luminaires 100 have been considered as scene elements, the skilled person will understand that the at least one scene element may comprise any object, such as vehicles, persons, trees, parking lots, bus stops, pavements, green areas, schools, hospitals, pedestrian crossings, loading bays, etc.

[0083] The step of processing may comprise determining whether a scene element of the at least one scene element is static or is moving. For example, it may be determined whether a vehicle 300a-c, 301 that has been detected is static, e.g. is parked in a parking space such as vehicles 301, or is moving, e.g. is moving on the surface of the road 200 such as 300a-c.

[0084] The processing may be performed using semantic segmentation. Semantic segmentation methods are deep learning methods, e.g. using convolutional neural networks, that associate a class with every pixel of an image and are used to recognize a collection of pixels that form distinct classes. For example, semantic segmentation may be used to identify vehicles, lanes, luminaires, and other road features. Such form of processing is described in more details in relation with Fig. 2 and Fig. 4a and 4c.

[0085] Based on the results of the processing, at least one application is selected from a list of applications and is installed. The edge device, e.g. the luminaire, may comprise a non-transitory computer- readable storage medium, on which at least one of the applications of the list of applications is pre- stored and / or preinstalled. Additionally or alternatively, the installing of the at least one application may comprise the downloading of the at least one application from a remote server and / or a cloud server, e.g. a remote server and / or a cloud server that is connected with the network of luminaires.

[0086] In the example of Fig. la, since all captured sequences of images of the area Al may most probably contain images of moving vehicles 300a-c, applications such as traffic monitoring and vehicle counting may be selected and installed since they may be the most relevant applications in that particular situation. In the example of Fig. lb, since most of the detected elements may most probably be the static vehicles that are parked 301, applications such as a parking application or parking monitoring may therefore be preferred and the traffic monitoring applications may not be relevant. In order to confirm the selection, the processing means may be configured to send to an operator a message indicating the selected application and to receive a confirmation message of the operator that the selected application is to be installed.

[0087] The processing may comprise determining a position and / or a type of the at least one scene element. Based on the position of the elements and / or the type of the at least one scene element, it may be determined whether these elements should be taken into account in the selecting of the application. For example, it may be determined that there is a lane 200b in the area A2, and that this lane 200b is found close to the borders of the image. This may be a sign that the accuracy of an application that would monitor the traffic of this lane 200b may not be very good. On the contrary, it may be determined that the parking spaces 201 are present in a middle portion of the image of the area A2. This may be a sign that the information captured by the capturing means 101 may be used with relatively high accuracy for parking monitoring, but not for traffic monitoring.

[0088] After the installation, the at least one application is configured based on the captured sequence. For example, the processing may comprise the determining of a number and / or a position of at least one of lanes of a road, e.g. in Fig. la, and / or parking spots in the area, e.g. in Fig. lb. The step of configuring of the at least one application may then comprise the setting of the determined number and / or position of at least one of lanes of a road and / or parking spots. In another example, e.g. in which the application is a counting application, the configuration of the application may comprise the determining of one or more counting locations for counting an object, e.g. a moving vehicle or a static vehicle, in the area, e.g. Al or A2. The step of configuring comprises setting said determined one or more counting locations. The configuration of the at least one application may be based on the density of detected scene elements of the predetermined type in the area. Some examples of the configuring of the application will be described in more detail in Figs. 3-4.

[0089] Although Fig. la-b have been discussed in relation to a system of the first aspect, the skilled person will understand that Figs, la-b are also relevant to the second aspect.

[0090] For example, for a system in accordance to the second aspect, the processing is configured to determine a position and type of objects in the area Al, A2 at consecutive moments in time. For example, the image capturing means 101 of Fig. la may detect the moving vehicles 300a and 300c on the road surface 200 and / or some of the luminaires 100 that are in the field of view of the image capturing means 101 and may determine their positions and types. Similarly, the image capturing means 101 of Fig. lb may detect some of the parked vehicles 301 and some of the luminaires 100 that are in the area A2 and may determine their positions and types.

[0091] Based on the results of the processing, e.g. on the determined position and type of objects in said area at consecutive moments in time, at least one application which uses the image capturing means 101 is configured. The at least one application may use information produced based on the images captured by the image capturing means 101. For example, the at least one application may be related to traffic monitoring and / or a parking. Based on the positions and the type of objects that have been detected, the at least one application may be configured, such that for example, the information produced by the application is improved.

[0092] In Fig. la, for example, the area Al comprises the road surface 200 and the luminaires 100, but may comprise a drainage portion and sidewalks of the road (not represented here). All information captured from the area Al may not be relevant to, e.g. a traffic monitoring application. The configuring may comprise the selecting of at least one first sub-area, e.g. the road surface 200, in the captured area where a same type of object, e.g. a moving vehicle, is detected multiple times at different locations within the at least one first sub-area. In this example, the at least one first subarea is chosen as the road surface 200 because most if not all vehicles detected should be found on the road surface 200. The road surface 200 comprising multiple lanes 200a-c, may therefore be selected as an at least one suitable sub-area for use in the at least one application, e.g. the traffic monitoring. In another example, the first application of the at least one application may be configured to count a number of moving objects of a predetermined type in a at least one sub-area, e.g. a moving vehicle counting application that specifically counts trucks that do not use the slowest lane 200c. The at least one first sub-area described above may comprise a plurality of first sub-areas. For example, the road of Fig. la is divided in multiple lanes. A plurality of first sub-areas may therefore be selected to correspond to each of the lanes 200a-c of the road (or, although not illustrated here, to the two directions of the road). The configuring may then further comprise selecting one of the plurality of first sub-areas as the suitable sub-area for use in the at least one application.

[0093] In another example, the selecting of the at least one suitable sub-area may be based on any one or more of a density of the determined positions and types of objects and an accuracy of the determined positions and types of objects. Two examples of this selecting based on such density are discussed in relation to Figs. 3a-d and 4a-b.

[0094] In Fig. lb, the area A2 comprises a portion of the road surface 200, the luminaires 100 and the parking spaces, but may comprise a drainage portion and sidewalks of the road (not represented here). All information captured from the area A2 may not be relevant to, e.g. a parking monitoring application. The configuring may comprise the selecting in the area of at least one second sub-area where a same type of object is detected multiple times at a same location within the at least one second sub-area. For example, most if not all detected static vehicles should be found in the parking spaces 201 and the at least one sub-area of the area A2, e.g. the parking spaces 201, may therefore be selected as an at least one suitable sub-area for use in the at least one application, e.g. the parking monitoring.

[0095] Fig. 2 illustrates an example of a detection of scene elements as a result of the processing of a captured image along a road by a processing means according to an exemplary embodiment of the system.

[0096] In this example, related to the system of the first aspect, the processing may for example be performed using semantic segmentation. Semantic segmentation methods are computer vision processes, e.g. deep learning methods using e.g. convolutional neural networks, that divide a digital image into various categories or segments and associates a category or class with every pixel of an image. These processes are used to recognize a collection of pixels that form distinct classes. Semantic segmentation is therefore distinct from image classification, which allots labels to an entire image, and from object detection, which identifies and locates objects within an image by drawing bounding boxes around them.

[0097] In image segmentation, an image is given as input to a neural network, which outputs a mask as the output, where each element of the mask determines which class or instance the pixels of the element belong to. Although deep neural networks architectures for image segmentation may differ in implementation, most follow similar basis structure and comprise an encoder (a set of layers that extract features of an image through a sequence of progressively narrower and deeper filters), a decoder (a set of layers that progressively grows the output of the encoder into a segmentation mask resembling the pixel resolution of the input image) and skip connections (long range connections in the neural network that allow the model to draw on features at varying spatial scales to improve model accuracy).

[0098] As illustrated in Fig. 2, semantic segmentation may be used to identify vehicles 300a-b and 301, a road surface 200 or the lanes 200a-b comprised therein, parking spaces 201, a pedestrian sidewalk 202, pedestrians 302, buildings 203 and other road features such as luminaires, trees, bus stops, pavements, green areas, school / hospitals, pedestrian crossings, loading bays, curbs, the sky, etc. (not all represented here). Segmentation of an image may therefore enable to segregate the foreground from the background and focus on the relevant scene elements.

[0099] Images obtained from a luminaire placed along a road such as the ones of Fig. 2 may comprise several different scene elements, such as a road surface 200, parking spaces 201, a pedestrian sidewalk 202, etc. It may be found through the processing that traffic monitoring, parking monitoring and crowd monitoring are all relevant applications to be installed. However, the image capturing means may have been placed with the purpose of monitoring the parking spaces 201 only. It may therefore be interesting to avoid the installing of the traffic monitoring and crowd monitoring applications. In order to confirm the selection, the processing means may be configured to send to an operator a message indicating the selected application and to receive a confirmation message of the operator that the selected application is to be installed.

[0100] Although Fig. 2 has been discussed in relation to a system of the first aspect, the skilled person will understand that it is also relevant to the second aspect, mutatis mutandis. Indeed, the determining of the positions and type of objects in the area at consecutive moments in time may also be achieved in a similar way.

[0101] Figs. 3a, 3b, 3c and 3d illustrate an example of the configuring of a traffic monitoring application and / or a vehicle counting application on the basis of a sequence of images captured along a road, according to an exemplary embodiment of the system.

[0102] The road comprises a road surface 200 with three lanes 200a-c, the first two 200a-b being used by cars going in one direction, and the third 200c by cars going in the opposite direction. The image is one image of the sequence of captured images captured by the capturing means associated with a luminaire, which are both not illustrated in the figure. The processing means according to the system of the first aspect is configured to process said captured sequence of images over time to detect at least one scene element, e.g. a vehicle 300a-c, in the area, e.g. the road surface 200. For example, the processing may comprise determining a position and / or a type of the at least one scene element.

[0103] In Fig. 3b are illustrated the results of this processing for a sequence of images, comprising a plurality of images such as the ones illustrated in Fig. 3a. Solid diamonds 400 represent the position of the detected scene elements of the type “vehicle”. It should be noted that while the human eye may determine with ease that the five objects that are illustrated in Fig. 3a are indeed vehicles, it may not be so for the processing means. Indeed, due to environmental conditions, e.g. weather, due to the resolution and / or the shutter speed of the camera, due to the speed of the vehicles, the position of the vehicle on the road, etc., the processing means may fail to detect the objects and / or may falsely classify the detected objects (the processing may detect a truck, while in fact, the detected object was a vehicle). The hollow hexagons 401 illustrate the positions of all detections that are not of the type “vehicle”.

[0104] In the example of a vehicle counting application, the processing means may, in the configuration step, determine three different counting locations, also known as virtual counting barriers, for counting detected objects, such as vehicles. For example, the processing means may determine a first barrier 501 under the first luminaire, a second barrier 502 under the second luminaire, and a third barrier 503 under the third luminaire. As it should be noted, the number of detections is higher in the middle of the image (in the portion under the second luminaire). This may be due for example, to the fact that in the middle of the image, the vehicle appears in its entirety and / or undistorted and / or in a good resolution, while on the borders of the image, only parts of the vehicle appear, e.g. only the back of the vehicle, and / or the vehicle appears in a distorted form and / or the image has a bad resolution in the borders. This may explain why in the first lane 200a, in the top side of the image, very few detections are made of the type “vehicle”, and proportionally a lot of detections of other types occur. Such observations may also be made for the third lane 200c, on the bottom side of the image.

[0105] The processing means may determine a density, rather than a number, of the position of the detected type of scene element “vehicle”. As it should be noted from Fig. 3c, and for the reasons discussed above, the density is higher in the middle of the image (in the portion under the second luminaire). It should also be noted that the determining of a number and / or a position of lanes of a road may also be based on such densities. Indeed, as visible from Fig. 3c, very few detections of vehicles occur between the lanes 200a-c, since vehicles tend to not change lane often. Such densities may therefore be used for the configuring of the installed application, e.g. for the positioning of counting locations (or virtual counting barriers) used for the counting of moving vehicles where the density of detected vehicles is the largest and / or for the setting of the determined number and / or position of the lanes of the road 200a-c.

[0106] An example of configuring based on densities such as the ones of Fig. 3c is shown in Fig. 3d, where three counting locations, e.g. virtual counting barriers 501-503, are placed at three different locations where the densities are the largest. It should be noted that contrary to Fig. 3b, the barriers do not span the whole width of the road and do not encompass all lanes 200a-c.

[0107] It should be noted that in the example of Fig. 3d, while the processing may determine that the placement of three virtual barriers for the counting of vehicles may provide the best accuracy for a counting application, limitations in bandwidth and / or in processing power may exist. Such configuration parameters may therefore not be ideal. The processing means may be configured to send to an operator a message indicating configuration parameter to be used in the selected application and to receive a confirmation message of the operator that the configuration parameter is to be used in the selected application.

[0108] Although Fig. 3a-d have been discussed in relation to a system of the first aspect, the skilled person will understand that it is also relevant to the second aspect, mutatis mutandis. Indeed, the determining of, e.g. the first sub-areas, may also be achieved in a similar way.

[0109] Figs. 4a, 4b and 4c illustrate examples of the configuring of a parking application on the basis of a sequence of images captured along a road comprising parking spaces, according to an exemplary embodiment of the system.

[0110] The image illustrated in Fig. 4a is an example of one of the images of the sequence of images of an area, that are captured by a capturing means associated with a luminaire and processed by the processing means. The area comprises a road comprising a road surface 200 with two lanes 200a-b and parking spaces 201, where cars 300a-d are parked. As visible from Fig. 4a, the processing means has detected four vehicles 300a-d. The processing may comprise determining a position and / or a type of the at least one scene element. Fig. 4a illustrates two types of processing. In a first type, the parked vehicles are detected using object detection, which provides with a bounding box 310 around the detected objects 300a-b. In a second type, the parked vehicles are detected through the use of image segmentation, which provides a mask 320 that determines the class of the detected objects 300c-d.

[0111] In Fig. 4b are illustrated the results of the processing through object detection, for a sequence of images, comprising a plurality of images. Each of the bounding boxes 310 represents the bounding box of a detected vehicle on one of these images. Since vehicles exist in different sizes and since users may park their vehicle in different ways, it is to be expected that the bounding boxes of parked vehicles will differ from one vehicle to the other. However, parked vehicles will most often park on definite parking spots such that the bounding boxes should significantly overlap. The processing may therefore determine densities 311 of the detections of the parked vehicles that should be well localized on the parking spots. This is illustrated in Fig. 4b, where the densities 311 of parked vehicles have been illustrated for two parking spots.

[0112] Similarly, Fig. 4c illustrates the results of the processing through image segmentation, for a sequence of images, comprising a plurality of images. Each of the masks 320 represents the mask of a detected vehicle on one of these images which may be used to determine densities 321 of detections of the parked vehicles.

[0113] In any case, in the example of a parking monitoring application, the processing means may, in the configuration step, determine a number and / or a position of parking spots in the area. For example, the processing means may determine the position of the delimitation 211 between two parking spots. The step of configuring of the at least one application may therefore comprise setting the determined number and / or position of the parking spots.

[0114] Similarly to the example of Fig. 3d, while the processing may determine configuration parameters, such as the number and / or position of parking spots, as the best for the parking monitoring application, limitations in bandwidth and / or in processing power may exist. Such configuration parameters may therefore not be ideal. The processing means may be configured to send to an operator a message indicating the configuration parameters to be used in the selected application. The processing means may then be configured to receive a confirmation message of the operator that the configuration parameters are to be used in the selected application or to receive a modified set of configuration parameters that may for example override the result of the processing and manually change the configuration parameters. Although Fig. 4a-c have been discussed in relation to a system of the first aspect, the skilled person will understand that it is also relevant to the second aspect, mutatis mutandis. Indeed, the determining of, e.g. the second sub-areas, may also be achieved in a similar way.

[0115] In an exemplary embodiment, the system further comprises at least one sensor configured to sense an environmental characteristic of the area, such as a sound sensor and / or a radar sensor, wherein the at least one application is configured to use an output of the at least one sensor as an input of the application. The step of configuring may then comprise configuring the application based on an output of the at least one sensor.

[0116] Multimodal data streams using an image capturing means and at least one sensor and deep learning architectures have several advantages over single -modality approaches for object detection, such as improved accuracy, robustness to noise and ambiguity, handling of complex scenarios and adaptability to diverse data. In a preferred embodiment, a feature-level fusion architecture for object detection based on YOLO features from images sensed by the image capturing means, spectral and rhythm features from sound sensed by a sound sensor and general descriptors from radar modalities sensed by a radar, may be used. After alignment, the sensed data may be combined with an attention mechanism into a single classification network.

[0117] The field of object detection is an integral part of various real-world applications, such as autonomous driving, surveillance, robotics or medical imaging. Accurately and efficiently detecting objects in a frame enables an easier interpretation of visual data which, in turn, facilitates an automated decision-making process for a wide variety of applications, such as object tracking, human-computer interaction or activity recognition.

[0118] Despite the progress brought forward by deep learning architectures in object detection tasks, there are still some challenges to overcome with regards to different real- world scenarios: deep learning models often struggle to handle occlusion or cluttered scenes, where objects of interest can be partially obscured or surrounded by other objects; a limited number of samples for training can lead to overfitting and poor generalization performance, since deep learning models typically require large amounts of labeled training data for effective learning; class imbalance can negatively impact the performance of these models, as they tend to be biased towards the majority classes; and achieving real-time object detection, especially in low-resource settings, can be extremely challenging since deep learning models with high accuracy may be computationally expensive. Consequently, multimodal approaches arose as a viable alternative to address these issues. A multimodal approach using an image capturing means and at least other one sensor, refers to the integration of different information sources (for example visual, textual, audio, radar, among others) to enhance the learning process of a machine / deep learning architecture: by combining the information from different modalities and leveraging their complementary nature, multimodal learning models capture richer representations of the data and allow for an improvement in performance. These modalities can be processed independently and then fused at different levels, namely, early fusion (input level), late fusion (decision level) or intermediate fusion (feature level), and bring several advantages over unimodal approaches, such as robustness to noise and ambiguity or handling data variability. On the other hand, multimodal approaches may lead to increased model complexity and computational requirements, alongside difficulties in the process of data alignment, modality selection and choice of fusion strategy.

[0119] Hence, in exemplary embodiments, an architecture for feature level fusion of image, sound and radar modalities in the context of an object detection application is proposed. A variety of multimodal fusion strategies may be used to combine visual, auditory, and other sensory modalities for object detection. For instance, depth information may be incorporated alongside RGB data for camerabased object detection, aiming to enhance the performance of existing single-shot detection networks by leveraging depth sensing, which can be easily obtained using depth cameras like Kinect or stereo setups. A network architecture that enables parameterization of the fusion point / layer for both RGB and depth information within the network layers, may be used. Preferably, RGB and depth information towards the mid to late layers of the network are fused.

[0120] Combining RGB with depth images may also be useful in the a-modal 3D detection problem, the task of detecting and localizing objects in a three-dimensional space, even when they are partially or completely occluded. A novel neural network called 3D-SSD may be used, which consists of two main components: hierarchical feature fusion and multi-layer prediction. The hierarchical feature fusion is where the appearance and geometric features from RGB and depth images are learned and fused at two stages using fusion blocks. The result of this fusion may then be used for object detection in the multi-layer prediction. The network may be evaluated on the SUN RGB-D and NYUv2 datasets. Multimodal machine learning may also be applied to sound detection. A method using a result-level fusion convolutional neural network (CNN) to detect drones and determine their presence in the surrounding environment, may be used. The features of sound signals, specifically Log-Mel spectrogram and Mel Frequency Cepstral Coefficient (MFCC) may be extracted and separately input into the networks. The results from the two networks are fused using evidence theory to obtain the final detection outcome. Preferably, the data fusion takes advantage of the complementarity and redundancy of the various sources of data to exploit both intra-modality and inter-modality relationships. Data fusion approaches can go from a simple concatenation of data to the implementation of a self-attention fusion network , and a YOLO-MS framework with a Feature Interaction and Self- Attention Fusion Network (FISAFN) may be used as the backbone. The FISAFN comprises two modules: the Feature Interaction Module (FIM) and the Self- Attention Feature Fusion Module (SAFFM). The FIM extracts correlations between two modalities (visible and infrared images) to reconstruct the information components of each modality and enhance information exchange. The SAFFM establishes long-range information dependence between the modalities, filtering redundant features and enhancing complementary features. Such YOLO-MS framework may achieve better information richness of the fused features

[0121] In a preferred embodiment, an architecture for feature level fusion of image, sound and radar Modalities is used, preferably based on a YOLOv7 model for images, MFCC features extracted from sound signals, and radar descriptors in the context of object detection, preferably fused at the input stage for a deep classification attention network.

[0122] Such architecture allows to build a localizable, interoperable, cyber-safe, resilient, distributed, autonomous and connected urban infrastructure to solve common smart city challenges. The public lighting infrastructure may be used as host and hub for smart cities: taking advantage of an infrastructure established for decades, individual sensors, cameras or small cells can be integrated into streetlights or light poles to leverage future data-centric smart city applications. Consequently, in this scenario where great amounts of data are collected from several different sensory devices, the design and development of an edge computing architecture for data fusion are an opportunity, since multimodal learning can not only reduce detection error, but also grant more reliability to the multiple distributed sources, while fixing problematic data and extracting higher level information.

[0123] In an exemplary embodiment a model is used to provide the integration of the sound and radar modalities to the pre-existing image capturing means network. A number of non-consecutive minutes of captured video samples (with sound) and the corresponding radar descriptors from sensors placed on a lighting pole may be used. Figure 5 shows an example of an image frame captured. First the processing means performs an annotation process. In the example, the data set is composed by seven different classes of objects: person, bicycle, car, motorcycle, siren, bus and truck; to evaluate the performance of our multimodal approach, in Figure 8 the results obtained by means of a confusion matrix are presented, as well as the precision, accuracy, and recall metrics, defined by: Precision = TP / (TP + FP) (1)

[0124] Recall = TP / (TP + FN) (2)

[0125] Accuracy = (TP + TN) / (TP + TN + FP + FN) (3) where TP, TN, FP, and FN correspond to true positives, true negatives, false positives, and false negatives, respectively.

[0126] Optionally YOLOv7 may be used due to the model’s capacity to extract features at different levels while maintaining a high-level understanding of the scene, and at the same time using an anchor box strategy that allows for the accurate localization of objects. The YOLOv7 model may be pre-trained. To extract features from the sound signals Mel Frequency Cepstral Coefficients (MFCC) may be used: the mel log-spectrogram analysis is a known approach in sound signal processing tasks; it brings several advantages such as perceptual alignment (mel scale approximates the nonlinear frequency response of the human auditory system), dynamic range compression (logarithmic compression reduces the dynamic range of the signal, allowing for the capture of both high and low- level acoustic features simultaneously) and frequency binning (aggregation of energy across adjacent frequency bands, allowing for identification of spectral patterns at different scales); as for the radar modality, descriptors, such as velocity, lifetime, azimuth and elevation, may be introduced as features and fed directly into an attention mechanism of the classification network.

[0127] Figure 6 illustrates an exemplary embodiment of the proposed architecture of the feature-level fusion process: for the image model, a pre-trained YOLOv7 model is loaded and evaluated on a subset of non-consecutive minutes samples, and when the training process is finished, features from the most relevant bounding boxes are extracted from the last layer before the detection unit - to extract the features from the middle layers of the YOLOv7 model, a PyTorch implementation may be used by incorporating the concept of a forward hook to extract features from a specific layer. The forward hook is a function that is executed during the forward pass of the model and is registered to a chosen layer. When the forward pass occurs, the hook captures the output of the specified layer and stores it as the “features” attribute of the model instance - by leveraging this mechanism, our custom model allows for the parametrization of the layer from which features are extracted, providing flexibility in capturing and utilizing specific features of interest for various downstream tasks. As for the corresponding sound samples, MFCC may be extracted and on both set of inputs (image and sound) Principal Component Analysis (PCA) may be applied - effectively reducing the feature space while preserving the essential information. The application of PCA results in a significant reduction in feature size, allowing for more efficient processing, a vital attribute on an architecture meant to be deployed in edge devices and performed in real time. Next, the image and sound-reduced features, alongside the radar descriptors, may be introduced to a deep attention network that will be able to select and identify the most important features with regards to the classification process, while reutilizing the original YOLOv7 bounding boxes to deal with the predictions of object localization. In summary, this architecture intends for the sound and radar modalities to possibly correct miss- classifications from the YOLOv7 image model, and by fusing all modalities at a feature level (while guaranteeing the proper time alignment of all data) an enhanced discriminative power (ability to capture complex relationships between modalities), learning of complementary information and a robustness to missing modalities, is obtained, all while reducing model complexity and increasing computational efficiency.

[0128] Experimental results evidence that this architecture for object detection represents a promising approach for addressing real-world challenges in smart city applications by combining image, sound and radar modalities. Figure 7 and Figure 8 show some preliminary results obtained with the proposed approach, without the usage of radar descriptors and with a two-layer neural network for classification, with evaluation metrics of: Accuracy = 0.83, Precision = 0.42, Recall = 0.40. On a first analysis, the application of PCA resulted in a significant reduction in feature size: for the YOEO features, the dimensionality was reduced from 3072000 to 750 components, resulting in a cumulative explained variance of 0.9593; similarly, the sound features were reduced from 38809 to 12 components, achieving a cumulative explained variance of 0.9996. This reduction in dimensionality not only helped to alleviate computational challenges but also aided in eliminating redundant information and focusing on the most informative features, ultimately enhancing the overall effectiveness of the multimodal architecture for object detection.

[0129] The results show that the approach is viable and the idea of enhancing the image model with sound and radar modalities is achievable: evaluation metrics show high accuracy alongside lower values for precision and recall, which indicates that the model is making correct overall predictions.

[0130] Multimodal learning offers several advantages over single-modality approaches: by combining information from different modalities, multimodal architectures demonstrate improved accuracy, robustness to noise and ambiguity, and adaptability to complex scenarios. These findings highlight the potential of multimodal learning to address the challenges faced by deep learning models, such as occlusion, limited training data, and class imbalance. In experimental results, the feature-level fusion of image and sound modalities lead to enhanced object detection performance compared to individual unimodal approaches: this suggests that the fusion of multiple modalities can effectively capture complementary information and improve the overall detection capabilities in real-world applications. Optionally YOLO’s features from different layers may be used to observe the impact of each of them on the final results. Also, an attention mechanism to highlight the important regions and features in the fused representation may be used: this may provide some level of interpretability by indicating which modalities contribute more significantly to the final classification.

[0131] Figures 9-14 illustrate another exemplary embodiment of an architecture of a multimodal fusion process. The developed multimodal system components are encapsulated as Docker containers whose interaction with sensors is presented in Figure 9. Figure 9 shows the overall multimodal system scheme, from data acquisition to the multimodal knowledge base. The usage of Docker containers provides an autonomous and light system with elements to run it, such as system tools, libraries, and configurations. Furthermore, containers guarantee that applications are consistently functioning despite of the environment around them.

[0132] As so, on the left side of the diagram are represented possible sources of sensors that collect data from the environment, for instance image sensors like surveillance video cameras and standalone cameras, temperature sensors such as room thermostats and air conditioners, radar sensors as traffic radars and satellites, among others. The resultant data is acquired by sensory components, represented in powder blue, whose function is to validate, standardize, normalize, and apply other preprocessing steps on the information. Afterwards, the processed data is sent to specific knowledge components, represented in grey, using for example a broker with the Message Queuing Telemetry Transport (MQTT) protocol, which is represented in white.

[0133] A message broker is used to sustain the communication between the system’s components. The proposed broker was the Eclipse Mosquitto since it is light and can be used on all devices, including low-power single board computers. MQTT protocol is a standard messaging protocol for loT designed to link remote devices with minimal code footprint and low network bandwidth, making it suitable for low-power sensors or mobile devices. In sum, the MQTT broker acts as a universal and straightforward interface for all connections.

[0134] The specific knowledge base components convert the processed data into knowledge to be fused. In this step may be included decision-making tasks, simple feature analysis to complex machine learning, Deep Learning (DL) methods, and others. The data flow is not necessarily unidirectional: these components can receive knowledge from other processing components, functioning under a continuous training condition in order to improve skills and learning to generate more robust conclusions. Then, after data exploration and preparation, the information is ingested by the knowledge fusion component. This component merges the data from all data sources, including an initial data comparison (e.g. based on timestamps) or successive stages of fusion between more similar modalities. The multimodal data fusion converts raw or processed data from multiple sources into a reasoned set of conclusions, creating comprehensive and complete information that may not be available from a single sensor. The fusion phase may rely on feature and / or decision-based fusion principles, namely DL approaches where higher weights are automatically assigned to more relevant modalities during training or where specific rules are followed to prioritize more relevant modalities over others.

[0135] Besides the decision of whether the data fusion should occur on a decision or feature level, the best stage of the data flow to merge information may be chosen. There are several methods for multimodal object detection. Given as example the illustrative scheme of the processing pipeline of the proposed architecture, represented in Figure 10, the fusion step is applied in several phases of the data flow. Video camera data may suffer preprocessing, and the resulting semantic knowledge is subject to modelling, producing predictions and tracking regarding vehicles. Non-image data, namely radar, acoustic and seismic data, may have their own pipeline of preprocessing and modelling, and then, are subjected to knowledge fusion, providing resultant vehicle predictions. Only then the predictions resultant from the image model are aligned with non-image resulting predictions, and consequently subjected to another fusion, generating aligned predictions. The different stages of fusion may be required due to the diversity of data type obtained from each sensor. Indeed, the data is typically represented in a data space which may have two or more dimensions. For example, for a camera the sensed data is typically represented in a two-dimensional data space, but for other sensors, such as a radar the data may be presented in a different data space, for example a data space which represents position data in combination with non-position data, such as a velocity.

[0136] Therefore, an early fusion merges the feature representation of different data modalities at an early stage at the preprocessing phase, typically before the learning or inference processes. It combines raw or preprocessed data from sources into a single feature representation, subsequently fed into the machine learning model. On the other hand, late fusion involves the combination of outputs or predictions of several unimodal models, after the detection being performed independently for each type of data. The merging’ results may be obtained using different types of processing techniques like averaging, voting, ensemble learning, stacking, and others, being then aggregated at the bounding box level. In contrast, on deep fusion are used features from different types of deep neural networks and merged at the feature level, creating a more robust model that can deal with a diversity of sensor’ s data and improve detection accuracy. The representations from different modalities are processed separately by dedicated subnetworks, and the outputs of these subnetworks are then fused in a deeper layer of the network, allowing the model to learn more complex relationships between modalities and capture high-level interactions between them. In Figure 10, the first fusion stage occurs after non-image data modelling, being so called a “deep fusion”, and the second fusion is considered a “late” fusion since it occurs after the inference process of different modalities.

[0137] For illustrative purposes the following case study was selected: it comprehends the identification of objects in smart cities using public lighting infrastructure as sensor hosts and hubs. These infrastructures provide video camera, radar, and audio data, that, combined, can be used to identify objects like a person, bicycle, car, motorbike, emergency vehicle, bus, and truck. Ultimately, the precision of combining video camera and radar signals constitutes a more reliable system to use in object identification. Therefore, this case study presents the process of gathering, preparing, and using the data combined from video camera and radar signals, based on the representation of the system disclosed in Figure 11. Figure 11 is an illustration of public lighting post data usage: (A) frame of the video camera perspective, (B) radar perspective projection and (C) video camera perspective projection.

[0138] In Figure 11 , it can be perceived two of the many signals that a public lighting post can collect or transmit. In this case, the video camera collects 2D data represented by sets of frames of a desired angle, like the one designated as (A) and whose perspective projection can be seen in (C). The radar collects radio-frequency electromagnetic signals, 3D data, that can result in perspective projections like the one designated as (B). The main issue before overlapping projections of both signals is to align the signals and the field of extraction. Even though the video camera is capturing frames of the same angle that the radar is exploring, the dimensionality and the range of the two signals are different, typically requiring a manual calibration.

[0139] Given that, multimodal data requires surpassing a few obstacles to be used MML algorithms. The data preparation is the harder and longer task that needs to be done before its ingestion and application of algorithms.

[0140] Therefore, the representation of the data pipeline of this multimodal system from the acquisition until the co-learning phase in object detection is shown in Figure 12. Figure 12 shows a system for identifying an object in an environment comprising a first sensor, here a video camera schematically shown in a “data acquisition” block, configured to capture first data of the environment, said first data being represented in a first data space, and a second sensor, here a radar schematically shown in the “data acquisition” block, configured to capture second data of the environment, said second data being represented in a second data space and preferably including data related to a movement of an object, such as a velocity. The system further comprises a processing means configured to process said first sensed data using a first model, indicated as “ModelVideo” to derive a first region of interest in said first data space related to an object in the environment and to process said second sensed data using a second model, indicated as “ModelRadar” to derive a second region of interest in said second data space related to an object in the environment. The processing means further comprise a data alignment and data translation module configured to transform the first region of interest in said first data space into a first region of interest in said second data space and / or to transform said second region of interest in said second data space into a second region of interest in said first data space. The processing means further comprises a data fusion module configured to fuse the transformed first region of interest or the transformed second region of interest with the second region of interest in the second data space or with the first region of interest in the first data space, respectively. Further, the processing means is preferably configured to train the first and / or second model using the fused data, see the “co-learning” module in Figure 12. Optionally, the processing means is configured to classify an object in the environment either based on the fused data. Alternatively, the trained first and / or second model may be used to classify an object in the environment. Preferably, the processing means is configured to use a homographic transformation matrix defining a link between the first data space and the second data space, to perform the transforming. The processing means may be configured to determine the homographic transformation matrix based on the first and second sensed data, preferably at predetermined moments in time or upon request. Preferably, the processing means is configured to transform said second region of interest represented in the second radar data space into a second region of interest represented in the first video data space, and to fuse the transformed second region of interest in the first video data space with the first region of interest in the first data space. Preferably, the first region of interest is a bounding box comprising an object.

[0141] Preferably, the system comprises a network of luminaires, wherein the first and second sensor, i.e. preferably the video camera and the radar, are arranged on or in a same luminaire of the network. The processing means may be a centralized or distributed processing means located at a luminaire of the network and / or at a location remote from the luminaires of the network.

[0142] Figure 12 is a representation of the data pipeline of this multimodal system from the acquisition until the co-learning phase in object detection. The first step is the representation of the data. Given that the system acquired signal from a video camera and a radar, the data is represented in two instances, being Dvtdeo20for video camera data and D Radar30for radar data. Video data is represented in two dimensions, x and y, for every frame of the video camera data, where x represents the horizontal position of the pixel and y the vertical position of the pixel. Radar data is represented in three dimensions, u, v and w, for every record of the radar data, where u represents horizontal position or azimuth, v represents the vertical position or elevation, and w represents the radial velocity. After the representation and preprocessing, each sensor has its own model to be trained.

[0143] The second step is alignment, which refers to identifying the relationships between elements of different modalities, specifically data that regards different inputs with different dimensions. Thus, and after a manual calibration of the signals, it is applied a homographic transformation matrix that allows to create links between both signals.

[0144] The third step is the transformation of data from one modality to another, having to deal with the heterogeneity of the data and the subjectiveness of the relationship between the modalities. In this step are created projections of the data and, if needed, subject the system to another calibration.

[0145] The fourth step is data fusion, which consists of merging data from different modalities to make a prediction. It draws a few difficulties since data from different data sources may have missing information, dissimilar types of noise, and respective variable predictive importance. It is in this step that there are attempts of combining data and resulting projections, as illustrated in Figure 13. Figure 13 is an illustration of the translation, fusion, and co-learning phases of the different perspectives: (A) presents a projection of a video camera’s frame; (B) presents a projection of a radar signal’s instance; (C) presents the alignment points transformed in a single resultant projection (D); (E) presents objects’ detection in the frame after fusion. On image (A) it is presented a projection of a frame of the video camera data whose alignment points are enhanced in image (C). Similarly, on image (B) it is presented a projection of an instance of radar signal whose alignment points obtained from a homographic transformation matrix are identified in image (D). The alignment points identified in images (C) and (D) are a result of the translation step.

[0146] Lastly, the fifth step regards co-learning. It explores the knowledge transfer between various modalities, their representations, and predictive models. Therefore, it tries to comprehend how the knowledge gained from learning a model can provide benefit to a model trained with a different modality, especially when one of the modalities has few resources to carry out the learning process. In this last step it is used the results prediction of the radar signal and the video camera signal to identify objects on the context. After the fusion phase, the co-learning phase makes it possible to detect objects as shown in image (E) of Figure 13. Recalling the illustrative scheme presented in Figure 10, the processing pipeline of multimodal data requires adjustments iteratively and the alignment and fusion steps are not always in the same stage of the data flow. Given that representation, the first fusion occurs after the modelling phase of nonimage data, inferences from all types of data are generated and only then is proceeded to a data alignment between image and non-image detections. The coordinates of the bounding boxes resulting from the image model belong to the 2D space while the coordinates from the non-image predictions are from the 3D space, plus being of different scale and dimensions. It is used a homographic transformation matrix to project the radar detections from the 3D space onto the 2D image space. Since a single homographic matrix may not be universally appropriated for accurately projecting radar coordinates onto image spaces across different camera and radar installations, there a regularly calibration may be used. Therefore, Nelder-Mead optimization method may be used as calibration algorithm to iteratively adjust the homographic coefficients, minimizing the Mean Squared Error between both detections, ensuring accurate alignment and consistency in integrated video camera-radar system.

[0147] Following occurs a data fusion and aligned predictions are obtained. As the radar data can easily change due to environment changes, the traditional machine learning methods typically struggle to adapt to these changes. The continuous learning method is applied, and the measured error is tracked. When there is a big threshold between detections, it is detected drifting, leading to an adaptation of the fusion algorithm, radar recalibration, creation of a new homographic matrix and restart of the process. The strategy to select the best-cases is to retain top candidates based on detection score and bounding box proximity, enhancing the efficiency and accuracy of the continuous leaning model. With the proposed decision-level fusion approach, it was developed a versatile and flexible system.

[0148] The last steps, data alignment, data fusion and inference process, can enter a cycle and being redone as many times as needed to achieve the desired outcomes, entering so the co-learning process. Figure 14 is a representation of the detections on both video camera and radar views: (A) presents the video camera view, (B) presents a projection of the radar view, and (C) presents the radar view projected in a satellite perspective. As seen in Figure 14, the image (A) is a view from the video camera with predictions and tracking registered. Similarly, in the image (B) it is represented a projection of the radar view with detections marked. The process to point out radar’s predictions results from defining four corners in the image, use a homographic matrix to correct the radar perspective and then align the video camera and radar detections, in the process describe before. Only afterwards it is possible to project a radar view in a satellite perspective, as seen in image (C). A nomographic transformation matrix, also known as homography or projective transformation matrix, is a mathematical concept used in CV and image processing. It describes the transformation between two different types of data, usually images taken from different viewpoints, or a combination of images and another type of data. In this case, it is used to combine the data from the video camera, a 2D vector characterizing a frame (image), and data from a radar, a 3D vector characterizing the positional information.

[0149] Therefore, a homography map points from one plane to another. The associated matrix is usually represented by a 3x3 matrix and can encode various types of transformations, including translation, rotation, scaling, and skewing. It can also represent more complex transformations like perspective distortions. The applicability of homography includes image stitching, like when panoramic image stitching to align and blend multiple images seamlessly; augmented reality, such as aligning virtual objects with real-world scenes in augmented reality applications; camera calibration to estimate camera parameters and correct for distortions; and calculation, like computing using algorithms (Direct Linear Transform (DLT), or Random Sample Consensus (RANSAC)) from corresponding points in two images.

[0150] A homography matrix has eight degrees of freedom, but it is usually normalized to have a determinant of 1 to ensure it represents a proper transformation. Additionally, the homography is only valid under the assumption of a planar scene or rigid scene motion between images.

[0151] In terms of algebraic representation, aligning data from video camera and radar involves several mathematical calculations and transformations. Time synchronization is one of the relations that needs to be taken into account, represented as equation (1) below that ensures the timestamps tvideo and tradar from the video camera and radar systems are synchronized.

[0152] Likewise, there is the need of a geospatial calibration using equations to map coordinates from the radar’s coordinate system to video camera’s coordinate system. If the radar signal is represented in polar coordinates (azimuth, elevation, speed), these need to be transformed into cartesian coordinates (x, y, z) to align with the video camera’s perspective.

[0153] Regarding data fusion, the combination of data from both sources requires appropriate fusion algorithms. For example, if speed is a relational feature, it can be used equation (2) to fuse radar speed data with video camera images, where wradar is the speed measured by radar, wvideo is the speed estimated from visual data, and f is a fusion function.

[0154] For object recognition and tracking are used equations related with computer vision algorithms, such as Kalman filter equations, and estimation equations of object trajectory, velocity, and acceleration based on consecutive frames of video camera data. Consequently, data validation requires equations to validate the accuracy of the aligned data like error metrics comparing fused data with true measured data, and confidence intervals for the estimated parameters based on the uncertainty on the measurements from both sources.

[0155] The homographic alignment matrix, H, represented in equation (3), is a 3x3 matrix in homogeneous coordinates. Given a set of corresponding points (u, vi ) in the source plane (e.g., radar plane) and (xi , yi ) in the destination plane (e.g., video camera plane), the goal is to find the matrix H such that:

[0156] Where (ui , vi , 1) are the homogeneous coordinates of a point in the source plane, and (xi , yi , 1) are the homogeneous coordinates of the corresponding point in the destination plane.

[0157] The calculation of the homographic alignment matrix involves solving a system of linear equations or using optimization techniques, such as least squares. The chosen method depends on the specific application and requirements. One common approach is using DLT or its refined versions like normalized DLT. Once the homographic alignment matrix H is obtained, it can be used to transform points between the two coordinate systems.

[0158] Considering an example set of corresponding points (ut, vi ) in the radar's coordinate system and (xi , yi ) in the video camera's coordinate system, represented in equations (4) and (5), respectively. To calculate the homographic matrix, a system of linear equations using the above correspondences is defined, as represented in equation (3). Expanding the equation, it results as follows, in equation (6):

[0159] Then, all equations can be stacked for all corresponding points and form a matrix to solve for the elements of the homographic matrix H, resulting in the system represented in (7).

[0160] The system can be solved using various techniques such as least squares, or direct methods like Singular Value Decomposition (SVD). Once the matrix H is obtained, it can be used to transform points from the radar's coordinate system to the camera's coordinate system.

[0161] Approaches using MML for object detection enhance the benefits of creating more comprehensive and robust algorithms with multimodal data rather with a unimodal system, whether it is in the context of autonomous driving, bird monitoring, or even material surface inspection. The synergy between video camera 2D data and radar 3D data enhance object recognition accuracy. The proposed multimodal video camera and radar system provides a general approach to MML problems on edge devices for object detection scenarios and illustrates the advantages of multimodal systems in handling complex and dynamic sensory environments. This system surpasses the complexity of the input data by adopting an iterative strategy that cannot be effectively addressed by a unimodal system. After a first fusion on the resulting data from unimodal models and a manual calibration of the signals as seen in Figure 10, the data alignment phase uses a transformation matrix allowing to create links between both signals. The homographic transformation matrix projects the radar detections from the 3D space onto the 2D image space, followed by a Nelder-Mead optimization to recalibrate the coefficients when needed. The resultant outcomes provide effective object detection and identification in a dynamic sensory environment, as illustrated in Figures 13 and 14. With the continuous learning method and tracking of the measured errors between detections, the process can be restarted and adjusted, resulting in a versatile and flexible system. Due to system’s adaptability and scalability, healthcare, smart cities, and emotion analysis are some of the areas where it can be integrated.

Claims

CLAIMS1. A system, in particular an auto-commissioning system comprising an image capturing means, such as a camera, arranged to capture an image of an area, and a processing means configured to perform the steps of: controlling the image capturing means so as to capture a sequence of images over time; processing said captured sequence of images over time to detect at least one scene element in said area; selecting at least one application from a list of applications, such as a traffic monitoring application or a parking monitoring application, based on a result of the processing; configuring the at least one application based on the captured sequence.

2. The system according to claim 1 , wherein the step of processing comprises determining a position and / or a type of the at least one scene element.

3. The system according to claim 1 or 2, wherein the step of processing comprises determining whether a scene element of the at least one scene element is static or is moving, and optionally determining a moving direction of the scene element if it is moving.

4. The system according to any of the previous claims, wherein the processing is performed using semantic segmentation.

5. The system according to any of the previous claims, wherein the processing comprises, for a predetermined type of scene element, determining a density of scene elements of said type in said area.

6. The system according to any of the previous claims, wherein the at least one scene element comprises at least one of: an object such as a vehicle or a person or a tree, a parking lot, a parking space, a road, a bus stop, a pavement, green area, school / hospital, pedestrian crossing, a loading bay, a curbside.

7. The system according to any of the previous claims, wherein the at least one application comprises at least one of: a traffic monitoring application configured to monitor traffic in said area, such as a counting application, a ghost driver detection application, a trafficviolation detection application; a parking monitoring application; a use type detection application configured to detect one or more use types of said area, such as a curbside use, e.g. use as a terrace, a bicycle parking use, a (un)loading area use, etc.; a user type detection application configured to detect one or more user types in said area, such as a child, animal, wheelchair user, elderly people, etc.

8. The system according to any of the previous claims, wherein the step of processing comprises determining a number and / or a position and / or a direction of at least one of: lanes of a road and / or parking spots in the area, based on the captured sequence of images, and wherein the step of configuring of the at least one application comprises setting said determined number and / or position and / or direction of at least one of: lanes of a road and / or parking spots.

9. The system according to any of the previous claims, wherein the step of processing comprises determining one or more counting locations for counting an object, such as a moving object like a vehicle or a person, in said area based on the captured sequence of images, and wherein the step of configuring comprises setting said determined one or more counting locations.

10. The system according to any of the previous claims, wherein the processing means is further configured to install the at least one application, after having selected the at least one application, wherein optionally the installing of the at least one application comprises the downloading of the at least one application from a remote server and / or a cloud server.

11. The system according to any of the previous claims, comprising a non-transitory computer- readable storage medium, one or more of the applications of the list of applications being pre-stored and / or preinstalled thereon.

12. The system according to any of the previous claims, wherein the processing means is configured to send to an operator a message indicating a selected application and / or a configuration parameter to be used in the selected application, and optionally to receive a confirmation message of the operator that the selected application is to be installed or used and / or that the configuration parameter is to be used for configuring the selected application.

13. A system, preferably an auto-commissioning system comprising an image capturing means, such as a camera, arranged to capture an image of an area, and a processing means configured to perform the steps of: controlling the image capturing means so as to capture a sequence of images over time; processing said captured sequence of images over time to determine a position and type of scene elements in said area at consecutive moments in time; configuring at least one application which uses the image capturing means, based on the determined position and type of scene elements in said area at consecutive moments in time.

14. The system according to claim 13, wherein the step of configuring comprises the selecting of at least one first sub-area in the area where a same type of object is detected multiple times at different locations within the at least one first sub-area as at least one suitable subarea for use in the at least one application.

15. The system according to the previous claim, wherein the at least one first sub-area comprises a plurality of first sub-areas and wherein the configuring further comprises selecting one of the plurality of first sub-areas as the suitable sub-area for use in the at least one application.

16. The system according to claims 14 or 15, wherein the selecting of the at least one suitable sub-area is based on any one or more of: a density of the determined positions of scene elements, an accuracy of the determined positions and / or types of scene elements, the number of detections of an object of a same type in a sub-area.

17. The system according to any one of claims 14-16, wherein a first application of the at least one application is configured to count a number of moving scene elements of a predetermined type in the at least one first sub-area.

18. The system according to claim 17, wherein said area comprises a road and the first application of the at least one application is configured to count a number of vehicles using the road in the at least one first sub-area.

19. The system according to claim 18, wherein said at least one first sub-area comprises one or more lanes of the road.

20. The system according to any one of claims 13-19, wherein the configuring comprises the selecting in the area of at least one second sub-area where a same type of object is detected multiple times at a same location within the at least one second sub-area as a suitable subarea for use in the at least one application.

21. The system according to claim 20, wherein a second application of the at least one application is configured to count a number of static objects of a predetermined type in the at least one second sub-area and / or to determine a position and / or presence of a static object in the at least one second sub-area.

22. The system according to claim 21, wherein said area comprises one or more parking spaces and the second application of the at least one application is configured to count a number of parked vehicles and / or empty parking spaces in the at least one second sub-area and / or to determine a position and / or presence of a parked vehicle in the at least one second sub-area.

23. The system of any one of the previous claims, wherein the processing means is part of any one or more of the following: the image capturing means, an edge device such as a luminaire which also includes the image capturing means, a fog device in communication with the image capturing means, a central server in communication with the image capturing means.

24. The system of any one of the previous claims, wherein the at least one application uses one or more models, and wherein the step of configuring comprises configuring the one or more models.

25. The system of the previous claim, wherein the one or more models comprise any one or more of the following: an object detection model, an object trajectory determining model, a model configured to determine if an object crosses a line, an object type determining model configured to determine a type of object.

26. The system of claim 23 or 24, wherein the one or more models are pre-trained models and / or wherein the processing means is configured to train the one or more models based on images captured by the image capturing means, wherein optionally the one or more models are one or more neural networks models.

27. The system of any one of the previous claims, wherein the system is a luminaire system comprising a light source and a drive means for driving the light source.

28. The system of the previous claim, wherein the system further comprises a control means, optionally as part of the processing means, configured to control the drive means in function of an output of the at least one application.

29. The system of any one of the previous claims, further comprising at least one sensor configured to sense an environmental characteristic of the area, such as a sound sensor and / or a radar sensor, wherein the at least one application is configured to use an output of the at least one sensor as an input of the application.

30. The system of the previous claim, wherein the step of configuring comprises configuring the application based on an output of the at least one sensor.

31. A system for identifying an object in an environment comprising: a first sensor configured to capture first data of the environment, said first data being represented in a first data space, a second sensor configured to capture second data of the environment, said second data being represented in a second data space and preferably including data related to a movement of an object or a sound made by an object, a processing means configured to process said first sensed data using a first model to derive a first region of interest in said first data space related to an object in the environment and to process said second sensed data using a second model to derive a second region of interest in said second data space related to an object in the environment, to transform the first region of interest in said first data space into a first region of interest in said second data space or to transform said second region of interest in said second data space into a second region of interest in said first data space, to fuse the transformed first region of interest or the transformed second region of interest with the second region of interest in the second data space or with the first region of interest in the first data space, respectively, and to classify an object in the environment based on the fused data and / or to train the first and / or second model using the fused data.

32. The system of the previous claim, wherein the processing means is configured to use a homographic transformation matrix defining a link between the first data space and the second data space, to perform the transforming.

33. The system of the previous claim, wherein the processing means is configured to determine the homographic transformation matrix based on the first and second sensed data, preferably at predetermined moments in time or upon request.

34. The system of any of the claims 31-33, wherein the first sensor is an imaging sensor and the first data is two-dimensional image data represented in a two-dimensional data space, for example a data space with cartesian coordinates.

35. The system of any one of the claims 31-34, wherein the second sensor is a radar.

36. The system of claim 34 and 35, wherein the processing means is configured to transform said second region of interest represented in the second data space into a second region of interest represented in said first data space, and to fuse the transformed second region of interest with the first region of interest in the first data space.

37. The system of any one of the claims 31-36, wherein the second data space is a three- dimensional data space comprising the polar coordinates: azimuth, elevation, velocity.

38. The system of claim 34 and 35, wherein the first data comprises multiple sets of frames taken at consecutive moments in time, and wherein the fusing uses a velocity of an object derived from the second sensed data as well as a velocity of an object derived from the multiple frames.

39. The system of any one of the claims 31-34, wherein the second sensor is an acoustic sensor or a seismic sensor.

40. The system of any one of the claims 31-39, wherein the first and second model are configured to detect an object in the sensed first and second data, respectively.

41. The system of any one of the claims 31-40, wherein the first and second region of interest contain an object.

42. The system of any one of the claims 31-41, wherein the first region of interest is a bounding box comprising an object.

43. The system of any one of the claims 31-42, wherein the fusing is obtained using any one or more of the following types of processing techniques: averaging, voting, ensemble learning, stacking.

44. The system of any one of the claims 31-43, comprising a network of luminaires, wherein the first and second sensor are arranged on or in a same luminaire of the network.

45. The system of any one of the claims 31-44, wherein the processing means are a centralised or distributed processing means located at a luminaire of the network and / or at a location remote from the luminaires of the network.

46. The system of any one of the claims 31-45, wherein the field-of-view of the first sensor overlaps with the field-of-view of the second sensor.

Images