Method and system for locating individuals in a surveillance area

Abstract

Method and system for locating individuals in a surveillance space The system (2) for locating individuals comprises a plurality of sensors (4A, 4B, 4C) of at least two distinct sensor categories and a locating device (6), adapted to acquire sensor location data (23At, 23Bt, 23Ct), relating to at least one individual.The device (6) implements, for at least one pair of sensors, fusion modules (26, 28, 30) of the sensor location data provided by the sensors of said pair of sensors at a current time instant, allowing at least one adjusted location data point (25t) to be obtained, and a calculation module (34) of at least one consolidated location data point (29t) at the current time instant, comprising a fusion of at least one adjusted location data point (25t) and at least one predicted location data point (27t) from at least one previous consolidated location data point, associated with a previous time instant. Figure for the abstract: Figure 1.

Description

Title of the invention: Method and system for locating individuals in a surveillance space

[0001] The present invention relates to a method for locating individuals in a surveillance space.

[0002] It also relates to an associated localization system and an associated computer program.

[0003] The invention lies in the general field of video surveillance, and finds numerous applications in fields such as infrastructure management, robotics or autonomous navigation.

[0004] The precise location of individuals likely to move within a monitored area is useful for ensuring general security and optimizing various operations. The monitored area could be, for example, a commercial or transport infrastructure (train station, airport, etc.) or any other place likely to accommodate the public, or the interior of a transport vehicle, or even a road intersection. This list is not exhaustive.

[0005] Various sensor-based individual localization systems for obtaining sequences of 2D or 3D images (i.e., videos) of the surveillance space have been proposed. The sensors used include, in particular, optical or infrared video surveillance cameras, time-of-flight (TOF) cameras, and 3D Lidar sensors.

[0006] The various known sensors exhibit operational weaknesses depending on the application and do not consistently provide sufficient accuracy. Indeed, some sensors perform less well under certain lighting conditions or crowd density, and are therefore not robust to all types of situations.

[0007] Moreover, the cost of some sensors is particularly high, so it is difficult to use a large number of them.

[0008] To improve the accuracy of individual location, solutions based on wearing a radio-frequency identification (RFID) badge or other transmitters worn by individuals have been proposed. However, these solutions are intrusive and unsuitable for a public surveillance environment.

[0009] The invention aims to remedy the drawbacks of the prior art by proposing a solution allowing for a more precise and robust localization of individuals in the face of contextual changes, while not being intrusive.

[0010] To this end, the invention proposes, according to one aspect, a method for locating individuals in a surveillance area, comprising a temporal succession of acquisitions of location data, referred to as sensor location data, of at least one individual, obtained from data acquired by a plurality of distinct sensors, the sensors belonging to at least two distinct categories of sensors. This method comprises the following steps: - for at least one selected pair of sensors, fusion of the sensor location data provided by the sensors of said pair of sensors at a current time instant, allowing at least one adjusted location data to be obtained, associated with the current time instant and said pair of sensors, - calculation of at least one consolidated location data point, associated with the current time instant, said calculation comprising: • obtaining at least one previous consolidated location data point, associated with a previous time point, • a prediction of at least one predicted location data point for the current time instant, based on said at least one previous location data point,

[0011] the fusion of at least one predicted location data point and at least one adjusted location data point. Advantageously, taking into account sensor location data from a plurality of sensors and at least one predicted location data point makes it possible to improve the accuracy of locating individuals from data from multiple sensors.

[0012] The method for locating individuals in a surveillance space, according to the invention, may also have one or more of the characteristics below, taken independently or according to all technically conceivable combinations.

[0013] The method includes an estimation of a confidence score associated with the sensor location data, the fusion of sensor location data being a function of said confidence score.

[0014] Estimating a confidence score implements an artificial intelligence engine, previously trained by machine learning to assign a confidence score to the location data obtained by each sensor based on context parameters.

[0015] Estimating a confidence score uses previously recorded confidence score tables.

[0016] The fusion of location data includes steps for calculating a distance matrix between initial location data from a first sensor and second location data from a second sensor of said sensor pair, and matching between first and second location data.

[0017] The matching implements a Munkres combinatorial optimization algorithm.

[0018] The method further includes a step of calculating location data adjusted according to the result of the matching step and the confidence scores associated with the sensor location data.

[0019] The calculation of the adjusted location data implements a weighted centroid calculation by the confidence scores associated with the location data.

[0020] The plurality of distinct sensors comprising at least two pairs of sensors, and the fusion of location data is implemented for each pair of sensors, the method further comprising an iteration of the fusion on the adjusted location data associated with two distinct pairs of sensors.

[0021] The time prediction step implements a Kalman filter.

[0022] According to another aspect, the invention relates to a system for locating individuals in a surveillance space comprising a plurality of sensors belonging to at least two distinct sensor categories and a localization device adapted to perform a temporal succession of acquisitions of location data, referred to as sensor localization data, of at least one individual, obtained from data acquired by the plurality of distinct sensors, the device comprising a processor configured to implement: - for at least one selected pair of sensors, a module for merging sensor location data provided by the sensors of said pair of sensors at a current time instant, allowing at least one adjusted location data point associated with the current time instant and said pair of sensors, - a calculation module for at least one consolidated location data point, associated with the current time instant, said calculation comprising • a module for obtaining at least one previous consolidated location data point, associated with a previous time point, • a prediction module for at least one predicted location data point for the current time instant, based on said at least one previous location data point, • a module for merging at least one predicted location data point and at least one adjusted location data point.

[0023] According to one variant, the individual location system is configured to further implement a module for estimating a confidence score associated with the sensor location data, the sensor location data fusion module using said confidence score for each sensor.

[0024] According to another aspect, the invention relates to an information recording medium, on which software instructions are stored for the execution of a method of locating individuals in a surveillance space as briefly described above, when these instructions are executed by a programmable electronic device.

[0025] According to another aspect, the invention relates to a computer program comprising software instructions which, when implemented by a programmable electronic device, implement a method for locating individuals in a surveillance space as briefly described above.

[0026] The invention will become clearer upon reading the following description, given solely by way of non-limiting example, and made with reference to the drawings in which:

[0027] [Fig-1] [Fig.1] represents the main functional blocks of a system of location of individuals in a surveillance area according to a mode of implementation;

[0028] [Fig.2] [Fig.2] is a synoptic diagram of the steps of a localization process according to a method of implementation;

[0029] [Fig.3] [Fig.3] is a synoptic diagram of steps following the steps of [Fig.2] of a localization method according to an embodiment;

[0030] [Fig.4] [Fig.4] is a schematic illustration of location data at various stages of the localization process.

[0031] Fig. 1 schematically illustrates a system 2 for locating individuals in a surveillance space.

[0032] The surveillance space is not shown.

[0033] System 2 comprises at least one pair of distinct sensors, and in the example shown comprises a plurality of sensors 4A, 4B, 4C, in variable number.

[0034] By way of example, the sensors belong to several distinct sensor categories and include, for example, two-dimensional image sensors, in particular optical cameras 4A and / or infrared cameras 4B, and / or TOF cameras 4C which provide distance measurements between the camera and a detected individual.

[0035] The number of sensors in each sensor category varies depending on the application case.

[0036] The respective sensors 4A, 4B, 4C are positioned at various spatial positions in the surveillance space, the spatial positions being chosen so as to cover the surveillance space with the shooting fields of each of the sensors.

[0037] It is understood that the term category of sensors refers to sensors having similar modes of operation and constituent elements.

[0038] Thus, sensors of the same category of sensors have a similar performance, but which varies according to external conditions, also called operating context, for example lighting conditions, or crowd density in the surveillance space.

[0039] The various sensors are configured to acquire temporal sequences of data, for example temporal sequences of data forming two-dimensional images when the sensors are optical or infrared cameras.

[0040] The individual localization system 2 also includes a localization device 6, configured to communicate with sensors 4A, 4B, 4C and in particular configured to receive image data and / or sensor localization data from the respective sensors 4A, 4B, 4C.

[0041] The location device 6 is a programmable electronic device, e.g. a computer, and comprises, in one embodiment, one or more processors 8, an electronic memory unit 10, an input / output interface 12 and a communication interface 14, these elements being configured to communicate with each other via a communication bus 15 internal to the device 6.

[0042] The input / output interface 12 includes in particular a display screen, and means of data entry (keyboard, mouse or touchpads) allowing a monitoring operator to carry out real-time monitoring of the monitoring area.

[0043] In one embodiment, as illustrated in [Fig.1], the localization device 6 comprises modules 20A, 20B, 20C for calculating sensor localization data of at least one individual from successive image data obtained by sensors 4A, 4B, 4C.

[0044] Alternatively, at least part of the modules 20A, 20B, 20C are integrated into the respective sensors 4A, 4B, 4C, and the sensor location data are provided by each of the sensors 4A, 4B, 4C to the location device 6.

[0045] Modules 20A, 20B, 20C implement any spatial localization algorithm for individuals, adapted for the category of sensor considered, allowing the obtaining of sensor localization data 23At, 23Bt, 23Ct, associated with time instants t.

[0046] The following description describes the processing of location data, tracked over time, of a given individual.

[0047] It should be noted that the method and system described here for locating individuals apply in a similar way to locating other entities of interest, for example vehicles, animals...

[0048] The localization device 6 further includes a module 22 for preprocessing the localization data from sensors.

[0049] It further includes a module 24 for estimating a confidence score associated with the sensor location data for each sensor. According to an unshown variant, the confidence score estimation module 24 groups several confidence score estimation modules, each estimation module being associated with a sensor category. The confidence score estimation module 24, or each confidence score estimation module, makes it possible to estimate a confidence score based on the sensor, parameters and context, and optionally, the location algorithm implemented.

[0050] The localization device 6 further comprises a distance matrix calculation module 26, a matching module 28, and a post-processing module 30, which performs the calculation of adjusted localization data. Modules 26, 28, and 30 cooperate to perform a fusion of localization data from two distinct sources, for example, a pair of sensors, making it possible to obtain adjusted localization data 25t associated with the time instant t.

[0051] The localization device 6 also includes a module 32 for predicting a predicted location data 271 for a current time instant from at least one previous location data, associated with a previous time instant tk, for example k being an integer greater than or equal to 1, and a module 34 for calculating at least one consolidated location 29t associated with a current time instant.

[0052] In one embodiment, modules 20, 22, 24, 26, 28, 30, 32, 34 are implemented in the form of software instructions forming a computer program, which, when executed by a programmable electronic device, implements a method for locating individuals in a surveillance space as described.

[0053] In an alternative not shown, modules 20, 22, 24, 26, 28, 30, 32, 34 are each implemented as programmable logic components, such as FPGAs (Field Programmable Gamut Arras), microprocessors, GPUs (General-purpose graphics processing), or dedicated integrated circuits, such as ASICs (Application-Specific Integrated Circuits).

[0054] The computer program comprising software instructions is further capable of being recorded on a non-transient, computer-readable information storage medium. This computer-readable medium is, for example, a medium capable of storing electronic instructions and being connected to a bus of a computer system. By way of example, this medium is an optical disc, a magneto-optical disc, a ROM, a RAM, any type of non-volatile memory. (e.g. EPROM, EEPROM, FLASH, NVRAM), a magnetic card or an optical card.

[0055] Figure 2 is a synoptic diagram of the steps in the process of locating individuals in a surveillance space, implemented to carry out a first phase of location data merging.

[0056] The method includes a first step 40 of acquiring sensor location data from a plurality of distinct sensors, belonging to at least two distinct sensor categories.

[0057] According to the embodiments, either location data are directly provided by the sensors, or an algorithm for calculating sensor location data of at least one individual from successive image data obtained by sensors 4A, 4B, 4C.

[0058] The sensor location data includes Cartesian coordinates (xi5y i,Zi) of at least one point belonging to the individual in a 3D reference frame.

[0059] In practice, for each sensor, the sensor location data includes a plurality of points corresponding to one or more individuals.

[0060] Step 40 is followed by a normalization pre-processing step 42, comprising distortion correction, coordinate alignment and normalization of measurement units, to ensure that sensor location data from sensors of distinct sensor categories are comparable.

[0061] The method then includes processing the location data of sensors D_lt, D_2t by pairs of sensors.

[0062] The sensor pairs are chosen beforehand.

[0063] For example, a pair of sensors groups sensors of the same category in a first processing phase.

[0064] The process then includes a step 44 of estimating a confidence score associated with the sensor location data D_lt, D_2t.

[0065] A confidence score is associated with the location data; in other words, a confidence score makes it possible to evaluate the reliability of the location data obtained from the sensor in question.

[0066] The estimation 44 of a confidence score is a function of the sensor category for each sensor in the pair of sensors considered, as well as the context of each sensor, for example lighting conditions, or crowd density in the surveillance space.

[0067] Advantageously, the estimation 44 of a confidence score is carried out dynamically, so as to take into account any changes in the context which may impact the performance of each sensor.

[0068] Thus, for example, in low illumination conditions, location data obtained from images provided by an optical camera have a low confidence score, whereas location data obtained from images provided by an infrared camera have a high confidence score.

[0069] In one embodiment, step 44 implements an artificial intelligence engine, previously trained by machine learning to assign a confidence score to the location data obtained by each sensor based on the sensor category, context parameters, for example based on the time of day, lighting conditions (i.e. sensor exposure relative to a light source), weather conditions when the sensor is positioned outdoors, etc.

[0070] For example, the artificial intelligence engine implements a neural network.

[0071] Alternatively, estimation 44 of a confidence score uses previously recorded tables, developed with the help of experts, providing a confidence score based on context parameters, for example based on the time of day, weather conditions when the sensor is positioned outdoors.

[0072] Context parameters are obtained for example from separate suitable sensors or are obtained from an external server, for example a meteorological data server, via a communication link with the location device 2.

[0073] The process then includes a step 46 of merging (or first merging) the first sensor location data D_lt and the second location data D_2t.

[0074] In practice, for each sensor in the pair of sensors, the sensor location data D_lt, D_2t include different numbers of points represented by Cartesian coordinates, for example M points for a first sensor in the pair of sensors, N points for a second sensor in the pair of sensors, M and N being distinct integers.

[0075] The fusion 46 then includes a step 48 of calculating a distance matrix between the location data D_lt, D_2t, then a step 50 of matching (or assignment) between sensor location data D_lt from the first sensor and sensor location data D_2t from the second sensor.

[0076] Advantageously, the matching step 50 implements a Munkres combinatorial optimization algorithm, known for solving the assignment problem between entities of sets of different cardinalities, in this case between the M points from the first sensor and the N points from the second sensor.

[0077] Advantageously, Munkres' combinatorial optimization algorithm makes it possible to solve the assignment problem in polynomial time.

[0078] The fusion 46 then includes a step 52 of calculating the adjusted location data D_a,t as a function of the result of the matching step 50 and the confidence scores associated with the sensor location data for each sensor.

[0079] In one embodiment, step 52 implements a weighted centroid calculation based on the confidence scores associated with the location data.

[0080] By way of example, we consider two respective points PI with Cartesian coordinates (xl,yl,zl) and P2 with Cartesian coordinates (x2,y2,z2), which were matched in step 50, PI being a sensor location data of the first sensor and P2 being a sensor location data of the second sensor.

[0081] In step 52, the confidence score SI, S2 associated with each sensor is taken into account. For example, a new point P3 which is an adjusted location data is obtained, the Cartesian coordinates (x3, y3, z3) of P3 being calculated by weighted average of the Cartesian coordinates of points PI and P2, with weighting coefficients deduced from the confidence scores.

[0082] For example, the following formulas are applied:

[0083] v S,x,^ x3- S,+S2

[0084] s^2

[0085] _ ~3 ~ Si+S2

[0086] Point P3 is then the barycenter of points PI and P2, calculated with the confidence scores SI, S2 as weighting coefficients (or weights).

[0087] At the end of the fusion step 46, adjusted location data are thus obtained. The adjusted location data are stored.

[0088] Steps 44 for estimating confidence scores and 46 for merging location data are repeated for each pair of sensors chosen.

[0089] At the end of this first processing phase, in a second processing phase, if necessary (i.e. if the number of sensor pairs is strictly greater than 1), the location data fusion step 46 is repeated to fuse previously calculated adjusted location data, until at least one adjusted location data D'_a,t corresponding to the plurality of sensors is obtained.

[0090] Confidence scores are also merged; when adjusted location data from two separate sources are merged, the associated confidence score is, for example, the average of the confidence scores from each data source.

[0091] The process then involves calculating at least one consolidated location data, as explained below with reference to [Fig.3].

[0092] The method then includes the acquisition of at least one previous consolidated location data D_c,tk, associated with a previous time instant tk, for example t-1, and a temporal prediction 60 of at least one predicted location data Dpred_t, for the current time instant t, from the at least one previous consolidated location data D_c,tk.

[0093] The time prediction step 60 implements, for example, a Kalman filter.

[0094] In other words, for each point Pl(tk), P2(tk), P3(tk) corresponding to the consolidated location data at time tk, a predicted point Pl-pred(t), P2-pred(t), P3-pred(t) is obtained, as schematically illustrated in [Fig.4].

[0095] At least one adjusted location data D'_a,t corresponding to the plurality of sensors and at least one predicted location data Dpred_t are provided as input to a fusion step 62 (second fusion).

[0096] The fusion step 62 includes a step 64 of calculating a distance matrix between the adjusted location data D'_a,t and the predicted location data Dpred_t, analogous to step 48 described above, and then a matching (or assignment) step 66 analogous to step 50 described above.

[0097] It then includes a step 68 of calculation of at least one consolidated location data D_c,t, associated with the current time t.

[0098] In one embodiment, during step 68, when an association has been made during the matching step 66 between a predicted location data Dpred_t, and an adjusted location data D'_a,t the adjusted location data is retained as consolidated location data D_c,t.

[0099] According to one variant, the Cartesian coordinates of the consolidated location data D_c,t are calculated by averaging the Cartesian coordinates of the adjusted location data D'_a,t and the predicted location data Dpred_t

[0100] Figure 4 schematically illustrates, by way of example, in the same 2D spatial reference frame: points Pl_a,t, P2_a,t, P3_3,t (reference 70 in Figure 4) corresponding to the adjusted location data calculated for time t; points Pl(tk), P2(tk), P3(tk) corresponding to the consolidated location data at time tk and the corresponding predicted points at time t, denoted Pl-pred(t), P2-pred(t), P3-pred(t) (reference 72 in Figure 4). The results of the matching 66 are illustrated in reference 74 of Figure 4.

[0101] Finally, the points Pl-c(t), P2-c(t), P3-c(t) corresponding to the consolidated location data associated with the current time, calculated from the points matched, are illustrated in reference 76 of [Fig.4].

[0102] Of course, the example in [Fig.4] is schematic and simple, the location data to be dealt with in practical cases being much more numerous.

Claims

Demands

1. A method for locating individuals in a surveillance space, comprising a temporal succession of acquisitions of location data, referred to as sensor location data (23At, 23Bt, 23Ct), of at least one individual, obtained from data acquired by a plurality of distinct sensors (4A, 4B, 4C), the sensors belonging to at least two distinct sensor categories, the method being characterized in that it comprises the steps of: - for at least one chosen pair of sensors, fusion (46) of the sensor location data provided by the sensors of said pair of sensors at a current time instant, making it possible to obtain at least one adjusted location data associated with the current time instant and with said pair of sensors, - calculation of at least one consolidated location data, associated with the current time instant,said calculation comprising: • Obtaining at least one previous consolidated location data point associated with a previous time instant, • A temporal prediction (60) of at least one predicted location data point for the current time instant, based on said at least one previous location data point, • A merging (62) of the at least one predicted location data point and the at least one adjusted location data point.

2. A method according to claim 1, comprising an estimation (44) of a confidence score associated with sensor location data, the fusion (46) of sensor location data being a function of said confidence score.

3. A method according to claim 2, wherein the estimation (44) of a confidence score implements an artificial intelligence engine, previously trained by machine learning to assign a confidence score to the location data obtained by each sensor as a function of context parameters.

4. A method according to claim 2, wherein the estimation (44) of a confidence score uses previously recorded confidence score tables.

5. A method according to any one of claims 2 to 4, wherein the fusion (46) of location data comprises steps of calculating (48) a distance matrix between first location data from a first sensor and second location data from a second sensor of said sensor pair, and of matching (50) between first and second location data.

6. Method according to claim 5, wherein said matching (50) implements a Munkres combinatorial optimization algorithm.

7. A method according to any one of claims 5 or 6, further comprising a step of calculating the adjusted location data (52) based on the result of the matching step and the confidence scores associated with the sensor location data.

8. A method according to claim 7, wherein the calculation of the adjusted location data (52) implements a weighted centroid calculation by the confidence scores associated with the location data.

9. A method according to any one of claims 1 to 8, the plurality of separate sensors comprising at least two pairs of sensors, wherein the fusion of location data (46) is implemented for each pair of sensors, the method further comprising an iteration of the fusion on the adjusted location data associated with two pairs of separate sensors.

10. A method according to any one of claims 1 to 9, wherein the time prediction step (60) implements a Kalman filter.

11. A computer program comprising software instructions which, when executed by a programmable electronic device, implement a method for locating individuals in a surveillance space in accordance with claims 1 to 10.

12. A system for locating individuals in a surveillance space, comprising a plurality of sensors (4A, 4B, 4C) belonging to at least two distinct sensor categories and a device for

13. localization (6) adapted to perform a temporal succession of acquisitions of localization data, referred to as sensor localization data, of at least one individual, obtained from data acquired by the plurality of distinct sensors, the device (6) being characterized in that it comprises a processor (8) configured to implement: - for at least one selected pair of sensors, a fusion module (26, 28, 30) of the sensor location data provided by the sensors of said pair of sensors at a current time instant, allowing at least one adjusted location data associated with the current time instant and said pair of sensors, - a calculation module (34) for at least one consolidated location data point, associated with the current time instant, configured to implement: • a module for obtaining at least one previous consolidated location data point, associated with a previous time point, • a prediction module for at least one predicted location data point for the current time instant, based on said at least one previous location data point, • a module for merging at least one predicted location data point and at least one adjusted location data point. Individual localization system according to claim 12, configured to further implement a module (24) for estimating a confidence score associated with sensor localization data, the sensor localization data fusion module using said confidence score.

Images