Passenger counting device at the gate, counting system, vehicle and associated counting method
The passenger counting system uses an AI model to calculate passenger numbers from video streams acquired by directive cameras, addressing errors due to perspective and reducing intrusion, thereby offering an accurate and versatile counting solution for rail vehicles.
Patent Information
- Application Number
- FR2023010151
- Authority / Receiving Office
- FR · FR
- Patent Type
- Applications
- Current Assignee / Owner
- Filing Date
- 2023-09-25
- Publication Date
- 2025-03-28
AI Technical Summary
Existing passenger counting systems in rail vehicles, which use directive cameras, are prone to errors due to perspective effects, making them unsuitable for efficient counting and requiring the use of omnidirectional cameras. Additionally, these systems are intrusive as they involve individualized detection and monitoring of passengers.
A passenger counting system that employs a processing unit connected to a memory storing an artificial intelligence model trained on labeled datasets, including movement estimate matrices calculated from successive images of a video stream acquired by directive cameras. This system calculates movement estimate matrices from video flows and feeds them into the AI model to estimate the number of incoming and outgoing passengers without individualized detection.
The system provides accurate passenger counting even with directive cameras, reducing the need for omnidirectional cameras and minimizing intrusion by not requiring individual passenger identification, thus offering a more versatile and less intrusive solution.
Smart Images
Figure 00000000_0000_ABST
Abstract
Description
Title of the invention: Door passenger counting device, counting system, vehicle and counting method associated therewith Technical field
[0001] The present invention relates to a device for counting passengers at the door.
[0002] The invention also relates to a counting system, a vehicle embarking such a counting system, and an associated counting method.
[0003] The invention applies to the field of transport, in particular rail transport. State of the art
[0004] It is known to equip a vehicle, in particular a rail transport vehicle, with a passenger counting system at the door.
[0005] By "door counting" (also called "cumulative counting") for a given passenger access, such as a vehicle door, is meant, in the sense, the count of the number of passengers entering and exiting through said passenger access.
[0006] Such a counting system generally comprises at least one camera, positioned so as to acquire a video stream representative of a given passenger access, and a computer connected together.
[0007] Conventionally, the calculator is configured to receive the video stream and to: • for each image of the video stream, detect passengers and calculate, for each passenger detected, coordinates of a respective bounding box; • identify and track, in the video stream, each passenger detected; • for each passenger detected, compare a position, over time, of the respective bounding box to a predetermined crossing line associated with passenger access; • for each passenger detected, determine whether the passenger exited or re-entered the vehicle via the passenger access, based on the result of the comparison.
[0008] However, such a counting system is not satisfactory.
[0009] Indeed, to achieve efficient counting, such a counting system requires the use of omnidirectional cameras, arranged above the passenger access so that their optical axis is perpendicular to the direction of movement of the passengers.
[0010] Otherwise, especially when using directional cameras, a person inside the vehicle may be counted as exiting (or vice versa), with the relative position of the respective bounding box being underestimated due to the effects perspective. This therefore distorts the counting.
[0011] Therefore, the need for the use of omnidirectional cameras, for the known counting system, excludes the implementation of such a system in railway vehicles pre-equipped with directional cameras, which represents a large part of the cameras present in railway video surveillance systems.
[0012] Furthermore, such a counting system is intrusive in terms of personal data, insofar as it involves individualized detection and tracking of a passenger over several consecutive images of the acquired video stream.
[0013] An aim of the present invention is to remedy at least one of the drawbacks of the state of the art.
[0014] Another object of the invention is to propose a counting system which is more versatile, and which exhibits satisfactory operation even in the case of the use of directional cameras.
[0015] Another aim of the invention is to propose a counting system which is less intrusive. Statement of the invention
[0016] For this purpose, the invention relates to a counting device of the aforementioned type, comprising a memory and a processing unit connected together, the memory being configured to store an artificial intelligence model previously trained on the basis of a labeled data set, the labeled data set comprising at least one motion estimation matrix, calculated from at least two successive images of a video stream, preferably acquired by means of a directional camera, representative of a door of a vehicle between an instant of opening of the door and an instant of closing of the door, each motion estimation matrix being associated with a corresponding number of incoming and outgoing passengers, the processing unit being configured to: • calculate, from at least two successive images of at least one received video stream, a corresponding motion estimation matrix, each video stream being representative of a monitored door of a vehicle between an instant of opening of the monitored door and an instant of closing of the monitored door, each video stream having been, preferably, acquired by means of a directional camera; • provide the motion estimation matrix calculated as input to the artificial intelligence model stored in memory.
[0017] Indeed, in such a device, for each video stream, the calculation of the motion estimation matrix gives access to spatio-temporal characteristics of said video stream. Such a motion estimation matrix is indicative of the exchange passengers during the time interval in which the door shown in the video stream is open.
[0018] Providing such a motion estimation matrix as input to a suitably trained artificial intelligence model then provides access to the number of passengers entering and / or exiting during the door opening period, without requiring the identification of each passenger.
[0019] Advantageously, the counting device according to the invention has one or more of the following characteristics, taken in isolation or in any technically possible combination:
[0020] the artificial intelligence model is a convolutional regressor model comprising a 3D feature extractor associated with a regression predictive layer, the regression predictive layer preferably comprising a first output associated with the number of incoming passengers and a second output associated with the number of outgoing passengers;
[0021] the processing unit is, furthermore, configured to provide as input to the artificial intelligence model, together with the motion estimation matrix, the corresponding video stream, the artificial intelligence model being, furthermore, configured to discriminate humans in the video stream;
[0022] for each video stream, in order to calculate the corresponding motion estimation matrix, the processing unit is configured to determine an optical flow of said video stream;
[0023] for each video stream, the processing unit is configured to resample said received video stream before calculating the corresponding motion estimation matrix, the resampled video stream having a number of images equal to a nominal length associated with the artificial intelligence model.
[0024] The device according to the invention can be any type of device such as a server, a computer, a tablet, a calculator, a processor, a computer chip.
[0025] According to another aspect of the invention, there is provided a counting system comprising a counting device as defined above and at least one camera, each camera being connected to the counting device to transmit a video stream to said counting device, each camera preferably being a directional camera.
[0026] Advantageously, the counting system according to the invention is configured to, in use, trigger and interrupt an acquisition of the video stream by at least one corresponding camera as a function of a control signal from at least one associated monitored door.
[0027] According to another aspect of the invention, there is provided a vehicle, in particular a rail transport vehicle, comprising a counting system as defined above. above, each camera of the counting system being on board the vehicle and positioned so that a corresponding door of the vehicle, forming a monitored door, is located in a field of view of said camera.
[0028] According to another aspect of the invention, there is provided a method of counting passengers at the gate, the method being computer-implemented and comprising the steps: • calculation, from at least two successive images of at least one received video stream, of a corresponding motion estimation matrix, each video stream being representative of a monitored door of a vehicle between an instant of opening of the monitored door and an instant of closing of the monitored door, each video stream having been, preferably, acquired by means of a directional camera; • supply of the motion estimation matrix calculated as input to a artificial intelligence model previously trained on the basis of a labeled dataset, the labeled dataset comprising at least one motion estimation matrix, calculated from at least two successive images of a video stream, preferably acquired by means of a directional camera, representative of a door of a vehicle between an instant of opening of the door and an instant of closing of the door, each motion estimation matrix being associated with a corresponding number of incoming and outgoing passengers.
[0029] According to another aspect of the invention, there is provided a computer program comprising executable instructions which, when executed by computer, implement the steps of the method as defined above.
[0030] The computer program can be in any computer language, such as for example machine language, C, C++, JAVA, Python, etc. Brief description of the figures
[0031] The invention will be better understood on reading the following description, given solely by way of non-limiting example and made with reference to the appended drawings in which:
[0032] [Fig.l] is a schematic representation of a railway vehicle according to the invention; and
[0033] [Fig.2] is a flowchart of a counting method implemented by a counting system of the railway vehicle of [Fig.l].
[0034] It is understood that the embodiments which will be described below are in no way limiting. In particular, it will be possible to imagine variants of the invention comprising only a selection of characteristics described below isolated from the other features described, if this selection of features is sufficient to confer a technical advantage or to differentiate the invention from the state of the prior art. This selection includes at least one preferably functional feature without structural details, or with only part of the structural details if it is this part which is only sufficient to confer a technical advantage or to differentiate the invention from the state of the prior art.
[0035] In particular, all the variants and all the embodiments described can be combined with each other if nothing prevents this combination from a technical point of view.
[0036] In the figures and in the remainder of the description, the elements common to several figures retain the same reference. Detailed description
[0037] A vehicle 2 according to the invention is illustrated by [Fig.l].
[0038] The vehicle 2 is, in particular, a passenger transport vehicle, preferably a rail transport vehicle such as a train.
[0039] The vehicle 2 comprises at least one door 4, intended to allow passengers to board the vehicle 2, and to disembark from the vehicle 2.
[0040] Furthermore, the vehicle 2 comprises a counting system 6 intended to carry out a passenger count at the door.
[0041] The counting system 6 comprises at least one camera 8 and one counting device 10 connected to each other. Camera 8
[0042] Each camera 8 is mounted on board the vehicle 2. Furthermore, each camera 8 is positioned so that, in its field of view, there is a corresponding door 4 of the vehicle 2, called the “monitored door”.
[0043] Preferably, each camera 8 is a directional camera (as opposed to an omnidirectional camera). In this case, each camera 8 is, for example, oriented so that a respective optical axis forms an angle of between 30° and 60° with a floor of the vehicle 2. In this way, each camera 8 is able to collect front images of the passengers present in the corresponding field of view.
[0044] Each camera 8 is configured to acquire a video stream from the corresponding monitored door 4, and to transmit the acquired video stream to the counting device 10.
[0045] Such a video stream comprises N images of dimensions (X,Y) each. For example, in each of the images of the video stream, each corresponding pixel is, in addition, associated with its amplitude for each channel among the red, green and blue channels.
[0046] Preferably, each camera 8 is an IP technology camera (also called an “IP camera”) connected to a port of a network switch (also called "switch") of the vehicle 2 of a communication network of the vehicle 2, such as an Ethernet IP wired communication network having a ring topology. In this case, each camera 8 is configured to transmit the acquired video stream to the counting device 10 via said communication network.
[0047] Preferably, the counting system 6 is installed in the vehicle 2 so as to receive, for at least one monitored door 4, a control signal from said monitored door 4. More precisely, such a control signal is a control signal for the opening and / or closing of the monitored door 4. In this case, the counting system 6 is configured so that, for at least one camera 8, the triggering and / or interruption of the acquisition of the corresponding video stream depends on the control signal. More precisely still, the counting system 6 is configured so that, for at least one camera 8, the acquisition of the corresponding video stream is triggered when the associated monitored door is opened and / or interrupted when said monitored door 4 is closed. Counting device 10
[0048] The counting device 10 is intended to determine, for each monitored door 4, the number of incoming and outgoing passengers as a function of the video stream received from each corresponding camera 8.
[0049] The counting device 10 is on board the vehicle 2. Alternatively, the counting device 10 is a ground-based device. In this case, the transmission of each video stream acquired from each camera 8 to the counting device 10 is likely to be carried out by means of any appropriate communication means, such as a wireless communication network.
[0050] The counting device 10 comprises a memory 12 and a processing unit 14 connected together. Memory 12
[0051] The memory 12 is configured to store an artificial intelligence model 16 (hereinafter, “counting model”).
[0052] The counting model 16 has been previously trained on the basis of a labeled dataset. In this case, the labeled dataset comprises at least one motion estimation matrix associated with a corresponding number of incoming and outgoing passengers. Such a motion matrix has been calculated from at least two successive images of a video stream representative of a door of a vehicle, between an instant of opening of the door and an instant of closing of the door.
[0053] Advantageously, the counting model 16 is a convolutional neural network with regression head model comprising a 3D feature extractor associated with a predictive regressor layer. aggression.
[0054] For example, the counting model 16 is a convolutional regressor model of type X3D (for "Expand 3D"), as described by Christoph Feichtenhofer in the digital preprint "X3D: Expanding Architectures for Efficient Video Recognition", referenced arXiv:2004.04730.
[0055] The 3D feature extractor is adapted to extract the spatiotemporal features of the video stream.
[0056] For example, the 3D feature extractor is chosen from 3D convolutional networks (for example: X3D, or C3D, for “Convolutional 3D”, as described by Du Tran et al. in the digital preprint “Learning Spatiotemporal Features with 3D Convolutional Networks”, referenced arXiv: 1412.0767), recurrent neural networks (for example: LSTM, for “Eong Short-Term Memory”, or long short-term memory in French; GRU, for “Gated Recurrent Unit”, or closed recurrent unit in French), or so-called “two-stream” convolutional neural networks, which comprise two convolutional neural networks operating in parallel.
[0057] Further, the regression predictive layer is adapted to receive outputs from the 3D feature extractor to derive the number of incoming passengers and outgoing passengers. Preferably, the regression predictive layer comprises a first output associated with the number of incoming passengers and a second output associated with the number of outgoing passengers.
[0058] Preferably, the counting model 16 is associated with a nominal length, representative of the length that a motion estimation matrix provided at its input must have.
[0059] As will emerge from the following description, the length of the motion estimation matrix is equal to the number of images of the video stream from which said motion estimation matrix is calculated. Processing Unit 14
[0060] The processing unit 14 is configured to receive the video stream acquired from each camera 8, and to implement the counting model 16 stored in the memory 12 in order to determine, for each door 4, the number of incoming passengers and the corresponding number of outgoing passengers.
[0061] Preferably, the processing unit 14 has a compact shape guaranteeing a small footprint in the vehicle 2, and is, in particular, compliant with railway standards for electromagnetic compatibility, resistance to fire and smoke, and resistance to shock and vibrations.
[0062] In order to determine, for each door 4, the number of incoming passengers and the corresponding number of outgoing passengers, the processing unit 14 is configured to implement a counting method 20 ([Fig.2]).
[0063] As illustrated by [Fig.2], the counting method comprises an optional resampling step 22, a step 24 of calculating a motion estimation matrix (called a “calculation step”) and an estimation step 26.
[0064] Preferably, the processing unit 14 is configured to resample, during the optional resampling step 22, each received video stream according to the nominal length associated with the counting model 16. For example, if a given video stream comprises a number of images greater than the nominal length, the processing unit 14 is configured to subsample said video stream so that its final length is equal to the nominal length. Alternatively, or in a complementary manner, if a given video stream comprises a number of images less than the nominal length, the processing unit 14 is configured to oversample said video stream so that its final length is equal to the nominal length.
[0065] The processing unit 14 is also configured to, during the calculation step 24, calculate from at least two successive images of each video stream received, a corresponding motion estimation matrix.
[0066] Such a motion estimation matrix is a sequence of the same length N as the video stream, and which is representative, for each pixel, of the quantity of movement, for the point of the scene corresponding to said pixel, between the two successive images. The motion estimation matrix comprises N arrays of dimensions (X,Y) each, each array being associated with a respective image. For example, in each array, each cell is associated with a respective pixel of the corresponding image and is representative of a displacement of said pixel in two directions not merged between said image and the following image.
[0067] Preferably, to calculate the motion estimation matrix associated with the video stream received from a given camera 8, the processing unit 14 is configured to determine an optical flow (also called “dense optical flow”) of said video stream, for example by means of a RAFT (Recurrent All-pairs Field Transforma) artificial intelligence model, or by implementing a Lucas-Kanade algorithm.
[0068] For each camera 8, the processing unit 14 is also configured to, during the estimation step 26, provide the motion estimation matrix calculated as input to the counting model 16 stored in the memory 12. The estimated number of incoming passengers and the number of outgoing passengers are then provided as output from the counting model 16.
[0069] Advantageously, the processing unit 14 is configured to provide, as input to the counting model 16, together with the motion estimation matrix, the corresponding video stream. For example, the processing unit 14 is configured to provide, as input to the counting model 16, the motion estimation matrix concatenated to the corresponding video stream. In this case, the counting model 16 was configured to discriminate humans in the video stream.
[0070] This feature is advantageous, insofar as retaining the images from the video stream causes the counting model 16 to perform the necessary discrimination to count only humans. Functioning
[0071] The operation of the counting system 6 will now be described with reference to [Fig.2].
[0072] Each camera 8 acquires a video stream of the corresponding monitored door 4. Preferably, each camera 8 acquires the video stream only between an opening time and a closing time of the corresponding monitored door 4.
[0073] Then, each camera 8 transmits the acquired video stream to the counting device 10.
[0074] Preferably, the processing unit 14 of the counting device 10 resamples first, during the optional resampling step 22, each received video stream according to the nominal length associated with the counting pattern 16.
[0075] Then, during the calculation step 24, the processing unit 14 calculates the motion estimation matrix corresponding to each video stream received (potentially resampled), from at least two successive images of said video stream.
[0076] Then, during the estimation step 26, the processing unit 14 provides each calculated motion estimation matrix as input to the artificial intelligence model stored in the memory 12. The estimated number of incoming passengers and the number of outgoing passengers are then provided as output from the counting model 16.
[0077] Of course, the invention is not limited to the examples which have just been described.
Claims
Claims
1. Device (10) for counting passengers at the door, comprising a memory (12) and a processing unit (14) connected together, the memory (12) being configured to store an artificial intelligence model (16) previously trained on the basis of a labeled data set, the labeled data set comprising at least one motion estimation matrix, calculated from at least two successive images of a video stream, preferably acquired by means of a directional camera, representative of a door of a vehicle between an instant of opening of the door and an instant of closing of the door, each motion estimation matrix being associated with a corresponding number of incoming and outgoing passengers, the processing unit (14) being configured to: • calculate, from at least two successive images of at least one received video stream, a corresponding motion estimation matrix,each video stream being representative of a monitored door of a vehicle between an opening time of the monitored door and a closing time of the monitored door, each video stream having been, preferably, acquired by means of a directional camera; • provide the motion estimation matrix calculated as input to the artificial intelligence model stored in the memory.,
2. The counting device (10) of claim 1, wherein the artificial intelligence model is a convolutional regressor model comprising a 3D feature extractor associated with a regression predictive layer, the regression predictive layer preferably comprising a first output associated with the number of incoming passengers and a second output associated with the number of outgoing passengers.
3. Counting device (10) according to claim 1 or 2, wherein the processing unit (14) is further configured to provide as input to the artificial intelligence model, together with the motion estimation matrix, the corresponding video stream, the artificial intelligence model being further configured to discriminate the humans in the video stream.
4. Counting device (10) according to any one of claims 1 to 3, wherein, for each video stream, to calculate the corresponding motion estimation matrix, the processing unit (14) is configured to determine an optical flow of said video stream.
5. Counting device (10) according to any one of claims 1 to 4, wherein, for each video stream, the processing unit (14) is configured to resample said received video stream before calculating the corresponding motion estimation matrix, the resampled video stream having a number of images equal to a nominal length associated with the artificial intelligence model (16).
6. A counting system (6) comprising a counting device (10) according to any one of claims 1 to 5 and at least one camera (8), each camera being connected to the counting device (10) for transmitting a video stream to said counting device (10), each camera preferably being a directional camera.
7. A counting system (6) according to claim 6, configured to, in use, trigger and interrupt an acquisition of the video stream by at least one corresponding camera (8) as a function of a control signal from at least one associated monitored door (4).
8. Vehicle (2), in particular a rail transport vehicle, comprising a counting system (6) according to claim 6 or 7, each camera (8) of the counting system (6) being on board the vehicle (2) and positioned so that a corresponding door (4) of the vehicle, forming a monitored door (4), is located in a field of view of said camera (8).
9. Method for counting passengers at the door, the method being implemented by computer and comprising the steps: • calculation (24), from at least two successive images of at least one received video stream, of a corresponding motion estimation matrix, each video stream being representative of a monitored door of a vehicle between an instant of opening of the monitored door and an instant of closing of the monitored door, each video stream having been, preferably, acquired by means of a directional camera; • supply (26) of the calculated motion estimation matrix as input to an artificial intelligence model (16) previously trained on the basis of a labeled dataset, the labeled dataset comprising at least one motion estimation matrix, calculated from at least two successive images of a video stream, preferably acquired by means of a directional camera, representative of a door of a vehicle between an instant of opening of the door and an instant of closing of the door, each motion estimation matrix being associated with a corresponding number of incoming and outgoing passengers.
10. A computer program comprising executable instructions which, when executed by a computer, implement the steps of the method according to claim 9.
Citation Information
Patent Citations
Elevator traffic flow recognition method based on computer vision
CN115303901A
Passenger counter
US20140333769A1