Determining the closure of railway vehicle doors by audio and / or visual analysis

Abstract

Determination of door closure (2) of a railway vehicle by audio and / or visual analysis, preferably using cameras and / or microphones belonging to an on-board video surveillance system (6). The video surveillance system (6) is more specifically configured to acquire an image stream of the doors (2) using the cameras and / or to acquire sounds emitted by snorers (3). Figure for the abstract: Fig. 1

Description

Title of the invention: Determination of railway vehicle door closure by audio and / or visual analysis technical field

[0001] The invention relates to the field of transport, in particular rail transport, and to the field of processing information provided by equipment of a transport vehicle. Prior art

[0002] Modern trains use intelligent devices, particularly to assess the presence or number of passengers, typically through computer analysis of images acquired using cameras. Image capture and analysis require ensuring that certain conditions are met. An important condition relates to the closing of the doors, since the number of passengers on the train is not reasonably likely to change when the doors are closed.

[0003] The door closing command information is transmitted through a computer network known as a "control network".

[0004] In the context of modernizing trains that we wish to equip with an intelligent device as described above, several obstacles arise given the computer architecture equipping these trains.

[0005] A first type of train comprises a single computer network, consisting of the control and command network. Integrating an intelligent device into such an architecture would require modifying the control and command network and, consequently, conducting a safety and regression test on all network equipment. Indeed, the control and command network connects critical train equipment such as brakes. The cost of such an operation would be particularly high. Moreover, some control and command networks use old and obsolete technologies with which such an interconnection of an intelligent device is undesirable.

[0006] A second type of train includes both a control and command network for critical equipment and a second computer network, called the "comfort network," for non-critical equipment such as counting devices, video surveillance, or passenger information systems. These two networks are linked to each other by a gateway generally configured to transmit information from the control and command network to the comfort network in unicast mode, that is, from a given piece of equipment on the control and command network. to a specific piece of equipment in the comfort network. Such a gateway is typically configured to transmit door closure information to the passenger information system. Under these conditions, door closure information is not available to additional equipment such as the smart device that one wishes to integrate. Description of the invention

[0007] In order to overcome the aforementioned drawbacks, the invention relates to a system for determining the position of at least one door of a vehicle, preferably a railway vehicle. According to the invention, this system comprises a capture device and a processing device. The capture device is configured to capture one or more images of the door and / or of an opening intended to be closed by the door, and / or to capture one or more sounds emitted near the door. The processing device is configured to determine the position of the door based on the images and / or sounds thus captured.

[0008] The invention thus makes it possible to determine whether the doors are closed without resorting to door closure information provided by the control network.

[0009] It is therefore possible, in particular, to equip a train with an intelligent device requiring such information without modifying the control network.

[0010] More generally, the system of the invention can be integrated into an existing computer architecture either by creating a second dedicated network when this existing architecture only includes a control network or by integrating the processing device and, where applicable, the capture device into a comfort network already present in the existing architecture.

[0011] In one embodiment, the capture device includes one or more cameras and / or one or more microphones.

[0012] Preferably, the camera(s) and / or microphone(s) of the recording device form, or belong to, a video protection device.

[0013] In the context of a train modernization, this makes it possible in particular to use an existing video surveillance system, when it is present and adapted for such use.

[0014] Of course, the capture device can be a dedicated device, separate from any possible video surveillance device on board the vehicle.

[0015] In one embodiment, the processing device includes one or more intelligent models configured to identify in the images and / or sounds captured using the capture device one or more images and / or one or more sounds representative of a closed position of the door or a change in the position of the door.

[0016] Said change of position may be a current or future change.

[0017] The invention also relates to a vehicle, preferably a railway vehicle, comprising at least one door and a system as defined above.

[0018] In one embodiment, the vehicle includes at least one snorer, the processing device being configured to detect an activation state of the snorer on the basis of one or more sounds captured using the capture device, for example using one or more microphones of this capture device.

[0019] In one embodiment, the processing device is configured to detect a position of the door or part of the door such as a bay window, on the basis of one or more images captured using the capture device, for example using one or more cameras of this capture device.

[0020] In one embodiment, the vehicle includes one or more so-called critical equipment, one or more so-called non-critical equipment, a first computer network comprising one or more of said critical equipment and a second computer network comprising one or more of said non-critical equipment, the non-critical equipment comprising the capture device and the processing device.

[0021] In other words, the vehicle may include a computer architecture with two networks, the first network being a control network, the second network being a comfort network.

[0022] According to another aspect, the invention relates to a method for determining the position of at least one door of a vehicle, preferably a railway vehicle, using a system as defined above.

[0023] According to the invention, this method comprises: - capture by the capture device: - of one or more images of the door and / or an opening intended to be closed off by the door, and / or - one or more sounds emitted near the door, - a determination, by the processing device, of the position of the door based on the images and / or sounds thus captured.

[0024] In one embodiment, determining the position of the door includes searching, in sounds captured using the capture device, for one or more sounds, called characteristic sounds, representative of a closed position of the door or of a current or future change in the position of the door.

[0025] The method preferably includes a membership check of one or more characteristic sounds thus identified.

[0026] In one embodiment, this membership check includes a comparison of the characteristic sound(s) thus identified with one or more reference sounds.

[0027] This comparison can be carried out on the basis of at least one sound characteristic such as a sound level.

[0028] In one embodiment, the reference sound(s) are representative of an activation state of the aforementioned snorer.

[0029] In one embodiment, determining the position of the door includes searching, in images captured using the capture device, for one or more images, called characteristic images, representative of the door or a part of the door such as a bay window.

[0030] The method preferably includes checking the membership of one or more characteristic images thus identified.

[0031] This membership check may include a comparison of the characteristic image(s) with one or more reference images.

[0032] This comparison can in particular be carried out on the basis of at least one visual characteristic such as a dimension of the door or part of the door.

[0033] According to a first variant, the search for one or more characteristic images is carried out simultaneously with the search for one or more characteristic sounds.

[0034] According to a second variant, the search for one or more characteristic images is carried out on the condition that the search for one or more characteristic sounds has resulted in the identification of one or more characteristic sounds and that the membership check of one or more of these characteristic sounds has resulted in the validation of the membership of at least one of these characteristic sounds.

[0035] In the context of this second variant, it is preferred that the search for characteristic image(s) be initiated at the end of a period in which one or more characteristic sounds are detected, that is to say when the search for sound(s) no longer makes it possible to identify one or more characteristic sounds.

[0036] Other advantages and features of the invention will become apparent from the following detailed, non-limiting description. Brief description of the drawings

[0037] The detailed description that follows refers to the attached drawings in which:

[0038] [Fig-1] is a schematic illustration of an embedded computer architecture in a vehicle conforming to the invention;

[0039] [Fig.2] is a schematic illustration of steps in a process conforming to the invention. Detailed description of implementation methods

[0040] In the following description, the invention is implemented within a vehicle (not shown) of railway rolling stock carrying various equipment.

[0041] With reference to [Fig.1], these equipment include on the one hand critical equipment including a braking device 1, access doors 2 and a corresponding control device, buzzers 3 and a corresponding control device, and an event recording device 4 of the type “ATES” cassettes (for “Acquisition and Processing of Safety Events in Static”).

[0042] In a manner known per se, the buzzers 3 are configured to produce a buzzer warning passengers of the closing of the doors 2. In this non-limiting example, the buzzers 3 produce a buzzer starting three seconds before the closing of the doors 2 and ending when they are closed.

[0043] The vehicle also includes non-critical equipment including a video surveillance device 6, a passenger information device 7, an intelligent device 8 for counting passengers and a processing device 9 according to the invention.

[0044] The video protection device 6 in this example comprises six cameras (not shown) and a network recorder (not shown), known by the acronym "NVR" for "Network Video Recorder" in English.

[0045] By way of example, the cameras of device 6 are IP cameras connected to ports of a network switch and each equipped with a microphone. Each IP camera implements an application-layer communication protocol, known in English as "Real Time Streaming Protocol" (RTSP), which allows the transmission of video and audio streams over a packet-switched local area network protocol, such as "Ethernet IP". In this example, the network recorder uses "Gigabit Ethernet" technology to retrieve the video streams from the IP cameras and record them onto an internal storage module.

[0046] In this example, the cameras of the video surveillance system 6 are distributed in the vehicle on vehicle access platforms so as to film in direct view and from the front all of the access doors 2 and in particular the glazed openings of these doors 2.

[0047] The microphones integrated into the cameras are therefore positioned near the access doors 2, which allows them in particular to capture the sounds of the snorers 3 which are positioned at the level of the doors 2.

[0048] The video surveillance system 6 thus forms a recording device. The cameras allow, in particular, the recording of images of the doors 2, or of openings intended to be closed by these doors 2. The microphones equipping these cameras allow as for them to capture sounds produced by snorers 3, which in this example are representative of a change, current or future, in the position of doors 2.

[0049] In the example of [Fig.1], the various equipment 1-4 and 6-9 on board the vehicle are interconnected so as to form a computer architecture with two networks 11 and 12 which communicate with each other via a gateway 13 to achieve unicast communication.

[0050] In a manner known per se, the network 11 is a control network comprising critical equipment 1-4 and means 14 for interconnecting this critical equipment, while the network 12 is a so-called comfort network comprising non-critical equipment 6-9 and means 15 for interconnecting this non-critical equipment.

[0051] The invention relates more specifically to the processing device 9 which in this example forms an audiovisual analysis device.

[0052] The processing device 9 in this example includes an on-board railway computer of the PC type designed in a compact format suitable for the railway sector and conforming to railway standards.

[0053] The computer implements a Linux operating system of the "Ubuntu" type and includes software called "RTSP client" allowing the retrieval of audio and video streams from cameras and microphones using the RTSP protocol.

[0054] The capture device 6 is here digital and is connected on the same support network as the computer in order to be able to transmit the captured audio and video streams to it.

[0055] The processing device 9 includes algorithms, or intelligent models, configured to identify in the images and sounds captured using the device 6 visual and / or sound features corresponding to a closed position of the doors 2 or to a change in position of the doors 2.

[0056] More specifically, the device 9 includes, on the one hand, a sound detection algorithm trained on a large quantity of snoring sounds in order to be able to efficiently detect this type of sound occurrence. The algorithm used in this example is a known "YamNet" algorithm, trained according to state-of-the-art machine learning principles. The algorithm thus uses as input the sounds from the microphones of the recording device 6 or from similar microphones, enabling it to detect the sound of snorers 3.

[0057] The processing device 9 also includes a visual object detection algorithm trained on a large number of images of access door windows in order to efficiently detect this type of visual occurrence. The algorithm used in this example is a known "EfficientDet" algorithm, trained according to state-of-the-art machine learning principles.

[0058] In this example, the processing device 9 is activated upon receipt of information indicating the imminent start of a commercial mission. Alternatively, the processing device 9 can be activated when the rolling stock's computer systems are powered on, i.e., as soon as it is connected to a power supply.

[0059] An example of a method implementing the system described above, in order to determine the position of access doors 2, will now be described with reference to [Fig.2].

[0060] The steps described below illustrate a software sequence carried out by the processing device 9 which is not limiting.

[0061] In this scenario, the access doors 2 are initially in an open position, allowing passengers to disembark at a terminus station.

[0062] The capture device 6 simultaneously captures images of the vehicle openings which will be closed by the doors 2 when they are in a closed position, as well as sounds occurring in the environment of the microphones of this device 6 and therefore near the doors 2.

[0063] The processing device 9 thus acquires audio and visual data streams provided by the capture device 6.

[0064] In this example, a search step El is first carried out using the sound detection algorithm to identify in the audio data stream sound patterns characteristic of the snoring of the snorers 3. In other words, the search El is intended to identify characteristic sounds, which are representative of an activation state of the snorers 3 and as a result of an imminent or actual closing of the doors 2.

[0065] In this example, the first three seconds of ringing of the snorers 3, during which characteristic sounds can thus be identified, indicate the imminent command to close the doors 2. The following period, until the end of which characteristic sounds can still be detected, corresponds to an effective command to close the doors 2 resulting in a change of position of the doors 2. At the end of this period, characteristic sounds are no longer produced by the snorers 3 which cease to ring, indicating that the doors 2 are closed.

[0066] To avoid false positives, a verification procedure E2 for the characteristic sounds identified during the El search is implemented to ensure that the captured snoring sounds actually correspond to a snorer from the vehicle. Indeed, in railway operations, two trains may be located close to each other when stopped at a station. In this situation, the sound of a snorer from another vehicle may be picked up by the device 6 and thus cause a false positive.

[0067] The membership check procedure E2 can be performed using a discrimination method based on known physical properties, in this case, sound characteristics. In this non-limiting example, the discrimination is based on sound level. The membership check E2 is performed here by comparing the sound level of the characteristic sounds identified during the search E1 with a reference sound level. In a manner known per se, this reference sound level can be loaded from a text file stored in the processing device 9.

[0068] When the sound level of a given characteristic sound is identical or close to the reference sound level, membership is validated.

[0069] Such a membership check E2 makes it possible to exclude sounds identified as characteristic but not originating from the vehicle's snorers 3. In particular, if a characteristic sound identified during the search El has a sound level lower than a reference level, it is excluded because it is considered to originate from a distant sound source other than a snorer 3 of the vehicle, for example, a snorer from a neighboring train.

[0070] In this example, the next step E3 is triggered when the sound detection algorithm no longer detects characteristic sounds, i.e. in principle when the doors 2 are in the closed position.

[0071] Step E3 is a search step carried out using the visual detection algorithm to identify in the visual data stream images characteristic of the glass panes of the access doors 2. Indeed, when the doors 2 are in the closed position, the glass panes of these doors 2 are located in the field of view of the cameras of the device 6.

[0072] To avoid false positives, a membership check procedure E4 is implemented for the content of the characteristic images identified during the search E3 to ensure that the patterns identified as windows correspond to the windows of the vehicle's doors 2. Indeed, in the situation described above of two trains located close to each other during a station stop, the presence of a window on a door of another vehicle could be captured by a camera of the device 6 and thus cause a false positive.

[0073] The membership check procedure E4 can be performed using a discrimination method based on known physical properties, in this case visual characteristics. In this non-limiting example, the discrimination is based on bay dimensions, for example, a bay width. The membership check E4 is performed here by comparing the width of an object, considered to correspond to a bay, with a reference width in characteristic images identified during the search E3. In a manner known per se, this Reference width can be loaded from a text file stored in the processing device 9.

[0074] When the width of an object, considered to correspond to a bay, on a given characteristic image is identical or close to the reference width, the membership is validated.

[0075] Such a membership check E4 makes it possible to exclude images considered characteristic but not corresponding to images of a door 2 or part of a door 2 of the vehicle. In particular, if a characteristic image identified during the search E3 includes an object, considered as a window, having a width less than a reference width, this characteristic image is excluded, as the window identified as such in this image is likely to belong to the door of another vehicle.

[0076] When membership E4 is validated for at least one of the characteristic images identified during the search E3, it is considered that the access doors 2 are in the closed position.

[0077] In this case, the processing device 9 performs a step E5 of disseminating information on the computer network 12 about the closing of doors 2. This dissemination can be carried out in unicast mode or in broadcast mode, in particular towards the intelligent device 8.

[0078] Numerous variations can be made to the preceding description. In particular, the capture can be carried out using a device that is not a video surveillance system, for example, using one or more cameras and / or one or more microphones dedicated to determining the position of doors 2. More generally, the capture device may comprise only one or more cameras or only one or more microphones, so as to provide the processing device 9 with a data stream of either video or audio. When the capture device comprises one or more microphones, these can be configured to capture sounds according to directionality parameters chosen to reduce the capture of sounds from unwanted sound sources.The system can also be configured to acquire visual data from a part of a door other than a glass panel, for example, a marker on a door, and / or audio data other than a snorer's ringing, for example, the sound of a door sliding. As another example, one or more cameras of the capture device can be arranged to film doors 2 from a partial view and / or from an inclined angle.

[0079] In one embodiment, the capture device may be analog. For example, microphones of this capture device may be physically connected to the processing device 9 via physical inputs of the jack type.

[0080] In non-limiting embodiments, the processing device 9 may include a computer such as a mini-computer of the "Raspberry Pi" type, or a programmable electronic board of the FPGA or ASIC type.

[0081] The image and / or sound processing described above can also be carried out in a different way. For example, steps E1 and E3 can be carried out simultaneously using a single multimodal detection algorithm such as the one known as "data2vec".

[0082] Furthermore, other algorithms than those indicated above may be used. For example, the processing device 9 may include for sound detection an algorithm such as that known as "CRNN", or "YamNet", or "VGGish", or even "SoundNet" and / or for visual detection an algorithm such as that known as "YoloV3", or "YoloV4", or "YoloV5", or "EfficientDet", or even "VisionTransformer".

[0083] In an embodiment in which the processing includes a search for images corresponding to a door closed position, such as step E3, this processing may be devoid of the membership check procedure E4 described above. In this case, the processing device may be configured to detect additional information on the captured images, for example, a vehicle element such as a door opening control button.

[0084] In the preceding description, membership checks E2 and E4 implement a deterministic comparison of parameters such as a sound level or a window width with expected values. Alternatively or complementarily, a non-deterministic membership check can be implemented, for example by machine learning.

Claims

Demands

1. An assembly comprising a passenger counting device and a system for determining the closing position of at least one door (2) of a vehicle, preferably a railway vehicle, characterized in that the system comprises: • a capture device (6): • of one or more images of the door (2) and / or of an opening intended to be closed by the door (2), and • of one or more sounds emitted near the door (2), • a processing device (9) configured to determine a closing position of the door (2) based on the images and sounds thus captured and to transmit a door (2) closure information to the passenger counting device.

2. Assembly according to claim 1, wherein the capture device (6) comprises one or more cameras and / or one or more microphones which preferably form a video protection device.

3. Assembly according to claim 1 or 2, wherein the processing device (9) comprises one or more intelligent models configured to identify in the images and / or sounds captured using the capture device (6) one or more images and / or one or more sounds representative of a closed position of the door (2) or of a change in position of the door (2).

4. Vehicle, preferably railway, comprising at least one door (2) and an assembly according to any one of claims 1 to 3.

5. Vehicle according to claim 4, comprising one or more so-called critical equipment (1-4), one or more so-called non-critical equipment (6-9), a first computer network (11) comprising one or more of said critical equipment (1-4) and a second computer network (12) comprising one or more of said non-critical equipment (6-9), the non-critical equipment comprising the capture device (6), the passenger counting device and the processing device (9).

6. A method for determining the closed position of at least one door (2) of a vehicle, preferably a railway vehicle, using an assembly according to any one of claims 1 to 3, comprising: • a capture by the capture device (6): • of one or more images of the door (2) and / or of an opening intended to be closed by the door (2), and • of one or more sounds emitted near the door (2), • a determination, by the processing device (9), of a closing position for the door (2) based on the images and sounds thus captured, • a transmission to the passenger counting device of information indicating that the door is closed (2).

7. A method according to claim 6, wherein the determination of the closed position of the door (2) comprises: • a search (El), in sounds captured using the capture device (6), for one or more sounds, called characteristic sounds, representative of the closed position of the door (2) or of a current or future change in the position of the door (2) and • a membership check (E2) of one or more characteristic sounds thus identified, this membership check (E2) comprising a comparison of this or these characteristic sounds with one or more reference sounds, this comparison being carried out on the basis of at least one sound characteristic such as a level sound.

8. A method according to claim 6 or 7, wherein the determination of the closing position of the door (2) comprises: • a search (E3), in images captured using the capture device (6), for one or more images, called characteristic images, representative of the door (2) or of a part of the door (2) such as a window, and • a membership check (E4) of one or more characteristic images thus identified, this membership check (E4) comprising a comparison of this or these characteristic images with one or more reference images, this comparison being carried out on the basis of at least one visual characteristic such as a dimension of the door (2) or part of the door (2).

9. A method according to claim 8 including the features of claim 7, wherein the search (E3) for one or more characteristic images is carried out either simultaneously with the search (El) for one or more characteristic sounds or provided that the search (El) for one or more characteristic sounds has resulted in the identification of one or more characteristic sounds and that the membership check (E2) of one or more of these characteristic sounds has resulted in the validation of the membership of at least one of these characteristic sounds.