Improved monitoring a data link in a rail vehicle

Abstract

The invention provides, amongst other aspects, a method of monitoring a data link in a rail vehicle comprising at least one image apparatus and a display apparatus, the method comprising monitoring the data link between the at least one image apparatus and the display apparatus over a plurality of wireless networks, wherein the data link comprises a transmission of image frames captured with the at least one image apparatus, wherein the transmission of image frames is replicated over the wireless networks, and generating an alert upon detecting that a time interval between consecutive image frames over each or any of the wireless networks exceeds a first predetermined threshold relating to an acceptable delay. The invention further provides a system comprising at least one image apparatus, a display apparatus wireless connected to the at least one image apparatus and a device configured to perform the method steps.

Description

Field of the invention

[0001] The present invention relates to the technical domain of rail transportation and more particularly to techniques for monitoring data links in rail vehicles.Background art

[0002] Rail vehicles are essential modes of transport for goods and people over long distances as well as in local transportation networks. In shunting operations, a rail vehicle is maneuvered in a confined area or reversed direction to couple or decouple other railway cars. This process typically requires the assistance of one or more personnel on or near the track to communicate instructions and coordinate movements. Shunting operations in particular can be time-consuming and dangerous for the personnel involved, as they require close proximity to moving trains and a high level of attention from the rail driver.

[0003] The current state of the art in rail vehicle conducting and / or shunting operations relies heavily on human intervention. A rail vehicle for transporting goods and / or people is typically operated by a rail driver who must manage several functions, including controlling the speed of the rail vehicle, detecting track signs, and ensuring safe movement of the train. However, research shows that 75% of rail incidents are caused by human error or poor judgment. Common causes of such incidents include excessive speeding, lack of attention, violation of traffic rules, and very slow reaction speed in the event of an approaching collision. These risks are particularly prevalent during shunting operations due to the proximity of personnel to moving trains and the reverse direction of travel.

[0004] Communication between personnel and the rail vehicle driver is often limited to verbal instructions and / or hand signals, which can be misinterpreted or lost in noisy environments. Additionally, manual shunting operations require a significant amount of time and personnel resources, making such operations inefficient.

[0005] These disadvantages necessitate a system that can improve rail vehicle conducting and / or shunting operations. It is therefore an objective of the present invention to provide a rail vehicle monitoring system capable of enhancing the safety and efficiency of rail vehicle conducting and / or shunting operations, ultimately benefiting both the rail industry and the general public.Summary of the invention

[0006] One aspect of the present invention relates to a method of monitoring a data link in a rail vehicle comprising at least one image apparatus and a display apparatus. The data link between the at least one image apparatus and the display apparatus allows for real-time monitoring and optionally control of rail vehicle operations, including conducting and shunting.

[0007] It may be provided that this method comprises monitoring the data link over a plurality of wireless networks. These networks can be any type of wireless communication systems capable of transmitting image frames captured by the image apparatus to the display apparatus. By replicating the transmission of image frames over multiple networks, data reception by the display apparatus is improved. This redundancy ensures reliable data transfer and minimizes interruptions in the data link, which is critical during rail vehicle operations.

[0008] One advantage of monitoring the data link over a plurality of wireless networks is the enhanced reliability of the communication pathway. In case one network experiences challenges or becomes unavailable, the transmission can continue through other available networks, thereby minimizing interruptions and ensuring consistent, real-time image frame transfer. This is particularly important during critical rail vehicle operations such as conducting and shunting where precise communication between the at least one image apparatus and the display apparatus is necessary for safety and efficiency.

[0009] Another advantage of monitoring the data link between the image apparatus and the display apparatus over multiple wireless networks is improved system resilience. By distributing the communication load across various networks, the overall system becomes more robust to network failures or disruptions. This ensures that the rail vehicle's onboard systems remain operational and provide accurate real-time information for the driver, even when one or more of the wireless networks are compromised. This is particularly advantageous in harsh environments, where any one or combination of the following would typically affect communication: low coverage, low bandwidth, few connectivity hardware on site, interference with buildings and / or poles and / or machines, and Electromagnetic Compatibility (EMC).

[0010] Furthermore, it may be provided that this method comprises generating an alert upon detecting that a time interval between consecutive image frames over each or any of the wireless networks exceeds a first predetermined threshold relating to an acceptable delay. This threshold is based on a desired response time for the system and should not significantly impact the overall throughput or bandwidth requirements.

[0011] The advantage of generating an alert upon detecting a time interval between consecutive image frames that exceeds a predetermined threshold is the ability to quickly identify potential issues with the data link and take corrective measures. For instance, this could indicate network congestion, equipment failure, or other problems that may require intervention from maintenance personnel or system administrators. By addressing these issues promptly, the overall performance of the rail vehicle's onboard systems can be maintained, ensuring efficient operation and safety.

[0012] Thus, the present invention will not only improve safety but also enhance efficiency by reducing the need for manual intervention. This reduction in labor requirements can lead to cost savings for railway companies and increased productivity. Additionally, the system's advanced features, such as real-time and improved information communication, will help reduce human error and miscommunication, further increasing overall safety.

[0013] In particularly advantageous embodiments, any or each of the at least one image apparatus comprises at least one image sensor, preferably the at least one image sensor comprises a plurality of image sensors. In preferred embodiments, the transmission of image frames comprises stitching the image frames from the image sensors at each time stamp or the images frames from the image sensors are stitched at each time stamp after transmission.

[0014] In particularly advantageous embodiments, the rail vehicle is being used in a shunting operation. In shunting operations, a rail vehicle is typically operated in the reverse direction of travel, where a rail vehicle driver requires assistance from personnel on or near the rail track of an industrial site (e.g., rail site, railway, railroad, railyard, etc.). Thus, an advantage of the present invention is to allow for improved communication of information to the rail driver, and thereby reducing or eliminating the need for personnel on or near the track of the site.

[0015] According to a second aspect, the present invention provides a device comprising means for carrying out the method according to the invention.

[0016] According to a third aspect, the present invention provides a system comprising the device according to the invention, at least one image apparatus; and a display apparatus wirelessly connected to the at least one image apparatus, wherein the device is configured to perform the method steps described above.

[0017] Such a system may advantageously allow for improved integration with a rail vehicle. In addition to the advantages provided by the method, as described above, the system may accommodate additional apparatuses, such as an additional sensor, a horn / buzzer, a microphone, a speaker, etc., and thereby, allowing for a modular system. Furthermore, such a system allows for integration with several types of rail vehicles, which is particularly advantageous for older rail vehicles.

[0018] According to a fourth aspect, the present invention provides a computer program comprising instructions which, when the program is executed by a computer, cause the computer to carry out the method according to the invention. The computer program product may comprise at least one readable medium in which computer-readable program code portions are saved, which program code portions comprise instructions for carrying out said method.

[0019] According to a fifth aspect, the present invention provides a computer-readable medium comprising instructions which, when executed by a computer, cause the computer to carry out the method according to the invention.

[0020] According to a further aspect, the present invention provides a rail vehicle comprising the system according to the present invention.

[0021] Preferred embodiments and their advantages are provided in the description and the dependent claims.Brief description of the drawings

[0022] The present invention will be discussed in more detail below, with reference to the attached drawings. Fig. 1 shows a first example of a method and / or system according to the invention. Fig. 2A shows a second example of a method and / or system according to the invention. Fig. 2B shows a third example of a method and / or system according to the present invention. Description of embodiments

[0023] The following descriptions depict only example embodiments and are not considered limiting in scope. Any reference herein to the disclosure is not intended to restrict or limit the disclosure to exact features of any one or more of the exemplary embodiments disclosed in the present specification.

[0024] Furthermore, the terms first, second, third and the like in the description and in the claims are used for distinguishing between similar elements and not necessarily for describing a sequential or chronological order. The terms are interchangeable under appropriate circumstances and the embodiments of the invention can operate in other sequences than described or illustrated herein.

[0025] Furthermore, the various embodiments, although referred to as "preferred" are to be construed as exemplary manners in which the invention may be implemented rather than as limiting the scope of the invention.

[0026] The term "comprising", used in the claims, should not be interpreted as being restricted to the elements or steps listed thereafter; it does not exclude other elements or steps. It needs to be interpreted as specifying the presence of the stated features, integers, steps or components as referred to, but does not preclude the presence or addition of one or more other features, integers, steps or components, or groups thereof. Thus, the scope of the expression "a device comprising A and B" should not be limited to devices consisting only of components A and B, rather with respect to the present invention, the only enumerated components of the device are A and B, and further the claim should be interpreted as including equivalents of those components.

[0027] In this document, the term "data link" refers to a communication pathway between components within a system, such as an image apparatus and a display apparatus and / or an additional apparatus and the display apparatus, according to the present invention. Thus, the data link may comprise the communication of sensor data (e.g., image frames and / or data from any additional sensor), preferably further comprising at least one of the following communication data: time stamps, control signals, sounds, text, binary data, structured data (e.g., Extensible Markup Language (XML), and JavaScript Object Notation (JSON)). In this context, a rail vehicle refers to a self-propelled vehicle designed for transportation on rails, such as a (freight and / or passenger) train, commuter railcar, monorail, Maglev train, tram, metro / subway, hyperloop pod, etc. In this context, the term image apparatus refers to an electronic device which comprises an image sensor that captures data relating to images or moving pictures, and a display apparatus refers to an electronic device comprising a display that shows / displays visual information.

[0028] In embodiments, the method of monitoring a data link in a rail vehicle comprising at least one image apparatus and a display apparatus, wherein the device is configured to perform the steps of: 1.a. monitoring the data link between the at least one image apparatus and the display apparatus over a plurality of wireless networks, wherein the data link comprises a transmission of image frames captured with the at least one image apparatus, wherein the transmission of image frames is replicated over the wireless networks, and 1.b. generating an alert upon detecting that a time interval between consecutive image frames over each or any of the wireless networks exceeds a first predetermined threshold relating to an acceptable delay.

[0029] In embodiments, the wireless networks comprise any one or combination of: one or more cellular networks, one or more Wi-Fi networks, one or more satellite networks, one or more Bluetooth networks, one or more Zigbee networks, and one or more industrial networks. The one or more cellular networks may be from the group of: 4G, 5G, Time Division-Spectrum Code Division Multiple Access (TD-SCDMA), Wideband Code Division Multiple Access (WCDMA), Long-Term Evolution (LTE), 5G New Radio (NR), Stand Alone (SA), Non-SA, 5G SA, and 5G NSA. The one or more cellular network may involve at least one base station (e.g., a tower cell or a node), which acts as a gateway and provides coverage. The one or more Wi-Fi networks may be from the group of: 802.11g, 802.11n, 802.11ac, and 802.11ad, 802.11 ax. The one or more satellite networks may be from the group of: Starlink, OneWeb, LeoSat and Telesat. The one of more satellite networks may involve at least one satellite, preferably a plurality of satellites, which act(s) as a gateway and provide(s) coverage. The one or more Bluetooth networks may be from the group of: Bluetooth 3.0, Bluetooth 4.0, Bluetooth 5.0, Bluetooth Smart, Bluetooth Mesh, and Bluetooth Classic. The one or more Zigbee networks may be from the group of: Zigbee 3.0, Zigbee RF4CE, Zigbee PRO, Zigbee Green Power (ZGP), Zigbee Industrial Automation (ZIA). It will be understood that newer wireless network protocols and / or standards may also be used.

[0030] In particularly advantageous embodiments, the wireless networks comprise the one or more cellular networks comprising a private 5G infrastructure utilizing dedicated industrial frequencies. This can ensure a stable and secure connection for communicating data, such as sensor data and / or communication data. The one or more cellular network further comprises one or more public 4G / 5G networks. This can ensure comprehensive coverage across an industrial site.

[0031] In particularly advantageous embodiments, the wireless networks comprise the one or more Wi-Fi networks comprising a Wi-Fi Mesh Network. Here, the system may comprise at least one router, such as a wireless router (e.g., mesh routers and / or mesh nodes, Wi-Fi access points, etc.) and / or a wired router (e.g., backhaul connections). For instance, in a wired backhaul connection, each mesh node can be connected via wired Ethernet to another node or the main router to ensure a stable and high-speed connection between nodes. This is especially advantageous when the wireless signal strength between two nodes is weak.

[0032] In particularly advantageous embodiments, the wireless networks comprise the one or more satellite networks comprising at least one satellite-Wi-Fi connector, such as a Starlink Wi-Fi pole, for connecting with the at least one satellite. The at least one satellite-Wi-Fi connector may be configured for connecting with the at least one image apparatus and with the display apparatus. Thus, the at least one satellite-Wi-Fi connector may provide reliable satellite connectivity where terrestrial network coverage (e.g., the one or more cellular networks) is inadequate, thereby ensuring the at least one image apparatus continue to transmit data essential for safe and efficient train shunting and / or conducting operation.

[0033] In particularly advantageous embodiments, the wireless networks comprise the one or more cellular networks comprising at least one repeater (e.g., 4G / 5G outdoor repeater) for extending the cellular connection between the at least one image apparatus and the display apparatus. This can amplify weak 4G / 5G signals in specific areas of the site with insufficient coverage, bolstering the connectivity required for image frame streaming (e.g., live video) and / or sensor data and / or communication data transmission.

[0034] In particularly advantageous embodiments, the wireless networks comprise the one or more cellular networks comprising at least one central network infrastructure component (e.g., a 5G core) for connecting and managing various apparatuses (i.e., the at least one image apparatus and / or the display apparatus) and / or user (e.g., personnel, such as on site via a mobile device or in a command center via a receiving device). Said at least one central network infrastructure component may be installed on the rail vehicle, emitting a private network that extends around the rail vehicle, covering the entire rail vehicle. Alternatively, said at least one central network infrastructure component may be installed on the industrial site, emitting a private network that extends around the rail vehicle, covering gaps in the public 4G / 5G network. These private networks can provide a secure and dedicated channel for the at least one image apparatus, ensuring consistent and reliable connectivity for the system's operations.

[0035] In embodiments, the data link comprises a communication protocol for transmitting the image frames and providing VPN encryption. The communication protocol may comprise establishing a secure tunnel between the at least one image apparatus and the display apparatus using any one or combination of cryptographic techniques, such as SSL / TLS encryption and IPsec authentication. This allows to reduce the overhead associated with encrypting and decrypting data, resulting in faster transmission speeds and lower latency. The communication protocol may comprise any one or combination of the wireless network protocol described above. The afore-mentioned wireless networks may be configured or integrated in the communication protocol comprised in the data link. This can allow for a unified and secure network infrastructure that provides reliable connectivity and high levels of performance.

[0036] In embodiments, the at least one image sensor is any one or combination from the group of: a monocular camera, an infrared camera, a thermal imaging sensor, a low-light imaging sensor, a radar sensor, a light detection and ranging (LiDAR) sensor, a time-of-flight (ToF) camera, and a 3D camera.

[0037] In embodiments, any of the at least one image sensor has a low-light vision mode and / or a night vision mode for capturing image frames in low-light environments and / or during nighttime. Additionally or alternatively, any of the at least one image sensor is equipped with at least one image processing algorithm for enhancing the quality of the images captured in low-light environments.

[0038] In embodiments, the image frames are processed by the at least one image processing algorithm before transmitting (e.g., by the at least one image sensor and / or an external processor) and / or after transmitting (e.g., by the display apparatus and / or the external processor). The at least one image processing algorithm may be any one or combination from the group of: contrast enhancement, brightness enhancement, noise reduction, average color filtering, histogram equalization, Gamma correction, white balance adjustment, sharpening, unsharp masking, edge detection, thresholding, watershed transformation, morphological operations, etc. Additionally or alternatively, the at least one image processing algorithm may relate to object detection and / or classification in any image frame, and / or be any one or combination from the group of: Region-based Convolutional Neural Network (R-CNN), Single Shot Detector (SSD), RetinaNet, DeepLab, U-Net, etc.

[0039] In embodiments, the at least one image sensor comprises a plurality of image sensors. The plurality of image sensors may be any one or combination from the group of: one or more monocular cameras, one or more infrared cameras, one or more radar sensors, one or more LiDAR sensors, one or more laser cameras, one or more ToF cameras, and one or more 3D cameras.

[0040] In embodiments, any or each of the at least one image apparatus comprises at least one additional sensor and / or at least one light source (e.g., LED) and / or at least one horn / buzzer and / or at least one microphone and / or at least one speaker. The at least one additional sensor may comprise any one or combination of: at least one temperature sensor, at least one acceleration sensor, at least one humidity sensor, at least one ultrasonic sensor, and at least one proximity sensor. In this context, the sensor data may comprise any one or combination of temperature data, acceleration data, humidity data, ultrasonic wave data (or distance data determined based on the ultrasonic wave data), and proximity data.

[0041] In embodiments, the at least one image apparatus comprises a plurality of image apparatuses. The plurality of image apparatuses may comprise no similar image sensors, at least one similar image sensor, a plurality of similar image sensors or only similar image sensors. In further embodiments, any or each of the plurality of image apparatuses comprises at least one of: at least one additional sensor, at least one light source (e.g., LED), at least one horn / buzzer, at least one microphone, and at least one speaker.

[0042] In embodiments, the plurality of image sensors has at least partially overlapping field-of-views (FoVs). Alternatively, the plurality of image sensors has end-to-end FoVs. It is preferred that the total or combined FoV of the plurality of image sensors has an angle of at least 120 degrees, preferably at least 150 degrees, more preferably at least 180 degrees, most preferably 210 degrees. At each time point (e.g., time stamp), the image frames captured by said image sensors may be merged. Said merging of the image frames may comprise any one or combination of: Kalman filter-based registration, Random sample consensus (RSC), Mutual information-based registration, Feature-based alignment, Deep learning-based registration, Iterative closest point (ICP), and other optimization algorithms (e.g., Gradient descent-based registration, Non-linear least squares registration, etc.). The Kalman filter-based registration may relate to uses a Kalman filter to estimate the camera poses (position and orientation) and then registers the images using an affine transformation. The RSC may relate to using a random sample of pixels from each image frame and computes the similarity between the samples. The mutual information-based registration may relate to using mutual information to measure the similarity between the images and then registers the images using a maximum likelihood estimation. The RSC and the mutual information-based registration result in the final merged image being based on the most similar pixel pairs. The feature-based alignment may relate to This method aligns the images by matching features such as corners, edges, or blobs. Once the images are aligned, they can be combined using an averaging technique. The deep learning-based registration may relate to mapping the images with their corresponding registers using deep learning. The ICP may relate to alternating between finding the best registration transformation and verifying that the resulting image is close to the original image.

[0043] In embodiments, said merging of the image frames is performed before transmitting by the at least one image apparatus (e.g., a processor comprised therein) or after transmitting the image frames, i.e., after receiving the image frames by the display apparatus (e.g., a processor comprised in the display apparatus).

[0044] In embodiments, the first predetermined threshold relating to an acceptable time interval between consecutive image frames is expressed in milliseconds (ms) and may be any one of: 700 ms, 600 ms, 500 ms, 400 ms, 300 ms, etc. For example, step 1.b may comprise generating the alert upon detecting that the time interval between consecutive image frames is less than or equal to 700 ms, preferably less than or equal to 600 ms, more preferably less than or equal to 500 ms, most preferably less than or equal to 400 ms.

[0045] In embodiments, the transmission of the image frames comprises the transmission of data packets of the image frames, wherein step 1.b further comprises generating the alert: 2.a. upon detecting that a round-trip time of a data packet from among the data packets over each or any of the wireless networks a second predetermined threshold relating to an acceptable round-trip delay, and / or 2.b. upon detecting that a loss of data packets over each or any of the wireless networks exceeds a third predetermined threshold relating to an acceptable degree of pixelation.

[0046] Steps 2.a-2.b may relate to delay and / or loss metrics. Examples thereof are described hereinafter.

[0047] In step 2.a, the round-trip time (RTT) may refer to a measure for evaluating the time it takes for a data packet to travel from a transmitter to a receiver and back, i.e., from the at least one image apparatus to a display apparatus and back. A lower RTT indicates faster communication between the at least one image apparatus and the display apparatus, and a better network performance. The RTT may be expressed in milliseconds (ms). The second predetermined threshold relating to an acceptable delay may be any one of: 200 ms, 300 ms, 400 ms. 500 ms, 600 ms, 700 ms, 800 ms, 900 ms, 1000 ms, etc. For example, step 1.b may comprise generating the alert upon detecting that the RTT is greater than or equal to 200 ms, preferably greater than or equal to 400 ms, more preferably greater than or equal to 600 ms, most preferably greater than or equal to 800 ms.

[0048] In step 2.b, the loss of data packets may refer to a Bit error rate (BER) which is a measure for quantifying the number of bits that are received in error relative to the total number of bits transmitted. The lower the BER, the better the performance of the communication system. The BER may be expressed in errors per bit (err / bit). The third predetermined threshold relating to an acceptable degree of pixelation may be any one of: 10 -12< , 10 -10< , 10 -9< , 10 -8< , 10 -7< , 10 -6< , 10 -5< , 10 -4< , 10 -3< , 10 -2< , and 10 -1< . For example, step 1.b may comprise generating the alert upon detecting that the BER is greater than or equal to 10 -6< , preferably greater than or equal to 10 -5< , more preferably greater than or equal to 10 -4< , most preferably greater than or equal to 10 -3< .

[0049] In embodiments, step 1.b further comprises generating the alert: 3.a. upon detecting that a received signal strength exceeds a fourth predetermined threshold relating to an acceptable power level the display apparatus is receiving over each or any of the wireless networks, and / or 3.b. upon detecting that a reference signal received power exceeds a fifth predetermined threshold relating to an acceptable power level of the reference signal in each or any of the wireless networks, and / or 3.c. upon detecting that a reference signal received quality exceeds a sixth predetermined threshold relating to an acceptable quality level of the reference signal in each or any of the wireless networks, and / or 3.d. upon detecting that a signal-to-interference and / or noise-ratio exceeds a seventh predetermined threshold relating to an acceptable power level of the signal to the power level of background interference and / or noise in each or any of the wireless networks.

[0050] Steps 3.a-3.d may relate to connectivity and / or modem metrics. Examples thereof are described hereinafter.

[0051] In step 3.a, the received signal strength may refer to a received signal strength indicator (RSSI) which is a metric that measures the power level a device (in our case, the router) is receiving from the cell tower. A higher RSSI value indicates a stronger signal, leading to better connection quality. The RSSI may be expressed in decibels (dBm). The fourth predetermined threshold relating to an acceptable power level the display apparatus is receiving over each or any of the wireless networks may be any one of: -65 dBm, -70 dBm, -75 dBm, -80 dBm, -85 dBm, -90 dBm, and -95 dBm. For example, step 1.b may comprise generating the alert upon detecting that the RSSI is less than or equal to -65 dBm, preferably less than or equal to -75 dBm, more preferably less than or equal to -85 dBm, most preferably less than or equal to -90 dBm.

[0052] In step 3.b, the reference signal received power (RSRP) may refer to a measure of the power level of the Reference Signal in a cellular network. The RSRP may provide a deeper understanding of the actual signal strength at any given location. A higher RSRP signifies a more stable connection. The fifth predetermined threshold relating to an acceptable power level of the reference signal in each or any of the wireless networks may be any one of: -80 dBm, -85 dBm, -90 dBm, -95 dBm, and -100 dBm. For example, step 1.b may comprise generating the alert upon detecting that the RSRP is less than or equal to -80 dBm, preferably less than or equal to -85 dBm, more preferably less than or equal to -90 dBm, most preferably less than or equal to -95 dBm.

[0053] In step 3.c, the reference signal received quality (RSRQ) may refer to a measure of the quality of the received signal. It's a ratio of the RSRP to the interference and noise. The RSRQ may provide a deeper understanding of the actual signal strength at any given location. A higher RSRQ signifies a cleaner signal with less interference. The sixth predetermined threshold relating to an acceptable quality level of the reference signal in each or any of the wireless networks may be any one of: -10 dBm, -12 dBm, -15 dBm, -17 dBm, and -20 dBm. For example, step 1.b may comprise generating the alert upon detecting that the RSRQ is less than or equal to -10 dBm, preferably less than or equal to -12 dBm, more preferably less than or equal to -15 dBm, most preferably less than or equal to -17 dBm.

[0054] In step 3.d, the signal-to-interference and / or noise-ratio (SIR, SNR, SINR) may refer to a measure of the quality of the cellular signal by comparing the level of the signal (e.g., LTE signal) to the level of background noise and / or interference. Higher SIR and / or SNR and / or SINR indicate(s) a clearer channel for data transmission, resulting in better throughput and data rates. The seventh predetermined threshold relating to an acceptable power level of the signal to the power level of the background interference and / or noise in each or any of the wireless networks may be any one of: 20 dBm, 17 dBm, 13 dBm, 10 dBm, 7 dBm, 3 dBm and 0 dBm. For example, step 1.b may comprise generating the alert upon detecting that the SIR and / or SNR and / or SINR is / are less than or equal to 17 dBm, preferably less than or equal to 13 dBm, more preferably less than or equal to 10 dBm, most preferably less than or equal to 7 dBm.

[0055] In embodiments, step 1.b further comprises generating the alert: 4.a upon detecting that a carrier-to-noise ratio exceeds an eighth predetermined threshold relating to an acceptable power level of the carrier to the power level of background noise in each or any of the wireless networks.

[0056] In step 4.a, the carrier-to-noise-ratio (CNR) may refer to a measure of the strength of the carrier signal by comparing the level of the carrier signal to the level of background noise at the carrier frequency. Higher CNR indicates a stronger data transmission, resulting in better throughput and data rates. This may be particularly useful for satellite communication where atmospheric conditions and interferences can affect the quality of the received signal carrier. The eighth predetermined threshold relating to an acceptable power level of the carrier signal to the level of background noise at the carrier frequency in each or any of the wireless networks may be any one of: 50 dBm, 45 dBm, 40 dBm, 35 dBm, 30 dBm, 25 dBm, 20 dBm, 15 dBm, and 10 dBm. For example, step 1.b may comprise generating the alert upon detecting that the CNR is less than or equal to 30 dBm, preferably less than or equal to 25 dBm, more preferably less than or equal to 20 dBm, most preferably less than or equal to 10 dBm.

[0057] In embodiments, the method comprises displaying a blank screen on the display apparatus, preferably on the display comprised therein, upon generating the alert. In embodiments, the alert comprises an alert message, wherein preferably the alert message is displayed on the blank screen. For example, the alert message is displayed as "Delay is too high" or "No image".

[0058] In embodiments, the method comprises shutting down the at least one image apparatus. In preferred embodiments, the method comprises shutting down any of the at least one image apparatus, preferably the at least one image sensor comprised therein based on the detected time interval between consecutive image frames over the respective of the wireless networks exceeds the first predetermined threshold relating to an acceptable delay. For example, an image sensor is shut down based on the detection in any one or combination of steps 1.b, 2.a, 2.b, 3.a, 3.b, 3.c, 3.d and 4.a.

[0059] In embodiments, the method or step 1.b comprises resetting a real-time streaming protocol of the at least one image apparatus, preferably of the at least one image sensor comprised therein. In preferred embodiments, the method or step 1.b comprises resetting a real-time streaming protocol of any of the at least one image apparatus, preferably of the at least one image sensor comprised therein based on the detected time interval between consecutive image frames over the respective of the wireless networks exceeds the first predetermined threshold relating to an acceptable delay. For example, an image sensor and / or the streaming protocol of the image sensor is reset based on the detection in any one or combination of steps 1.b, 2.a, 2.b, 3.a, 3.b, 3.c, 3.d and 4.a.

[0060] In embodiments, the transmission of image frames comprises transmitting the respective image frame of the respective of the plurality of image apparatuses or the plurality of image sensors over any wireless network at each time stamp. Each image frame may include a respective time stamp. Thus, the image frames at each time stamp or with a similar time stamp (e.g., with a predefined margin, such as at most at most 90 ms difference, at most 75 ms difference, at most 60 ms difference, at most 45 ms difference, etc.), preferably with the same time stamp, may be transmitted over the same wireless networks. For example, a first image frame captured by a first image sensor, with a first time stamp (hours:minutes:seconds:milliseconds) of 05:01:10:15AM, and a second image frame captured by a second image sensor, with a second time stamp (hours:minutes:seconds:milliseconds) of 05:01:10:20 AM, are transmitted over each of the plurality of wireless networks at the same time.

[0061] In embodiments, the image frames are stitched at each time stamp or with a similar time stamp, preferably with the same time stamp, after transmission over the wireless networks. For example, the image frames at each time stamp or with a similar time stamp, preferably with the same time stamp, are stitched at the display apparatus. Thus, the image frames at each time stamp or with a similar time stamp, preferably with the same time stamp, may be stitched together before transmitting over the wireless networks. It may be preferred that the at least one image apparatus, preferably the plurality of apparatuses comprises a single router, wherein the single router comprises a plurality of modems, thereby allowing for transmitting the stitched image frames over the wireless networks, respectively. Stitching the image frames after transmission can reduce the delay of simultaneously displaying the image frames at each time stamp. As explained herein, any delayed image frame transmitted over a first wireless network can be replaced by a replica image frame transmitted over a second wireless network, thereby allowing for stitching of the image frames after transmission to be more seamlessly and effectively performed. Furthermore, this can allow for the simultaneous transmission of the image frames (without any delay caused for example by stitching) and for a quicker detection of any delays and / or data loss and / or connectivity / modem issues, which allows to use any necessary replica of an image frame more efficiently.

[0062] In embodiments, the transmission of image frames comprises stitching the image frames captured by the image sensors at each time stamp. Thus, the image frames at each time stamp or with a similar time stamp, preferably with the same time stamp, may be stitched together before transmitting over the wireless networks. It may be preferred that the at least one image apparatus, preferably the plurality of apparatuses comprises a single router, wherein the single router comprises a plurality of modems, thereby allowing for transmitting the stitched image frames over the wireless networks, respectively. Stitching the image frames before transmission allows for ensuring that the image frames at each time stamp are transmitted together, thereby, ensuring that the image frames at each time stamp are displayed simultaneously, with minimal or no delay. It will be understood that any delayed stitched image frame transmitted over a first wireless network can be replaced by a replica stitched image frame transmitted over a second wireless network, thereby allowing for a seamless transmission and a more effective and simultaneous display of the image frames at each time stamp.

[0063] In embodiments, the wireless networks comprise a primary wireless network and at least one secondary wireless network, wherein the method comprises: upon detecting that the time interval between consecutive image frames over the primary wireless network exceeds the first predetermined threshold, selecting a replica of the successive image frame transmitted over any of the at least one secondary wireless network. Preferably, the at least one secondary wireless network comprises a plurality of secondary wireless networks.

[0064] In embodiments, the method comprises selecting a replica of the successive image frame transmitted over any of the at least one secondary wireless network upon detecting that the time interval between consecutive image frames over the primary wireless network exceeds the first predetermined threshold.

[0065] In embodiments, the method comprises selecting at least one replica image frame transmitted over any of the at least one secondary wireless network based on the detection in any one or combination of steps 1.b, 2.a, 2.b, 3.a, 3.b, 3.c, 3.d and 4.a.

[0066] In embodiments, the selecting of a replica of the successive image frame comprises detecting from among the plurality of secondary wireless networks the secondary wireless network having the smallest detected time interval between respective replicas of the consecutive image frames transmitted over said secondary wireless network.

[0067] In embodiments, the selecting of the least one replica image frame comprises detecting from among the plurality of secondary wireless networks the secondary wireless network having any one or combination of: the smallest round-trip time of a data packet (i.e., step 2.a), the smallest loss of data packets (i.e., step 2.b), the largest RSSI (i.e., step 3.a), the largest RSRP (i.e., step 3.b), the largest RSRQ (i.e., step 3.c), the largest SIR and / or SNR and / or SNIR (i.e., step 3.d), and the largest CNR (i.e., step 4.a).

[0068] The selecting of a replica image frame from a secondary wireless network allows to smooth out the variations in network delay and packet loss, allowing for a seamless transmission of data and providing a more consistent and reliable connection for real-time applications such as voice and video. This is particularly done by any one or combination of the following: Packet loss concealment, Time stretching, Clock synchronization, Forward error correction, and Quality of Service (QoS) marking. Packet loss concealment may relate to inserting redundant data packets (e.g., replica image frames) into the network stream to hide missing packets and improve the overall quality of the connection. Time stretching may relate to slowing down the transmission rate of data packets to reduce the effects of latency and jitter. Clock synchronization may relate to synchronizing the clocks of all devices on a wireless network to ensure that time stamps are accurate and consistent. Forward error correction may relate to adding redundant data (e.g., replica image frames) to the network stream to detect and correct errors that may occur during transmission. QoS may relate to assigning priority levels to different types of traffic in a wireless network, ensuring that critical applications receive adequate bandwidth and low latency.

[0069] In embodiments, the secondary wireless network having the smallest detected time interval between the replicas of the consecutive image frames is set as the primary wireless network. The previous primary wireless network is thus set as a secondary wireless network comprised in the plurality of secondary wireless networks.

[0070] In embodiments, any or each of the at least one image apparatus comprises at least one router for connecting each of the at least one image apparatus to the wireless networks, wherein each of the at least one router comprises a plurality of modems for connecting each of the at least one image sensor to the plurality of wireless networks, respectively. For example, a first image apparatus comprises a first router and a second image apparatus comprises a second router, each of the first and second routers having two modems which connect the first and second image apparatuses to first and second wireless connections, respectively. The at least one router may be a mesh router / node.

[0071] In alternative embodiments, any or each of the at least one image apparatus comprises a plurality of routers / modems for connecting each of the at least one image apparatus to the plurality of wireless networks, respectively. For example, a first image apparatus comprises a first and a second router and a second image apparatus comprises a third and a fourth router, the first and third routers connecting the first and second image apparatuses to first wireless connections, and the second and fourth routers connecting the first and second image apparatuses to second wireless connections. The plurality of routers / modems may be mesh routers / nodes.

[0072] In preferred alternative embodiments, the at least one image apparatus comprises a single router for connecting the at least one image apparatus to the wireless networks, wherein the single router may comprise a plurality of modems. This can ensure that the image frames from each of the at least one image apparatus are sent over the same wireless connection. For example, a single router has two modems, where a first modem connects a first and second image apparatus to the same first wireless connection and a second modem connects the first and second image apparatus to the same second wireless connection.

[0073] In embodiments, the display apparatus comprises at least one display. Preferably, the display apparatus further comprises any one or combination of: at least one microphone, at least one speaker, at least one means (e.g., an input, such as a button, a touch input / icon) for transmitting control signals. The at least one microphone may be controlled by an input means (e.g., a button or a touch icon) for transmitting sound captured by the at least one microphone. The sound may be transmitted to the at least one image apparatus (e.g., to the at least one speaker comprised therein) and / or to another personnel (e.g., on site via a mobile device or in a command center via a receiving device). The at least one speaker may receive sound from the at least one microphone comprised in the at least one image apparatus and / or from another personnel (e.g., on site via a mobile device or in a command center via a transmitting device). The control signals may be transmitted via the display apparatus to the at least one image apparatus and / or to the rail vehicle control system. The control signals transmitted to the at least one image apparatus and / or the at least one additional apparatus may control any one or combination of: the at least one light source, the at least one image sensor, the at least one horn / buzzer, the at least one additional sensor, the at least one microphone and the at least one speaker. For example, a control signal is transmitted to the at least one image apparatus to reset the same or the streaming protocol thereof. The control signals transmitted to the rail vehicle control system may control, among others, the acceleration, the braking, the lights, etc.

[0074] In embodiments, the display apparatus comprises a router for connecting the display apparatus to any of the wireless networks. The router may comprise a plurality of modems which connect the display apparatus to the plurality of wireless networks, respectively. The plurality of modems comprised in the router which is comprised in the display apparatus may be equal to the number of modems comprised in the router comprised in the at least one image sensor.

[0075] In embodiments, the router in the display apparatus and the router in the at least one image apparatus may each comprise a clock for synchronization. Alternatively, the router in the display apparatus and the router in each of the at least one image apparatus may each comprise a clock for synchronization.

[0076] In embodiments, each of the at least one image apparatus comprises means (e.g., a timer) for attributing a timestamp to each of the image frame captured by the at least one image sensor comprised in each of the at least one image apparatus. The timestamp may be allocated at the time of capturing of the image frames by the respective image sensor.

[0077] In embodiments, any one or combination of the display apparatus and the at least one image apparatus comprises a global positioning system (GPS) device for determining location information of the rail vehicle, particularly of any one or combination of the display apparatus and the at least one image apparatus. Additionally or alternatively, the location information of any one or combination of the display apparatus and the at least one image apparatus may be determined via any one or combination (e.g., via information fusion) of the wireless networks, such as by Wi-Fi location services, cellular network triangulation, IP address geolocation, Bluetooth-based location services (e.g., Beacon technology), satellite-based location services, satellite network triangulation, etc.

[0078] In embodiments, the system comprises the device according to the invention, the at least one image apparatus, and the display apparatus wirelessly connected to the at least one image apparatus, as described herein, wherein the device is configured to perform the method steps, as described herein, such as: a. monitoring the data link between the at least one image apparatus and the display apparatus over a plurality of wireless networks, wherein the data link comprises a transmission of image frames captured with the at least one image apparatus, wherein the transmission of image frames is replicated over the wireless networks, and b. generating an alert upon detecting that a time interval between consecutive image frames over each or any of the wireless networks exceeds a first predetermined threshold relating to an acceptable delay.

[0079] In embodiments, the system comprises any one or combination of: at least one router, such as a wireless router (e.g., mesh routers and / or mesh nodes, Wi-Fi access points, etc.) and / or a wired router (e.g., backhaul connections) for a Wi-Fi Mesh Network; at least one satellite-Wi-Fi connector, such as a Starlink Wi-Fi pole, for connecting with the at least one satellite; at least one repeater (e.g., 4G / 5G outdoor repeater) for extending the cellular connection between the at least one image apparatus and the display apparatus; and at least one central network infrastructure component (e.g., a 5G core) for connecting and managing various apparatuses (i.e., the at least one image apparatus and / or the display apparatus) and / or user (e.g., personnel, such as on site via a mobile device or in a command center via a receiving device).

[0080] In embodiments, the system comprises at least one additional apparatus comprising at least one additional sensor and / or at least one light source (e.g., LED) and / or at least one horn / buzzer and / or at least one microphone and / or at least one speaker. The at least one additional sensor may comprise any one or combination of: at least one temperature sensor, at least one acceleration sensor, at least one humidity sensor, and at least one proximity sensor.

[0081] In embodiments, any or each of the at least one additional apparatus comprises at least one router for connecting each of the at least one additional apparatus to the wireless networks. Thus, the at least one additional apparatus may transmit data, such as sensor data and / or communication data, to the display apparatus over the wireless networks. The at least one additional apparatus may receive communication data, such as control signals, from the display apparatus over the wireless networks. The at least one router comprised in the at least one additional apparatus may be a mesh router / node.

[0082] In embodiments, any or each of the at least one additional apparatus is located in proximity of the at least one image apparatus. Additionally or alternatively, any or each of the at least one additional apparatus is located on top of the rail vehicle and / or on a side of the rail vehicle and / or at the front of the rail vehicle in the forward direction.Examples

[0083] Example embodiments of the invention will be described with reference to Fig. 1, 2A and 2B, which are not intended to limit the scope of the invention in any way.Example 1: example of a method and / or a system according to the invention

[0084] This example is described with reference to Fig. 1. Fig. 1 shows a perspective view, particularly a side view, of a rail vehicle (30).

[0085] In this example according to embodiments of the invention, a system comprising a device for monitoring a data link in the rail vehicle (30) comprising at least one image apparatus (11) and a display apparatus (18) wirelessly connected to the at least one image apparatus (11).

[0086] Further in this example according to embodiments of the invention, the method of monitoring a data link in the rail vehicle (30) comprising the at least one image apparatus (11) and the display apparatus (18) comprises: 1.a. monitoring the data link between the at least one image apparatus (11) and the display apparatus (18) over a plurality of wireless networks (21, 22-24, 25-27), wherein the data link comprises a transmission of image frames captured with the at least one image apparatus (11), wherein the transmission of image frames is replicated over the wireless networks (21, 22-24, 25-27), and 1.b. generating an alert upon detecting that a time interval between consecutive image frames over each or any of the wireless networks (21, 22-24, 25-27) exceeds a first predetermined threshold relating to an acceptable delay.

[0087] In this example, the rail vehicle (30) comprises an image apparatus (11) and a display apparatus (18). The wireless networks comprise a Wi-Fi network (21), a cellular network (22-24) and a satellite network (25-27). The cellular network comprises a base station (24) which acts as a gateway between the image apparatus (11) and the display apparatus (18). Thus, the cellular network comprises a first connection (22) between the image apparatus (11) and the base station (24) and a second connection (23) between the display apparatus (18) and the base station (24). The satellite network comprises a satellite (27) which acts as a gateway between the image apparatus (11) and the display apparatus (18). Thus, the satellite network comprises a third connection (25) between the image apparatus (11) and the satellite (27) and a fourth connection (26) between the display apparatus (18) and the satellite (27).

[0088] Here, we assume that the Wi-Fi network (21) is the primary wireless network, thus, by transmitting replicas of image frames over any the other wireless networks (22-24, 25-27), at least one replica image frame can be selected from any of the other wireless networks (22-24, 25-27) upon detecting any one or combination of issues in delay, data loss, and connectivity (e.g., power, quality, noise, interference, etc.) according to the present invention.Example 2: example of a method and / or system according to the invention

[0089] This example is described with reference to Fig. 2A. Fig. 2A shows a perspective view, particularly a top view, of a rail vehicle (30).

[0090] In this example according to embodiments of the invention, a system comprising a device for monitoring a data link in the rail vehicle (30) comprising at least one image apparatus (11) and a display apparatus wirelessly connected to the at least one image apparatus (11).

[0091] The at least one image apparatus (11) comprises a radar sensor having a first field of view (FoV) or sensor range (12), a LiDAR sensor having a second FoV (13), an infrared camera having a third FoV (14), and a laser camera having a fourth FoV (15).

[0092] The plurality of FoVs (12-15) are partially overlapping. At each time stamp, the image frames captured by said sensors may be transmitted separately over the wireless networks or transmitted as a stitched image frames over the wireless networks or as merged image frames over the wireless networks, according to the present invention.Example 3: example of a method and / or system according to the invention

[0093] In this example we consider the at least one sensor of Fig. 2A. This example is described with reference to Fig. 2B. Fig. 2B shows a perspective view, particularly a top view, of a rail vehicle (30).

[0094] In this example according to embodiments of the invention, a system comprising a device for monitoring a data link in the rail vehicle (30) comprising a plurality of image apparatuses (11, 11', 11") and a display apparatus wirelessly connected to the plurality of image apparatuses (11, 11', 11").

[0095] The plurality of image apparatuses (11, 11', 11") comprise respectively a first laser camera, a second laser camera and a third laser camera having a first FoV (15), a second FoV (15') and a third FoV (15"), respectively. As can be seen in Fig. 2B, the FoV of the three laser cameras is oriented in a different direction, particularly the first FoV is in the reversed direction of the rail vehicle (30), the second FoV is in the reversed direction of the rail vehicle (30) with a predefined angle in the clockwise direction, and the third FoV is in the reversed direction of the rail vehicle (30) with a predefined angle in the counter clockwise direction. Thus, the combination of the three cameras, a larger FoV can be obtained. This may be done by merging the image frames captured by the cameras at each time stamp, preferably, where the overlapping FoVs may be merged, e.g. based on the most similar pixel pairs in the respective pair of image frames.

[0096] At each time stamp, the image frames captured by said sensors may be transmitted separately over the wireless networks or transmitted as a stitched image frame over the wireless networks or as merged image frames over the wireless networks, according to the present invention.(End of Example 3)

[0097] Although the present invention has been described above with reference to certain embodiments thereof, it will be evident that various modifications and changes may be made to these embodiments without departing from the broader scope of the present invention, as defined by the appended claims.

Examples

example 3

(End of Example 3)

Field of the invention

[0001] The present invention relates to the technical domain of rail transportation and more particularly to techniques for monitoring data links in rail vehicles.Background art

[0002] Rail vehicles are essential modes of transport for goods and people over long distances as well as in local transportation networks. In shunting operations, a rail vehicle is maneuvered in a confined area or reversed direction to couple or decouple other railway cars. This process typically requires the assistance of one or more personnel on or near the track to communicate instructions and coordinate movements. Shunting operations in particular can be time-consuming and dangerous for the personnel involved, as they require close proximity to moving trains and a high level of attention from the rail driver.

[0003] The current state of the art in rail vehicle conducting and / or shunting operations relies heavily on human intervention. A rail vehicle for transporting goods and / or people is typically operated by a rail driver who must manage several functions, including controlling the speed of the rail vehicle, detecting track signs, and ensuring safe movement of the train. However, research shows that 75% of rail incidents are caused by human error or poor judgment. Common causes of such incidents include excessive speeding, lack of attention, violation of traffic rules, and very slow reaction speed in the event of an approaching collision. These risks are particularly prevalent during shunting operations due to the proximity of personnel to moving trains and the reverse direction of travel.

[0004] Communication between personnel and the rail vehicle driver is often limited to verbal instructions and / or hand signals, which can be misinterpreted or lost in noisy environments. Additionally, manual shunting operations require a significant amount of time and personnel resources, making such operations inefficient.

[0005] These disadvantages necessitate a system that can improve rail vehicle conducting and / or shunting operations. It is therefore an objective of the present invention to provide a rail vehicle monitoring system capable of enhancing the safety and efficiency of rail vehicle conducting and / or shunting operations, ultimately benefiting both the rail industry and the general public.Summary of the invention

[0006] One aspect of the present invention relates to a method of monitoring a data link in a rail vehicle comprising at least one image apparatus and a display apparatus. The data link between the at least one image apparatus and the display apparatus allows for real-time monitoring and optionally control of rail vehicle operations, including conducting and shunting.

[0007] It may be provided that this method comprises monitoring the data link over a plurality of wireless networks. These networks can be any type of wireless communication systems capable of transmitting image frames captured by the image apparatus to the display apparatus. By replicating the transmission of image frames over multiple networks, data reception by the display apparatus is improved. This redundancy ensures reliable data transfer and minimizes interruptions in the data link, which is critical during rail vehicle operations.

[0008] One advantage of monitoring the data link over a plurality of wireless networks is the enhanced reliability of the communication pathway. In case one network experiences challenges or becomes unavailable, the transmission can continue through other available networks, thereby minimizing interruptions and ensuring consistent, real-time image frame transfer. This is particularly important during critical rail vehicle operations such as conducting and shunting where precise communication between the at least one image apparatus and the display apparatus is necessary for safety and efficiency.

[0009] Another advantage of monitoring the data link between the image apparatus and the display apparatus over multiple wireless networks is improved system resilience. By distributing the communication load across various networks, the overall system becomes more robust to network failures or disruptions. This ensures that the rail vehicle's onboard systems remain operational and provide accurate real-time information for the driver, even when one or more of the wireless networks are compromised. This is particularly advantageous in harsh environments, where any one or combination of the following would typically affect communication: low coverage, low bandwidth, few connectivity hardware on site, interference with buildings and / or poles and / or machines, and Electromagnetic Compatibility (EMC).

[0010] Furthermore, it may be provided that this method comprises generating an alert upon detecting that a time interval between consecutive image frames over each or any of the wireless networks exceeds a first predetermined threshold relating to an acceptable delay. This threshold is based on a desired response time for the system and should not significantly impact the overall throughput or bandwidth requirements.

[0011] The advantage of generating an alert upon detecting a time interval between consecutive image frames that exceeds a predetermined threshold is the ability to quickly identify potential issues with the data link and take corrective measures. For instance, this could indicate network congestion, equipment failure, or other problems that may require intervention from maintenance personnel or system administrators. By addressing these issues promptly, the overall performance of the rail vehicle's onboard systems can be maintained, ensuring efficient operation and safety.

[0012] Thus, the present invention will not only improve safety but also enhance efficiency by reducing the need for manual intervention. This reduction in labor requirements can lead to cost savings for railway companies and increased productivity. Additionally, the system's advanced features, such as real-time and improved information communication, will help reduce human error and miscommunication, further increasing overall safety.

[0013] In particularly advantageous embodiments, any or each of the at least one image apparatus comprises at least one image sensor, preferably the at least one image sensor comprises a plurality of image sensors. In preferred embodiments, the transmission of image frames comprises stitching the image frames from the image sensors at each time stamp or the images frames from the image sensors are stitched at each time stamp after transmission.

[0014] In particularly advantageous embodiments, the rail vehicle is being used in a shunting operation. In shunting operations, a rail vehicle is typically operated in the reverse direction of travel, where a rail vehicle driver requires assistance from personnel on or near the rail track of an industrial site (e.g., rail site, railway, railroad, railyard, etc.). Thus, an advantage of the present invention is to allow for improved communication of information to the rail driver, and thereby reducing or eliminating the need for personnel on or near the track of the site.

[0015] According to a second aspect, the present invention provides a device comprising means for carrying out the method according to the invention.

[0016] According to a third aspect, the present invention provides a system comprising the device according to the invention, at least one image apparatus; and a display apparatus wirelessly connected to the at least one image apparatus, wherein the device is configured to perform the method steps described above.

[0017] Such a system may advantageously allow for improved integration with a rail vehicle. In addition to the advantages provided by the method, as described above, the system may accommodate additional apparatuses, such as an additional sensor, a horn / buzzer, a microphone, a speaker, etc., and thereby, allowing for a modular system. Furthermore, such a system allows for integration with several types of rail vehicles, which is particularly advantageous for older rail vehicles.

[0018] According to a fourth aspect, the present invention provides a computer program comprising instructions which, when the program is executed by a computer, cause the computer to carry out the method according to the invention. The computer program product may comprise at least one readable medium in which computer-readable program code portions are saved, which program code portions comprise instructions for carrying out said method.

[0019] According to a fifth aspect, the present invention provides a computer-readable medium comprising instructions which, when executed by a computer, cause the computer to carry out the method according to the invention.

[0020] According to a further aspect, the present invention provides a rail vehicle comprising the system according to the present invention.

[0021] Preferred embodiments and their advantages are provided in the description and the dependent claims.Brief description of the drawings

[0022] The present invention will be discussed in more detail below, with reference to the attached drawings. Fig. 1 shows a first example of a method and / or system according to the invention. Fig. 2A shows a second example of a method and / or system according to the invention. Fig. 2B shows a third example of a method and / or system according to the present invention. Description of embodiments

[0023] The following descriptions depict only example embodiments and are not considered limiting in scope. Any reference herein to the disclosure is not intended to restrict or limit the disclosure to exact features of any one or more of the exemplary embodiments disclosed in the present specification.

[0024] Furthermore, the terms first, second, third and the like in the description and in the claims are used for distinguishing between similar elements and not necessarily for describing a sequential or chronological order. The terms are interchangeable under appropriate circumstances and the embodiments of the invention can operate in other sequences than described or illustrated herein.

[0025] Furthermore, the various embodiments, although referred to as "preferred" are to be construed as exemplary manners in which the invention may be implemented rather than as limiting the scope of the invention.

[0026] The term "comprising", used in the claims, should not be interpreted as being restricted to the elements or steps listed thereafter; it does not exclude other elements or steps. It needs to be interpreted as specifying the presence of the stated features, integers, steps or components as referred to, but does not preclude the presence or addition of one or more other features, integers, steps or components, or groups thereof. Thus, the scope of the expression "a device comprising A and B" should not be limited to devices consisting only of components A and B, rather with respect to the present invention, the only enumerated components of the device are A and B, and further the claim should be interpreted as including equivalents of those components.

[0027] In this document, the term "data link" refers to a communication pathway between components within a system, such as an image apparatus and a display apparatus and / or an additional apparatus and the display apparatus, according to the present invention. Thus, the data link may comprise the communication of sensor data (e.g., image frames and / or data from any additional sensor), preferably further comprising at least one of the following communication data: time stamps, control signals, sounds, text, binary data, structured data (e.g., Extensible Markup Language (XML), and JavaScript Object Notation (JSON)). In this context, a rail vehicle refers to a self-propelled vehicle designed for transportation on rails, such as a (freight and / or passenger) train, commuter railcar, monorail, Maglev train, tram, metro / subway, hyperloop pod, etc. In this context, the term image apparatus refers to an electronic device which comprises an image sensor that captures data relating to images or moving pictures, and a display apparatus refers to an electronic device comprising a display that shows / displays visual information.

[0028] In embodiments, the method of monitoring a data link in a rail vehicle comprising at least one image apparatus and a display apparatus, wherein the device is configured to perform the steps of: 1.a. monitoring the data link between the at least one image apparatus and the display apparatus over a plurality of wireless networks, wherein the data link comprises a transmission of image frames captured with the at least one image apparatus, wherein the transmission of image frames is replicated over the wireless networks, and 1.b. generating an alert upon detecting that a time interval between consecutive image frames over each or any of the wireless networks exceeds a first predetermined threshold relating to an acceptable delay.

[0029] In embodiments, the wireless networks comprise any one or combination of: one or more cellular networks, one or more Wi-Fi networks, one or more satellite networks, one or more Bluetooth networks, one or more Zigbee networks, and one or more industrial networks. The one or more cellular networks may be from the group of: 4G, 5G, Time Division-Spectrum Code Division Multiple Access (TD-SCDMA), Wideband Code Division Multiple Access (WCDMA), Long-Term Evolution (LTE), 5G New Radio (NR), Stand Alone (SA), Non-SA, 5G SA, and 5G NSA. The one or more cellular network may involve at least one base station (e.g., a tower cell or a node), which acts as a gateway and provides coverage. The one or more Wi-Fi networks may be from the group of: 802.11g, 802.11n, 802.11ac, and 802.11ad, 802.11 ax. The one or more satellite networks may be from the group of: Starlink, OneWeb, LeoSat and Telesat. The one of more satellite networks may involve at least one satellite, preferably a plurality of satellites, which act(s) as a gateway and provide(s) coverage. The one or more Bluetooth networks may be from the group of: Bluetooth 3.0, Bluetooth 4.0, Bluetooth 5.0, Bluetooth Smart, Bluetooth Mesh, and Bluetooth Classic. The one or more Zigbee networks may be from the group of: Zigbee 3.0, Zigbee RF4CE, Zigbee PRO, Zigbee Green Power (ZGP), Zigbee Industrial Automation (ZIA). It will be understood that newer wireless network protocols and / or standards may also be used.

[0030] In particularly advantageous embodiments, the wireless networks comprise the one or more cellular networks comprising a private 5G infrastructure utilizing dedicated industrial frequencies. This can ensure a stable and secure connection for communicating data, such as sensor data and / or communication data. The one or more cellular network further comprises one or more public 4G / 5G networks. This can ensure comprehensive coverage across an industrial site.

[0031] In particularly advantageous embodiments, the wireless networks comprise the one or more Wi-Fi networks comprising a Wi-Fi Mesh Network. Here, the system may comprise at least one router, such as a wireless router (e.g., mesh routers and / or mesh nodes, Wi-Fi access points, etc.) and / or a wired router (e.g., backhaul connections). For instance, in a wired backhaul connection, each mesh node can be connected via wired Ethernet to another node or the main router to ensure a stable and high-speed connection between nodes. This is especially advantageous when the wireless signal strength between two nodes is weak.

[0032] In particularly advantageous embodiments, the wireless networks comprise the one or more satellite networks comprising at least one satellite-Wi-Fi connector, such as a Starlink Wi-Fi pole, for connecting with the at least one satellite. The at least one satellite-Wi-Fi connector may be configured for connecting with the at least one image apparatus and with the display apparatus. Thus, the at least one satellite-Wi-Fi connector may provide reliable satellite connectivity where terrestrial network coverage (e.g., the one or more cellular networks) is inadequate, thereby ensuring the at least one image apparatus continue to transmit data essential for safe and efficient train shunting and / or conducting operation.

[0033] In particularly advantageous embodiments, the wireless networks comprise the one or more cellular networks comprising at least one repeater (e.g., 4G / 5G outdoor repeater) for extending the cellular connection between the at least one image apparatus and the display apparatus. This can amplify weak 4G / 5G signals in specific areas of the site with insufficient coverage, bolstering the connectivity required for image frame streaming (e.g., live video) and / or sensor data and / or communication data transmission.

[0034] In particularly advantageous embodiments, the wireless networks comprise the one or more cellular networks comprising at least one central network infrastructure component (e.g., a 5G core) for connecting and managing various apparatuses (i.e., the at least one image apparatus and / or the display apparatus) and / or user (e.g., personnel, such as on site via a mobile device or in a command center via a receiving device). Said at least one central network infrastructure component may be installed on the rail vehicle, emitting a private network that extends around the rail vehicle, covering the entire rail vehicle. Alternatively, said at least one central network infrastructure component may be installed on the industrial site, emitting a private network that extends around the rail vehicle, covering gaps in the public 4G / 5G network. These private networks can provide a secure and dedicated channel for the at least one image apparatus, ensuring consistent and reliable connectivity for the system's operations.

[0035] In embodiments, the data link comprises a communication protocol for transmitting the image frames and providing VPN encryption. The communication protocol may comprise establishing a secure tunnel between the at least one image apparatus and the display apparatus using any one or combination of cryptographic techniques, such as SSL / TLS encryption and IPsec authentication. This allows to reduce the overhead associated with encrypting and decrypting data, resulting in faster transmission speeds and lower latency. The communication protocol may comprise any one or combination of the wireless network protocol described above. The afore-mentioned wireless networks may be configured or integrated in the communication protocol comprised in the data link. This can allow for a unified and secure network infrastructure that provides reliable connectivity and high levels of performance.

[0036] In embodiments, the at least one image sensor is any one or combination from the group of: a monocular camera, an infrared camera, a thermal imaging sensor, a low-light imaging sensor, a radar sensor, a light detection and ranging (LiDAR) sensor, a time-of-flight (ToF) camera, and a 3D camera.

[0037] In embodiments, any of the at least one image sensor has a low-light vision mode and / or a night vision mode for capturing image frames in low-light environments and / or during nighttime. Additionally or alternatively, any of the at least one image sensor is equipped with at least one image processing algorithm for enhancing the quality of the images captured in low-light environments.

[0038] In embodiments, the image frames are processed by the at least one image processing algorithm before transmitting (e.g., by the at least one image sensor and / or an external processor) and / or after transmitting (e.g., by the display apparatus and / or the external processor). The at least one image processing algorithm may be any one or combination from the group of: contrast enhancement, brightness enhancement, noise reduction, average color filtering, histogram equalization, Gamma correction, white balance adjustment, sharpening, unsharp masking, edge detection, thresholding, watershed transformation, morphological operations, etc. Additionally or alternatively, the at least one image processing algorithm may relate to object detection and / or classification in any image frame, and / or be any one or combination from the group of: Region-based Convolutional Neural Network (R-CNN), Single Shot Detector (SSD), RetinaNet, DeepLab, U-Net, etc.

[0039] In embodiments, the at least one image sensor comprises a plurality of image sensors. The plurality of image sensors may be any one or combination from the group of: one or more monocular cameras, one or more infrared cameras, one or more radar sensors, one or more LiDAR sensors, one or more laser cameras, one or more ToF cameras, and one or more 3D cameras.

[0040] In embodiments, any or each of the at least one image apparatus comprises at least one additional sensor and / or at least one light source (e.g., LED) and / or at least one horn / buzzer and / or at least one microphone and / or at least one speaker. The at least one additional sensor may comprise any one or combination of: at least one temperature sensor, at least one acceleration sensor, at least one humidity sensor, at least one ultrasonic sensor, and at least one proximity sensor. In this context, the sensor data may comprise any one or combination of temperature data, acceleration data, humidity data, ultrasonic wave data (or distance data determined based on the ultrasonic wave data), and proximity data.

[0041] In embodiments, the at least one image apparatus comprises a plurality of image apparatuses. The plurality of image apparatuses may comprise no similar image sensors, at least one similar image sensor, a plurality of similar image sensors or only similar image sensors. In further embodiments, any or each of the plurality of image apparatuses comprises at least one of: at least one additional sensor, at least one light source (e.g., LED), at least one horn / buzzer, at least one microphone, and at least one speaker.

[0042] In embodiments, the plurality of image sensors has at least partially overlapping field-of-views (FoVs). Alternatively, the plurality of image sensors has end-to-end FoVs. It is preferred that the total or combined FoV of the plurality of image sensors has an angle of at least 120 degrees, preferably at least 150 degrees, more preferably at least 180 degrees, most preferably 210 degrees. At each time point (e.g., time stamp), the image frames captured by said image sensors may be merged. Said merging of the image frames may comprise any one or combination of: Kalman filter-based registration, Random sample consensus (RSC), Mutual information-based registration, Feature-based alignment, Deep learning-based registration, Iterative closest point (ICP), and other optimization algorithms (e.g., Gradient descent-based registration, Non-linear least squares registration, etc.). The Kalman filter-based registration may relate to uses a Kalman filter to estimate the camera poses (position and orientation) and then registers the images using an affine transformation. The RSC may relate to using a random sample of pixels from each image frame and computes the similarity between the samples. The mutual information-based registration may relate to using mutual information to measure the similarity between the images and then registers the images using a maximum likelihood estimation. The RSC and the mutual information-based registration result in the final merged image being based on the most similar pixel pairs. The feature-based alignment may relate to This method aligns the images by matching features such as corners, edges, or blobs. Once the images are aligned, they can be combined using an averaging technique. The deep learning-based registration may relate to mapping the images with their corresponding registers using deep learning. The ICP may relate to alternating between finding the best registration transformation and verifying that the resulting image is close to the original image.

[0043] In embodiments, said merging of the image frames is performed before transmitting by the at least one image apparatus (e.g., a processor comprised therein) or after transmitting the image frames, i.e., after receiving the image frames by the display apparatus (e.g., a processor comprised in the display apparatus).

[0044] In embodiments, the first predetermined threshold relating to an acceptable time interval between consecutive image frames is expressed in milliseconds (ms) and may be any one of: 700 ms, 600 ms, 500 ms, 400 ms, 300 ms, etc. For example, step 1.b may comprise generating the alert upon detecting that the time interval between consecutive image frames is less than or equal to 700 ms, preferably less than or equal to 600 ms, more preferably less than or equal to 500 ms, most preferably less than or equal to 400 ms.

[0045] In embodiments, the transmission of the image frames comprises the transmission of data packets of the image frames, wherein step 1.b further comprises generating the alert: 2.a. upon detecting that a round-trip time of a data packet from among the data packets over each or any of the wireless networks a second predetermined threshold relating to an acceptable round-trip delay, and / or 2.b. upon detecting that a loss of data packets over each or any of the wireless networks exceeds a third predetermined threshold relating to an acceptable degree of pixelation.

[0046] Steps 2.a-2.b may relate to delay and / or loss metrics. Examples thereof are described hereinafter.

[0047] In step 2.a, the round-trip time (RTT) may refer to a measure for evaluating the time it takes for a data packet to travel from a transmitter to a receiver and back, i.e., from the at least one image apparatus to a display apparatus and back. A lower RTT indicates faster communication between the at least one image apparatus and the display apparatus, and a better network performance. The RTT may be expressed in milliseconds (ms). The second predetermined threshold relating to an acceptable delay may be any one of: 200 ms, 300 ms, 400 ms. 500 ms, 600 ms, 700 ms, 800 ms, 900 ms, 1000 ms, etc. For example, step 1.b may comprise generating the alert upon detecting that the RTT is greater than or equal to 200 ms, preferably greater than or equal to 400 ms, more preferably greater than or equal to 600 ms, most preferably greater than or equal to 800 ms.

[0048] In step 2.b, the loss of data packets may refer to a Bit error rate (BER) which is a measure for quantifying the number of bits that are received in error relative to the total number of bits transmitted. The lower the BER, the better the performance of the communication system. The BER may be expressed in errors per bit (err / bit). The third predetermined threshold relating to an acceptable degree of pixelation may be any one of: 10 -12< , 10 -10< , 10 -9< , 10 -8< , 10 -7< , 10 -6< , 10 -5< , 10 -4< , 10 -3< , 10 -2< , and 10 -1< . For example, step 1.b may comprise generating the alert upon detecting that the BER is greater than or equal to 10 -6< , preferably greater than or equal to 10 -5< , more preferably greater than or equal to 10 -4< , most preferably greater than or equal to 10 -3< .

[0049] In embodiments, step 1.b further comprises generating the alert: 3.a. upon detecting that a received signal strength exceeds a fourth predetermined threshold relating to an acceptable power level the display apparatus is receiving over each or any of the wireless networks, and / or 3.b. upon detecting that a reference signal received power exceeds a fifth predetermined threshold relating to an acceptable power level of the reference signal in each or any of the wireless networks, and / or 3.c. upon detecting that a reference signal received quality exceeds a sixth predetermined threshold relating to an acceptable quality level of the reference signal in each or any of the wireless networks, and / or 3.d. upon detecting that a signal-to-interference and / or noise-ratio exceeds a seventh predetermined threshold relating to an acceptable power level of the signal to the power level of background interference and / or noise in each or any of the wireless networks.

[0050] Steps 3.a-3.d may relate to connectivity and / or modem metrics. Examples thereof are described hereinafter.

[0051] In step 3.a, the received signal strength may refer to a received signal strength indicator (RSSI) which is a metric that measures the power level a device (in our case, the router) is receiving from the cell tower. A higher RSSI value indicates a stronger signal, leading to better connection quality. The RSSI may be expressed in decibels (dBm). The fourth predetermined threshold relating to an acceptable power level the display apparatus is receiving over each or any of the wireless networks may be any one of: -65 dBm, -70 dBm, -75 dBm, -80 dBm, -85 dBm, -90 dBm, and -95 dBm. For example, step 1.b may comprise generating the alert upon detecting that the RSSI is less than or equal to -65 dBm, preferably less than or equal to -75 dBm, more preferably less than or equal to -85 dBm, most preferably less than or equal to -90 dBm.

[0052] In step 3.b, the reference signal received power (RSRP) may refer to a measure of the power level of the Reference Signal in a cellular network. The RSRP may provide a deeper understanding of the actual signal strength at any given location. A higher RSRP signifies a more stable connection. The fifth predetermined threshold relating to an acceptable power level of the reference signal in each or any of the wireless networks may be any one of: -80 dBm, -85 dBm, -90 dBm, -95 dBm, and -100 dBm. For example, step 1.b may comprise generating the alert upon detecting that the RSRP is less than or equal to -80 dBm, preferably less than or equal to -85 dBm, more preferably less than or equal to -90 dBm, most preferably less than or equal to -95 dBm.

[0053] In step 3.c, the reference signal received quality (RSRQ) may refer to a measure of the quality of the received signal. It's a ratio of the RSRP to the interference and noise. The RSRQ may provide a deeper understanding of the actual signal strength at any given location. A higher RSRQ signifies a cleaner signal with less interference. The sixth predetermined threshold relating to an acceptable quality level of the reference signal in each or any of the wireless networks may be any one of: -10 dBm, -12 dBm, -15 dBm, -17 dBm, and -20 dBm. For example, step 1.b may comprise generating the alert upon detecting that the RSRQ is less than or equal to -10 dBm, preferably less than or equal to -12 dBm, more preferably less than or equal to -15 dBm, most preferably less than or equal to -17 dBm.

[0054] In step 3.d, the signal-to-interference and / or noise-ratio (SIR, SNR, SINR) may refer to a measure of the quality of the cellular signal by comparing the level of the signal (e.g., LTE signal) to the level of background noise and / or interference. Higher SIR and / or SNR and / or SINR indicate(s) a clearer channel for data transmission, resulting in better throughput and data rates. The seventh predetermined threshold relating to an acceptable power level of the signal to the power level of the background interference and / or noise in each or any of the wireless networks may be any one of: 20 dBm, 17 dBm, 13 dBm, 10 dBm, 7 dBm, 3 dBm and 0 dBm. For example, step 1.b may comprise generating the alert upon detecting that the SIR and / or SNR and / or SINR is / are less than or equal to 17 dBm, preferably less than or equal to 13 dBm, more preferably less than or equal to 10 dBm, most preferably less than or equal to 7 dBm.

[0055] In embodiments, step 1.b further comprises generating the alert: 4.a upon detecting that a carrier-to-noise ratio exceeds an eighth predetermined threshold relating to an acceptable power level of the carrier to the power level of background noise in each or any of the wireless networks.

[0056] In step 4.a, the carrier-to-noise-ratio (CNR) may refer to a measure of the strength of the carrier signal by comparing the level of the carrier signal to the level of background noise at the carrier frequency. Higher CNR indicates a stronger data transmission, resulting in better throughput and data rates. This may be particularly useful for satellite communication where atmospheric conditions and interferences can affect the quality of the received signal carrier. The eighth predetermined threshold relating to an acceptable power level of the carrier signal to the level of background noise at the carrier frequency in each or any of the wireless networks may be any one of: 50 dBm, 45 dBm, 40 dBm, 35 dBm, 30 dBm, 25 dBm, 20 dBm, 15 dBm, and 10 dBm. For example, step 1.b may comprise generating the alert upon detecting that the CNR is less than or equal to 30 dBm, preferably less than or equal to 25 dBm, more preferably less than or equal to 20 dBm, most preferably less than or equal to 10 dBm.

[0057] In embodiments, the method comprises displaying a blank screen on the display apparatus, preferably on the display comprised therein, upon generating the alert. In embodiments, the alert comprises an alert message, wherein preferably the alert message is displayed on the blank screen. For example, the alert message is displayed as "Delay is too high" or "No image".

[0058] In embodiments, the method comprises shutting down the at least one image apparatus. In preferred embodiments, the method comprises shutting down any of the at least one image apparatus, preferably the at least one image sensor comprised therein based on the detected time interval between consecutive image frames over the respective of the wireless networks exceeds the first predetermined threshold relating to an acceptable delay. For example, an image sensor is shut down based on the detection in any one or combination of steps 1.b, 2.a, 2.b, 3.a, 3.b, 3.c, 3.d and 4.a.

[0059] In embodiments, the method or step 1.b comprises resetting a real-time streaming protocol of the at least one image apparatus, preferably of the at least one image sensor comprised therein. In preferred embodiments, the method or step 1.b comprises resetting a real-time streaming protocol of any of the at least one image apparatus, preferably of the at least one image sensor comprised therein based on the detected time interval between consecutive image frames over the respective of the wireless networks exceeds the first predetermined threshold relating to an acceptable delay. For example, an image sensor and / or the streaming protocol of the image sensor is reset based on the detection in any one or combination of steps 1.b, 2.a, 2.b, 3.a, 3.b, 3.c, 3.d and 4.a.

[0060] In embodiments, the transmission of image frames comprises transmitting the respective image frame of the respective of the plurality of image apparatuses or the plurality of image sensors over any wireless network at each time stamp. Each image frame may include a respective time stamp. Thus, the image frames at each time stamp or with a similar time stamp (e.g., with a predefined margin, such as at most at most 90 ms difference, at most 75 ms difference, at most 60 ms difference, at most 45 ms difference, etc.), preferably with the same time stamp, may be transmitted over the same wireless networks. For example, a first image frame captured by a first image sensor, with a first time stamp (hours:minutes:seconds:milliseconds) of 05:01:10:15AM, and a second image frame captured by a second image sensor, with a second time stamp (hours:minutes:seconds:milliseconds) of 05:01:10:20 AM, are transmitted over each of the plurality of wireless networks at the same time.

[0061] In embodiments, the image frames are stitched at each time stamp or with a similar time stamp, preferably with the same time stamp, after transmission over the wireless networks. For example, the image frames at each time stamp or with a similar time stamp, preferably with the same time stamp, are stitched at the display apparatus. Thus, the image frames at each time stamp or with a similar time stamp, preferably with the same time stamp, may be stitched together before transmitting over the wireless networks. It may be preferred that the at least one image apparatus, preferably the plurality of apparatuses comprises a single router, wherein the single router comprises a plurality of modems, thereby allowing for transmitting the stitched image frames over the wireless networks, respectively. Stitching the image frames after transmission can reduce the delay of simultaneously displaying the image frames at each time stamp. As explained herein, any delayed image frame transmitted over a first wireless network can be replaced by a replica image frame transmitted over a second wireless network, thereby allowing for stitching of the image frames after transmission to be more seamlessly and effectively performed. Furthermore, this can allow for the simultaneous transmission of the image frames (without any delay caused for example by stitching) and for a quicker detection of any delays and / or data loss and / or connectivity / modem issues, which allows to use any necessary replica of an image frame more efficiently.

[0062] In embodiments, the transmission of image frames comprises stitching the image frames captured by the image sensors at each time stamp. Thus, the image frames at each time stamp or with a similar time stamp, preferably with the same time stamp, may be stitched together before transmitting over the wireless networks. It may be preferred that the at least one image apparatus, preferably the plurality of apparatuses comprises a single router, wherein the single router comprises a plurality of modems, thereby allowing for transmitting the stitched image frames over the wireless networks, respectively. Stitching the image frames before transmission allows for ensuring that the image frames at each time stamp are transmitted together, thereby, ensuring that the image frames at each time stamp are displayed simultaneously, with minimal or no delay. It will be understood that any delayed stitched image frame transmitted over a first wireless network can be replaced by a replica stitched image frame transmitted over a second wireless network, thereby allowing for a seamless transmission and a more effective and simultaneous display of the image frames at each time stamp.

[0063] In embodiments, the wireless networks comprise a primary wireless network and at least one secondary wireless network, wherein the method comprises: upon detecting that the time interval between consecutive image frames over the primary wireless network exceeds the first predetermined threshold, selecting a replica of the successive image frame transmitted over any of the at least one secondary wireless network. Preferably, the at least one secondary wireless network comprises a plurality of secondary wireless networks.

[0064] In embodiments, the method comprises selecting a replica of the successive image frame transmitted over any of the at least one secondary wireless network upon detecting that the time interval between consecutive image frames over the primary wireless network exceeds the first predetermined threshold.

[0065] In embodiments, the method comprises selecting at least one replica image frame transmitted over any of the at least one secondary wireless network based on the detection in any one or combination of steps 1.b, 2.a, 2.b, 3.a, 3.b, 3.c, 3.d and 4.a.

[0066] In embodiments, the selecting of a replica of the successive image frame comprises detecting from among the plurality of secondary wireless networks the secondary wireless network having the smallest detected time interval between respective replicas of the consecutive image frames transmitted over said secondary wireless network.

[0067] In embodiments, the selecting of the least one replica image frame comprises detecting from among the plurality of secondary wireless networks the secondary wireless network having any one or combination of: the smallest round-trip time of a data packet (i.e., step 2.a), the smallest loss of data packets (i.e., step 2.b), the largest RSSI (i.e., step 3.a), the largest RSRP (i.e., step 3.b), the largest RSRQ (i.e., step 3.c), the largest SIR and / or SNR and / or SNIR (i.e., step 3.d), and the largest CNR (i.e., step 4.a).

[0068] The selecting of a replica image frame from a secondary wireless network allows to smooth out the variations in network delay and packet loss, allowing for a seamless transmission of data and providing a more consistent and reliable connection for real-time applications such as voice and video. This is particularly done by any one or combination of the following: Packet loss concealment, Time stretching, Clock synchronization, Forward error correction, and Quality of Service (QoS) marking. Packet loss concealment may relate to inserting redundant data packets (e.g., replica image frames) into the network stream to hide missing packets and improve the overall quality of the connection. Time stretching may relate to slowing down the transmission rate of data packets to reduce the effects of latency and jitter. Clock synchronization may relate to synchronizing the clocks of all devices on a wireless network to ensure that time stamps are accurate and consistent. Forward error correction may relate to adding redundant data (e.g., replica image frames) to the network stream to detect and correct errors that may occur during transmission. QoS may relate to assigning priority levels to different types of traffic in a wireless network, ensuring that critical applications receive adequate bandwidth and low latency.

[0069] In embodiments, the secondary wireless network having the smallest detected time interval between the replicas of the consecutive image frames is set as the primary wireless network. The previous primary wireless network is thus set as a secondary wireless network comprised in the plurality of secondary wireless networks.

[0070] In embodiments, any or each of the at least one image apparatus comprises at least one router for connecting each of the at least one image apparatus to the wireless networks, wherein each of the at least one router comprises a plurality of modems for connecting each of the at least one image sensor to the plurality of wireless networks, respectively. For example, a first image apparatus comprises a first router and a second image apparatus comprises a second router, each of the first and second routers having two modems which connect the first and second image apparatuses to first and second wireless connections, respectively. The at least one router may be a mesh router / node.

[0071] In alternative embodiments, any or each of the at least one image apparatus comprises a plurality of routers / modems for connecting each of the at least one image apparatus to the plurality of wireless networks, respectively. For example, a first image apparatus comprises a first and a second router and a second image apparatus comprises a third and a fourth router, the first and third routers connecting the first and second image apparatuses to first wireless connections, and the second and fourth routers connecting the first and second image apparatuses to second wireless connections. The plurality of routers / modems may be mesh routers / nodes.

[0072] In preferred alternative embodiments, the at least one image apparatus comprises a single router for connecting the at least one image apparatus to the wireless networks, wherein the single router may comprise a plurality of modems. This can ensure that the image frames from each of the at least one image apparatus are sent over the same wireless connection. For example, a single router has two modems, where a first modem connects a first and second image apparatus to the same first wireless connection and a second modem connects the first and second image apparatus to the same second wireless connection.

[0073] In embodiments, the display apparatus comprises at least one display. Preferably, the display apparatus further comprises any one or combination of: at least one microphone, at least one speaker, at least one means (e.g., an input, such as a button, a touch input / icon) for transmitting control signals. The at least one microphone may be controlled by an input means (e.g., a button or a touch icon) for transmitting sound captured by the at least one microphone. The sound may be transmitted to the at least one image apparatus (e.g., to the at least one speaker comprised therein) and / or to another personnel (e.g., on site via a mobile device or in a command center via a receiving device). The at least one speaker may receive sound from the at least one microphone comprised in the at least one image apparatus and / or from another personnel (e.g., on site via a mobile device or in a command center via a transmitting device). The control signals may be transmitted via the display apparatus to the at least one image apparatus and / or to the rail vehicle control system. The control signals transmitted to the at least one image apparatus and / or the at least one additional apparatus may control any one or combination of: the at least one light source, the at least one image sensor, the at least one horn / buzzer, the at least one additional sensor, the at least one microphone and the at least one speaker. For example, a control signal is transmitted to the at least one image apparatus to reset the same or the streaming protocol thereof. The control signals transmitted to the rail vehicle control system may control, among others, the acceleration, the braking, the lights, etc.

[0074] In embodiments, the display apparatus comprises a router for connecting the display apparatus to any of the wireless networks. The router may comprise a plurality of modems which connect the display apparatus to the plurality of wireless networks, respectively. The plurality of modems comprised in the router which is comprised in the display apparatus may be equal to the number of modems comprised in the router comprised in the at least one image sensor.

[0075] In embodiments, the router in the display apparatus and the router in the at least one image apparatus may each comprise a clock for synchronization. Alternatively, the router in the display apparatus and the router in each of the at least one image apparatus may each comprise a clock for synchronization.

[0076] In embodiments, each of the at least one image apparatus comprises means (e.g., a timer) for attributing a timestamp to each of the image frame captured by the at least one image sensor comprised in each of the at least one image apparatus. The timestamp may be allocated at the time of capturing of the image frames by the respective image sensor.

[0077] In embodiments, any one or combination of the display apparatus and the at least one image apparatus comprises a global positioning system (GPS) device for determining location information of the rail vehicle, particularly of any one or combination of the display apparatus and the at least one image apparatus. Additionally or alternatively, the location information of any one or combination of the display apparatus and the at least one image apparatus may be determined via any one or combination (e.g., via information fusion) of the wireless networks, such as by Wi-Fi location services, cellular network triangulation, IP address geolocation, Bluetooth-based location services (e.g., Beacon technology), satellite-based location services, satellite network triangulation, etc.

[0078] In embodiments, the system comprises the device according to the invention, the at least one image apparatus, and the display apparatus wirelessly connected to the at least one image apparatus, as described herein, wherein the device is configured to perform the method steps, as described herein, such as: a. monitoring the data link between the at least one image apparatus and the display apparatus over a plurality of wireless networks, wherein the data link comprises a transmission of image frames captured with the at least one image apparatus, wherein the transmission of image frames is replicated over the wireless networks, and b. generating an alert upon detecting that a time interval between consecutive image frames over each or any of the wireless networks exceeds a first predetermined threshold relating to an acceptable delay.

[0079] In embodiments, the system comprises any one or combination of: at least one router, such as a wireless router (e.g., mesh routers and / or mesh nodes, Wi-Fi access points, etc.) and / or a wired router (e.g., backhaul connections) for a Wi-Fi Mesh Network; at least one satellite-Wi-Fi connector, such as a Starlink Wi-Fi pole, for connecting with the at least one satellite; at least one repeater (e.g., 4G / 5G outdoor repeater) for extending the cellular connection between the at least one image apparatus and the display apparatus; and at least one central network infrastructure component (e.g., a 5G core) for connecting and managing various apparatuses (i.e., the at least one image apparatus and / or the display apparatus) and / or user (e.g., personnel, such as on site via a mobile device or in a command center via a receiving device).

[0080] In embodiments, the system comprises at least one additional apparatus comprising at least one additional sensor and / or at least one light source (e.g., LED) and / or at least one horn / buzzer and / or at least one microphone and / or at least one speaker. The at least one additional sensor may comprise any one or combination of: at least one temperature sensor, at least one acceleration sensor, at least one humidity sensor, and at least one proximity sensor.

[0081] In embodiments, any or each of the at least one additional apparatus comprises at least one router for connecting each of the at least one additional apparatus to the wireless networks. Thus, the at least one additional apparatus may transmit data, such as sensor data and / or communication data, to the display apparatus over the wireless networks. The at least one additional apparatus may receive communication data, such as control signals, from the display apparatus over the wireless networks. The at least one router comprised in the at least one additional apparatus may be a mesh router / node.

[0082] In embodiments, any or each of the at least one additional apparatus is located in proximity of the at least one image apparatus. Additionally or alternatively, any or each of the at least one additional apparatus is located on top of the rail vehicle and / or on a side of the rail vehicle and / or at the front of the rail vehicle in the forward direction.Examples

[0083] Example embodiments of the invention will be described with reference to Fig. 1, 2A and 2B, which are not intended to limit the scope of the invention in any way.Example 1: example of a method and / or a system according to the invention

[0084] This example is described with reference to Fig. 1. Fig. 1 shows a perspective view, particularly a side view, of a rail vehicle (30).

[0085] In this example according to embodiments of the invention, a system comprising a device for monitoring a data link in the rail vehicle (30) comprising at least one image apparatus (11) and a display apparatus (18) wirelessly connected to the at least one image apparatus (11).

[0086] Further in this example according to embodiments of the invention, the method of monitoring a data link in the rail vehicle (30) comprising the at least one image apparatus (11) and the display apparatus (18) comprises: 1.a. monitoring the data link between the at least one image apparatus (11) and the display apparatus (18) over a plurality of wireless networks (21, 22-24, 25-27), wherein the data link comprises a transmission of image frames captured with the at least one image apparatus (11), wherein the transmission of image frames is replicated over the wireless networks (21, 22-24, 25-27), and 1.b. generating an alert upon detecting that a time interval between consecutive image frames over each or any of the wireless networks (21, 22-24, 25-27) exceeds a first predetermined threshold relating to an acceptable delay.

[0087] In this example, the rail vehicle (30) comprises an image apparatus (11) and a display apparatus (18). The wireless networks comprise a Wi-Fi network (21), a cellular network (22-24) and a satellite network (25-27). The cellular network comprises a base station (24) which acts as a gateway between the image apparatus (11) and the display apparatus (18). Thus, the cellular network comprises a first connection (22) between the image apparatus (11) and the base station (24) and a second connection (23) between the display apparatus (18) and the base station (24). The satellite network comprises a satellite (27) which acts as a gateway between the image apparatus (11) and the display apparatus (18). Thus, the satellite network comprises a third connection (25) between the image apparatus (11) and the satellite (27) and a fourth connection (26) between the display apparatus (18) and the satellite (27).

[0088] Here, we assume that the Wi-Fi network (21) is the primary wireless network, thus, by transmitting replicas of image frames over any the other wireless networks (22-24, 25-27), at least one replica image frame can be selected from any of the other wireless networks (22-24, 25-27) upon detecting any one or combination of issues in delay, data loss, and connectivity (e.g., power, quality, noise, interference, etc.) according to the present invention.Example 2: example of a method and / or system according to the invention

[0089] This example is described with reference to Fig. 2A. Fig. 2A shows a perspective view, particularly a top view, of a rail vehicle (30).

[0090] In this example according to embodiments of the invention, a system comprising a device for monitoring a data link in the rail vehicle (30) comprising at least one image apparatus (11) and a display apparatus wirelessly connected to the at least one image apparatus (11).

[0091] The at least one image apparatus (11) comprises a radar sensor having a first field of view (FoV) or sensor range (12), a LiDAR sensor having a second FoV (13), an infrared camera having a third FoV (14), and a laser camera having a fourth FoV (15).

[0092] The plurality of FoVs (12-15) are partially overlapping. At each time stamp, the image frames captured by said sensors may be transmitted separately over the wireless networks or transmitted as a stitched image frames over the wireless networks or as merged image frames over the wireless networks, according to the present invention.Example 3: example of a method and / or system according to the invention

[0093] In this example we consider the at least one sensor of Fig. 2A. This example is described with reference to Fig. 2B. Fig. 2B shows a perspective view, particularly a top view, of a rail vehicle (30).

[0094] In this example according to embodiments of the invention, a system comprising a device for monitoring a data link in the rail vehicle (30) comprising a plurality of image apparatuses (11, 11', 11") and a display apparatus wirelessly connected to the plurality of image apparatuses (11, 11', 11").

[0095] The plurality of image apparatuses (11, 11', 11") comprise respectively a first laser camera, a second laser camera and a third laser camera having a first FoV (15), a second FoV (15') and a third FoV (15"), respectively. As can be seen in Fig. 2B, the FoV of the three laser cameras is oriented in a different direction, particularly the first FoV is in the reversed direction of the rail vehicle (30), the second FoV is in the reversed direction of the rail vehicle (30) with a predefined angle in the clockwise direction, and the third FoV is in the reversed direction of the rail vehicle (30) with a predefined angle in the counter clockwise direction. Thus, the combination of the three cameras, a larger FoV can be obtained. This may be done by merging the image frames captured by the cameras at each time stamp, preferably, where the overlapping FoVs may be merged, e.g. based on the most similar pixel pairs in the respective pair of image frames.

[0096] At each time stamp, the image frames captured by said sensors may be transmitted separately over the wireless networks or transmitted as a stitched image frame over the wireless networks or as merged image frames over the wireless networks, according to the present invention.(End of Example 3)

[0097] Although the present invention has been described above with reference to certain embodiments thereof, it will be evident that various modifications and changes may be made to these embodiments without departing from the broader scope of the present invention, as defined by the appended claims.

Examples

example 3

(End of Example 3)

Claims

1. A method of monitoring a data link in a rail vehicle comprising at least one image apparatus and a display apparatus, the method comprising: a. monitoring the data link between the at least one image apparatus and the display apparatus over a plurality of wireless networks, wherein the data link comprises a transmission of image frames captured with the at least one image apparatus, wherein the transmission of image frames is replicated over the wireless networks, and b. generating an alert upon detecting that a time interval between consecutive image frames over each or any of the wireless networks exceeds a first predetermined threshold relating to an acceptable delay.

2. The method according to claim 1, wherein the transmission of the image frames comprises the transmission of data packets of the image frames, wherein step 1.b further comprises generating the alert: a. upon detecting that a round-trip time of a data packet from among the data packets over each of the wireless networks exceeds a second predetermined threshold relating to an acceptable round-trip delay, and / or b. upon detecting that a loss of data packets over each of the wireless networks exceeds a third predetermined threshold relating to an acceptable degree of pixelation.

3. The method according to claim 1 or claim 2, wherein step 1.b further comprises generating the alert: a. upon detecting that a received signal strength exceeds a fourth predetermined threshold relating to an acceptable power level the display apparatus is receiving over each of the wireless networks, and / or b. upon detecting that a reference signal received power exceeds a fifth predetermined threshold relating to an acceptable power level of the reference signal in each of the wireless networks, and / or c. upon detecting that a reference signal received quality exceeds a sixth predetermined threshold relating to an acceptable quality level of the reference signal in each of the wireless networks, and / or d. upon detecting that a signal-to-interference and / or noise-ratio exceeds a seventh predetermined threshold relating to an acceptable power level of the signal to the power level of background interference and / or noise in each of the wireless networks.

4. The method according to any one of claims 1-3, wherein the at least one image apparatus comprises a plurality of image sensors.

5. The method according to claim 4, wherein the transmission of image frames comprises stitching the image frames from the image sensors at each time stamp.

6. The method according to claim 4 wherein the images frames from the image sensors are stitched at each time stamp after transmission.

7. The method according to any one of claims 1-6, wherein step 1.b comprises resetting a real-time streaming protocol of the at least one image apparatus.

8. The method according to any one of claims 1-7, wherein the wireless networks comprise a primary wireless network and a plurality of secondary wireless networks, wherein the method further comprises: upon detecting that the time interval between consecutive image frames over the primary wireless network exceeds the first predetermined threshold, selecting a replica of the successive image frame transmitted over any of the secondary wireless networks.

9. The method according to claim 8, wherein the selecting of a replica of the successive image frame comprises: - detecting from among the plurality of secondary wireless networks the secondary wireless network having the smallest detected time interval between respective replicas of the consecutive image frames transmitted over said secondary wireless network; and - selecting the replica of the successive image frame transmitted over said secondary wireless network having the smallest detected time interval between respective replicas of the consecutive image frames.

10. The method according to claim 9, wherein the secondary wireless network having the smallest detected time interval between the respective replicas of the consecutive image frames is set as the primary wireless network.

11. The method according to any one of claims 1-10, wherein the rail vehicle is being used in a shunting operation.

12. A device comprising means for carrying out the method of any one of claims 1-11.

13. A system comprising: a device according to claim 12; at least one image apparatus; and a display apparatus wirelessly connected to the at least one image apparatus, wherein the device is configured to perform the steps of: a. monitoring the data link between the at least one image apparatus and the display apparatus over a plurality of wireless networks, wherein the data link comprises a transmission of image frames captured with the at least one image apparatus, wherein the transmission of image frames is replicated over the wireless networks, and b. generating an alert upon detecting that a time interval between consecutive image frames over each or any of the wireless networks exceeds a first predetermined threshold relating to an acceptable delay.

14. A computer program comprising instructions which, when the program is executed by a computer, cause the computer to carry out the method of any one of claims 1-11.

15. A computer-readable medium comprising instructions which, when executed by a computer, cause the computer to carry out the method of any one of claims 1-11.

Images