Computer-implemented method for determining at least one element under consideration, as well as technical system and computer program product

Abstract

The invention relates to a computer-implemented method for determining at least one element under consideration, comprising the steps: a. providing at least one data element (S1); wherein the at least one data element comprises at least one element (A, B, C, D), the at least one element (A, B, C, D) being located or moving on a route or section of a route of at least one means of transport; b. providing information (S2), wherein the information relates to the at least one element (A, B, C, D); wherein the information includes location information; c. determining a plurality of gaze vectors (1, 2, 3, 4) in a temporal sequence for the gaze directions of a person with respect to the at least one element (A, B, C, D) based on the at least one data element and the information by means of a graphic data processing system (S3); d.Analyzing the majority of gaze vectors (1, 2, 3, 4) with respect to at least one temporal superposition of the gaze vectors (S4); e. Identifying the at least one viewed element (B) from the at least one element (A, B, C, D) taking into account the at least one temporal superposition (S5); wherein the at least one viewed element (B) is viewed by the person over a certain period of time; and f. Providing the at least one viewed element (B) (S6). Furthermore, the invention relates to a technical system and a corresponding computer program product.

Description

1. Technical field

[0001] The invention relates to a computer-implemented method for determining at least one element under consideration. Furthermore, the invention relates to a corresponding technical system and a computer program product. 2. State of the art

[0002] Autonomous driving is becoming increasingly important. Various autonomous vehicles, such as cars and trains, are already known in this context as state-of-the-art technology. The degree of automation is also steadily increasing.

[0003] As autonomous trains and their control systems are further developed, train control is being gradually transferred from the driver to a technical system with automated control (also called train control). Obstacles on the track still pose serious risks to rail traffic. Drivers sometimes have to react very quickly to prevent major damage to the train and passengers. Obstacles can include parts of track infrastructure damaged by severe weather, such as overhead lines or masts, but also fallen trees or people. Reliable automated obstacle detection and the initiation of appropriate countermeasures, such as emergency braking, remain a significant challenge.

[0004] Train drivers are typically trained specifically for certain routes to ensure safe train operation. In addition to observing signals and signs along the route, drivers must also consider additional route-specific features to further enhance safety. These features can include typical hazards for pedestrians, changing weather conditions, or temporary construction sites. Training new drivers for a particular route therefore requires significant personnel resources, as experienced drivers must pass on their knowledge of these specific route characteristics or elements. These route elements are, accordingly, those elements to which an experienced driver pays particular attention on a given route.The disadvantage, however, is that this effort is associated with high costs and increased time expenditure. Furthermore, the deployment of the train drivers is limited to the routes they have been trained on and is therefore very restricted.

[0005] The recording and digitization of track elements remains a major challenge. This information is often not included in existing data systems, as it is not necessarily evident from official signage. The lack of digitized data on these critical track elements hinders the efficient training of new train drivers and the development of advanced driver assistance systems or autonomous train systems with automated control.

[0006] Conventional methods for capturing attentional focus areas, such as eye-tracking technologies, are only partially suitable for the dynamic environment of a moving train. These established methods are generally designed for static environments and cannot adequately capture the continuous movement and changes in the surroundings during a train journey. This significantly complicates the creation of meaningful heatmaps or attention analyses.

[0007] Therefore, there is a need for a method for capturing and analyzing eye-tracking data from a train driver that overcomes the aforementioned disadvantages.

[0008] The publication DE 10 2020 110 417 A1 discloses a method for assisting a driver. First, an intersection point 7 of the driver's gaze direction 6 with a virtual plane 8 is determined, which is tracked for a predefined period of time. This allows the degree of cognitive distraction of the driver 5 to be determined.

[0009] Publication EP 4 385 854 A1 discloses a computer-implemented method for providing a maximum speed for a train.

[0010] The present invention therefore aims to provide a computer-implemented method for determining at least one element under consideration, which is more efficient and reliable. 3. Summary of the invention

[0011] The above-mentioned problem is solved according to the invention by a computer-implemented method for determining at least one element under consideration, comprising the steps: a. Providing at least one data element; wherein which has at least one data element, wherein the at least one element is located or moving on a route or section of a route of at least one means of transport; b. Providing information, whereby the information relates to at least one element; wherein the information includes location information; c. Determining a plurality of gaze vectors in a temporal sequence for gaze directions of a person in relation to the at least one element based on the at least one data element and the information by means of a graphical data processing; d. Analyzing the majority of the viewing vectors with regard to at least one temporal superposition of the viewing vectors; e. Identifying the at least one element under consideration from the at least one element, taking into account the at least one temporal overlap; wherein the at least one element under consideration is viewed by the person over a specific period of time; and f. Providing the at least one element under consideration.

[0012] Accordingly, the invention relates to a computer-implemented method for identifying at least one element under consideration. Consequently, an element is identified that is already present and is classified as being considered or observed. The element is observed by the person over a specific period of time. This period can also be interpreted as a duration.

[0013] The preferred elements are track features or track elements. Track elements are those elements to which an experienced train driver pays particular attention or consideration on a route, as explained above. The data elements represent various elements such as objects and features that can be encountered along a track or route of the means of transport. Examples include signals, signs, construction sites, bridges, trees, barriers, rivers, embankments, and boundary fences.

[0014] In other words, the elements that capture the person's attention and are therefore relevant during the journey along the route or section of the transport vehicle are analyzed and identified.

[0015] In the first and second process steps, the input data is provided. The input data comprises the data element and the information. Preferably, a plurality of data elements is provided, each data element correspondingly comprising one or more elements. The data elements are preferably in the form of image data. The elements are each arranged along the route or section of the route of the means of transport. The elements are accordingly located in the environment or surroundings of the means of transport. The elements can move within the environment and / or be in continuous motion.

[0016] Each element is assigned location information. This location information can be understood as positional information. Location determination, also called localization, enables the assignment to a specific spatial location. Through location determination, the element's position along the route of the means of transport can be precisely determined. The position of the elements under consideration is usually moving. The input data is preferably in digital form as a digital map of the surroundings.

[0017] Input data can be received via one or more input interfaces. Additionally or alternatively, output data, such as the element, can also be sent via one or more output interfaces. The interfaces can be configured as serial or parallel interfaces. Advantageously, the interfaces ensure efficient and smooth data transmission between computing units. Data can be exchanged bidirectionally without data congestion.

[0018] In a further process step, the majority of gaze vectors are determined in temporal sequence for the person's gaze directions in relation to the element using graphical data processing. The person is preferably a train driver of the means of transport. Alternatively, the person can also be an operator, a driver, or a passenger of the means of transport. The person can be a human or a computing unit, e.g., in the form of a display unit. The determination is based on the input data. The gaze vectors can be represented as three-dimensional vectors that originate from the person's eyes and extend in the direction of their gaze. Each gaze vector can be assigned a timestamp. This enables the creation of a temporal sequence of the person's gaze directions.The gaze vectors, in conjunction with the location information, allow the correlation of the person's gaze direction with specific locations along the route of the means of transport.

[0019] For example, during the operation of the transport vehicle along its route, viewing vectors can be recorded at regular intervals, such as every second or every few seconds. Each recorded viewing vector can be linked to location information. Furthermore, each viewing vector can be linked to a timestamp. This data can be processed and mapped onto the digital representation of the route, creating a sequence of viewpoints that correspond to specific locations and times during the transport vehicle's journey.

[0020] The resulting temporal sequence of gaze vectors, mapped onto the digital representation of the route, provides valuable information about where the person was looking at different times during the journey. This data can be used for various purposes, such as analyzing attention patterns, identifying areas of interest along the route, or supporting training and safety applications for train drivers.

[0021] The viewing vectors are analyzed to determine the temporal superposition of the majority of the viewing vectors. The element under consideration is identified based on this temporal superposition and provided as output in the final step of the process.

[0022] In other words, identifying viewed elements from the elements in the digital map involves analyzing the viewing vectors with respect to their temporal superposition. An element can be determined as viewed over a specific period based on the intersection of multiple viewing vectors with the position of that element in the digital map across successive time points. Preferably, the viewing vectors are extended for this purpose.

[0023] For example, if multiple view vectors intersect the same element in the digital map over several consecutive time points, this element can be identified as being viewed for the duration of those time points. Temporal overlay allows for consideration of the temporal sequence of view vectors and makes it possible to distinguish between brief glances and sustained observations.

[0024] Different path elements can be identified as being viewed based on different patterns in the gaze vector analysis. For example: 1. A signal could be identified as having been viewed if several viewing vectors overlap with its position in rapid succession. This indicates that the train driver has checked the signal several times in a short period of time. 2. A construction site could be identified as being under observation if the gaze vectors consistently overlap with the position of the construction site over a longer period. This indicates sustained attention from the train driver. 3. A bridge or tunnel entrance could be identified as being viewed if the line of sight vectors intersects with the position at regular intervals as the train approaches. This indicates repeated checks by the train driver.

[0025] If multiple data elements are provided as input data, the elements can be divided into those considered and those not considered, and the results provided as output. The considered elements can be specially marked.

[0026] The method according to the invention is advantageously applicable to the dynamic environment of a moving means of transport and enables the efficient and reliable detection of focal points. In contrast to the prior art, a static environment is not required for identifying the element under consideration; rather, it can be in motion. The continuous movement and changes of the environment during the operation of the means of transport are detected.

[0027] In other words, this allows for the precise identification and tracking of route elements in relation to the route of the means of transport.

[0028] Another advantage is the significant improvement in the operation of the transport vehicle and its safety. In some cases, experienced train drivers can receive written comments or instructions regarding identified track features. These notes can serve as valuable guidance for less experienced drivers or for training purposes. For example, an experienced driver might receive a note about a particular curve that requires extra caution in rainy conditions.

[0029] Unless otherwise specified, all steps of the computer-implemented method can be performed by at least one computing unit, which can also be referred to as a data processing device. In particular, the data processing device, which comprises at least one processing circuit configured or adapted to carry out a computer-implemented method according to the invention, can perform the steps of the computer-implemented method. For this purpose, a computer program can be stored in the data processing device, in particular one containing instructions which, when executed by the data processing device, in particular the at least one processing circuit, cause the data processing device to execute the computer-implemented method.

[0030] In one embodiment, the location information is geoinformation, preferably in the form of GPS coordinates. Accordingly, the GPS coordinates enable the GPS tracking of the elements. Alternatively, other tracking systems such as Galileo or BeiDou can be used. GPS tracking has proven particularly advantageous for reliably assigning each element to a specific spatial location. In other words, the GPS coordinates determine the location or position of the element on the route or section of the route of the means of transport.

[0031] In a further configuration, the information also includes temporal relevance, at least one note, at least one warning, at least one type, at least one color, at least one height, and / or at least one size. Accordingly, the information can be supplemented with additional data. This additional data is taken into account in the process steps based on the input data and advantageously increases the reliability of identifying the element under consideration. The temporal relevance could be the duration of a construction site. The environment of the means of transport can therefore be depicted and considered in detail.

[0032] In a further development, the graphical data processing is based on 3DinSight. The 3DinSight method advantageously enables the determination of the block direction and the person's eye position based on data elements, including those that can be in motion. In other words, 3DinSight allows for the determination of gaze vectors in a temporal sequence for the person's gaze direction. This determination is based on the acquisition and processing of data relating to eye movements, head position, and posture of the person in relation to the position of the means of transport and the means of transport in relation to the map.

[0033] In a further embodiment, the computer-implemented method also exhibits - Extending the respective viewing vectors of the majority of the viewing vectors; and - Determining at least one temporal superposition of the majority of the extended viewing vectors.

[0034] Accordingly, the viewing vectors are extended. This extension step results in extended viewing vectors, which are used to determine at least one temporal superposition.

[0035] In a further refinement, the majority of viewing vectors are determined while the transport vehicle is in operation. Accordingly, the process can be carried out reliably and efficiently at any time while the vehicle is in motion. Consequently, there is absolutely no need to stop or halt the operation of the transport vehicle to perform the computer-implemented process. The advantage lies in the fact that the process incurs no downtime and / or costs. Operation can continue undisturbed and without any disruption.

[0036] In a further embodiment, the means of transport is an autonomous means of transport, preferably an autonomous train.

[0037] In another version, the person is a train driver of the means of transport.

[0038] Accordingly, the means of transport is an autonomous train with a driver who travels on the track or section of track. The driver observes the element, which is also located on the track or section of track, either visually or via a display unit such as a head-up display.

[0039] In a further embodiment, the computer-implemented method also exhibits - Generating a heatmap taking into account the at least one element under consideration.

[0040] Accordingly, a heatmap is created by adding the element under consideration. In other words, a heatmap can be created with observation points on a digital map, preferably in three dimensions. The observation points are preferably traversed by the train driver during a specific period of time.

[0041] In other words, a heatmap of the viewed elements is created by aggregating the viewed elements over time. The heatmap can be spatially defined. Elements that are viewed frequently or consistently have a higher concentration of overlapping viewing vectors, resulting in hot spots on the heatmap. This heatmap provides a visual representation of which elements in the vicinity of the vehicle receive the most attention from the train driver and allows for the identification of potentially important or safety-critical viewed elements along the route of the vehicle.

[0042] In a further embodiment, the computer-implemented method also exhibits - Analyzing the at least one element under consideration; - Supplementing the at least one element under consideration with further data; - Releasing the at least one element under consideration; and / or - Adapting the means of transport, a unit of the means of transport, an application of the means of transport depending on the element under consideration.

[0043] Accordingly, the element under consideration can be subjected to more detailed analysis. For example, it can be examined where the element is located on the route of the means of transport, over what period of time the element is being observed, and / or what type of element it is, etc. The element under consideration can be highlighted or marked to simplify the analysis. The analysis can be performed manually or automatically, for example, by the train driver or an application on a computer. The element under consideration can be supplemented with data such as the route, the type and state of the element, the period of observation, the actions performed by the train driver during or in immediate succession during the observation, and / or the responsible train driver, etc. The element under consideration can be released and thus confirmed, and / or rejected and thus discarded.For example, after a thorough analysis, the train driver can confirm and release the element under consideration.

[0044] Furthermore, adjustments can be made. For example, the control of the means of transport can be improved by taking the element under consideration into account. The control unit (or train control system) of the autonomous train can additionally or alternatively initiate appropriate measures, such as changing the timetable or adjusting the speed of the autonomous train. Adjusting the speed can also result in a change to the timetable.

[0045] In a further embodiment, the computer-implemented method also exhibits - Outputting the at least one element under consideration and / or associated data on a display unit, - Storing the at least one considered element and / or associated data in a storage unit, and / or - Transmitting the at least one element under consideration and / or associated data to a computing unit.

[0046] Accordingly, one or more measures can be initiated after the element under consideration has been provided as an output of the method according to the invention. The measures can be carried out simultaneously, sequentially, or stepwise.

[0047] Accordingly, the user, such as a train driver, can see the element under consideration displayed on a screen of a processing unit. This element can be supplemented with further output data, such as information about the means of transport, routes, etc. Furthermore, the output can be stored. The storage unit can be a database, a cloud, or other volatile or non-volatile storage device. The output can be transmitted directly or as a corresponding message or notification to another unit, such as a terminal device or other processing unit.

[0048] Furthermore, the invention relates to a technical system for carrying out the above method.

[0049] The invention further relates to a computer program product comprising a computer program which includes means for carrying out the method described above when the computer program is executed on a program-controlled device.

[0050] A computer program product, such as a computer program tool, can be provided or delivered from a server on a network, for example, as a storage medium such as a memory card, USB stick, CD-ROM, DVD, or as a downloadable file. This can be done, for example, in a wireless communication network by transmitting the corresponding file containing the computer program product or tool. A suitable program-controlled device is, in particular, a control unit such as an industrial control PC, a programmable logic controller (PLC), or a microprocessor for a smart card or similar device. 4. Brief description of the drawings

[0051] In the following detailed description, preferred embodiments of the invention are further described with reference to the following figures. Fig. Figure 1 shows a schematic flowchart of the method according to the invention. Fig. Figure 2 shows a schematic view of the viewing vectors in a temporal sequence with respect to the elements according to an embodiment of the invention. 5. Description of preferred embodiments

[0052] Preferred embodiments of the present invention are described below with regard to the Fig. 1 described.

[0053] Fig. Figure 1 schematically represents a flowchart of the method according to the invention, comprising process steps S1 to S6. In the first process step, the at least one data element S1 is provided. This at least one data element comprises the at least one element A, B, C, D. The at least one element A, B, C, D is located on a route 10 or a section of the route of the at least one means of transport. In the second process step, the information is provided. The information relates to the at least one element A, B, C, D and includes location information. Furthermore, the majority of gaze vectors 1, 2, 3, 4, in the temporal sequence 0, 1, 2, 3, for the gaze directions of the person with respect to the at least one element A, B, C, D, are determined based on the at least one data element and the information by means of graphical data processing S3.Furthermore, the majority of gaze vectors are analyzed with regard to at least one temporal superposition of the gaze vectors (S4). Furthermore, the at least one considered element B is identified from the at least one element A, B, C, D, taking into account the at least one temporal superposition (S5). The at least one considered element B is viewed by the person over the specified period. The at least one considered element B is provided (S6).

[0054] Identifying the element under consideration B: The gaze vectors 1, 2, 3, 4 can be recorded during the journey using the vehicle's location information and arranged in a temporal sequence 0, 1, 2, 3. By extending the gaze vectors 1, 2, 3, 4 and combining the extensions of the preceding and following gaze vectors 1, 2, 3, 4, a specific element (B) can be identified as being viewed for a given period. This viewed element (B) from the majority of elements A, B, C, D can also be highlighted. Furthermore, a heatmap of the viewed elements (B) can be generated.

[0055] Fig. Figure 2 shows a schematic view of the identification of a considered element B from a plurality of elements A, B, C, D according to an embodiment of the invention.

[0056] The location information is in Fig. 2 illustrated by arrows. Fig. Figure 2 shows four different viewing vectors 1, 2, 3, 4 and their extensions in the temporal sequence 0, 1, 2, 3 with t=0 to t=3. Furthermore, it shows Fig. 2 four elements A, B, C, D. The elements A, B, C, D are represented by boxes or blocks.

[0057] The temporal superposition of the extended viewing vectors 1, 2, 3, 4 reveals that element B, out of the four elements A, B, C, D, was viewed over three time units from t=0 to t=3. Elements A and D, however, were not viewed. Viewing element C appears possible by considering the extended viewing vectors at times t=0 and t=1 individually. However, the analysis can be limited to element B by also including the viewing vectors at times t=2 and t=3.

[0058] Consequently, a single viewing point can be identified for a time period from t=0 to t=3, or four separate viewing points for t=0, t=1, t=2 and t=3 on element B.

[0059] In this way, evaluating the gaze vectors 1, 2, 3, 4, and including their time points 0, 1, 2, 3 and the location information of elements A, B, C, D, generates a heatmap of the viewing points. The heatmap represents the three-dimensional area traversed by the train driver over a specific period and provides valuable insights into which elements in the environment receive the most attention. Use cases:

[0060] The element under consideration and / or the heatmap can be used in different ways. • A new train driver, who has been trained for a route but has not yet driven it frequently, can be alerted to specific elements by means of a display unit or computing device such as a head-up display. These elements are those to which the driver should pay increased attention or which are of particular importance or relevance. • Experienced train drivers can add comments or notes to the special elements, for example in written form, and thus support the virtual training of new train drivers. • The identified elements and knowledge gained can be transferred to routes not yet evaluated with observation points, for example using artificial intelligence, and help to identify further special elements. • For autonomous driving systems, attention control of the technical system can be dynamically carried out based on the element under consideration, such as adjusting the resolution of the imaging sensors for this area or prioritizing these areas for processing the sensor data. • Prioritization of training and test data of the technical systems with information from the perspective of the train driver. • Special elements that a train driver might not see could pose a hazard to safe train operation. If a train driver misses seeing a special element, they can be alerted to it. For example, the element could be highlighted or flashing on a display unit or computer system, such as a head-up display, to draw the driver's attention to it.

Claims

[1] Computer-implemented method for determining at least one element (B) under consideration, wherein the at least one element (B) under consideration is a line segment element; comprising the steps: a. Providing at least one data element (S1); wherein the at least one data element has at least one element (A, B, C, D) wherein where at least one element (A, B, C, D) is located or moves on a route (10) or a section of a route of at least one means of transport; wherein that at least one means of transport is an autonomous train; b. Providing information (S2), wherein The information relates to at least one element (A, B, C, D); the information includes location information; c. Determining a plurality of gaze vectors (1, 2, 3, 4) in a temporal sequence for gaze directions of a person in relation to the at least one element (A, B, C, D) based on the at least one data element and the information by means of a graphical data processing (S3); wherein the person is a train driver of at least one means of transport; d. Analyzing the majority of the gaze vectors (1, 2, 3, 4) with regard to at least one temporal superposition of the gaze vectors (S4); e. Identifying the at least one element under consideration (B) from the at least one element (A, B, C, D) taking into account the at least one temporal superposition (S5); wherein that at least one element (B) is viewed by the person over a certain period of time; and f. Providing the at least one element under consideration (B) (S6). [2] Computer-implemented method according to claim 1, wherein the location information is geoinformation, preferably comprising GPS coordinates. [3] Computer-implemented method according to claim 1 or claim 2, wherein the information further comprises a temporal relevance, at least one hint, at least one warning, at least one type, at least one color, at least one height and / or at least one size. [4] Computer-implemented method according to one of the preceding claims, wherein the graphical data processing (S3) is based on 3DinSight. [5] Computer-implemented method according to any of the preceding claims, wherein the analysis comprises - Extending the respective viewing vectors of the majority of the viewing vectors (1, 2, 3, 4); and - Determining at least one temporal superposition (S5) of the majority of the extended viewing vectors (1,2,3,4). [6] Computer-implemented method according to one of the preceding claims, wherein the determination of the majority of the viewing vectors (1, 2, 3, 4) is carried out during the operation of the means of transport. [7] Computer-implemented method according to any one of the preceding claims, further comprising - Generating a heatmap taking into account the at least one element under consideration (B) . [8] Computer-implemented method according to any one of the preceding claims, further comprising - Analyzing the at least one element under consideration (B); - Supplementing the at least one element (B) under consideration with further data; - Releasing the at least one element under consideration (B); and / or - Adapting the means of transport, a unit of the means of transport, an application of the means of transport depending on the element under consideration. [9] Computer-implemented method according to any one of the preceding claims, further comprising - Outputting the at least one considered element (B) and / or associated data on a display unit, - Storing the at least one considered element (B) and / or associated data in a storage unit, and / or - Transmitting the at least one element (B) under consideration and / or associated data to a computing unit. [10] Technical system for carrying out the method according to any of the preceding claims. [11] Computer program product comprising a computer program comprising means for carrying out the method according to any one of claims 1 to 9 when the computer program is executed on a program-controlled device.

Images