COMPUTER-IMPLEMENTED METHOD FOR ASSIGNING A MOBILE TERMINAL DEVICE TO A ROAD USER

Abstract

A computer-implemented method 132 for assigning 144 a mobile terminal 116 in a road section 100 to a road user 102, 104 in the road section 100 comprises a first obtaining step 134 in which a first data set 150 is obtained which is assigned to the mobile terminal 116. In a second obtaining step 138, at least one second data set 146, 148 is obtained in which the road section 100 is at least partially recorded. In a first calculation step 136, a first trajectory 160 is calculated from the first data set 150. In a second calculation step 140, a plurality of second trajectories 159 are calculated from the at least one second data set 146, 148. In a third calculation step 142, deviations between the first trajectory 160 and the plurality of second trajectories 159 are calculated.In an assignment step 144, the first trajectory 160 is assigned to that second trajectory from the plurality of second trajectories 159 which has a minimal deviation from the first trajectory 160.

Description

Technical FieldThe present invention relates to the field of sensor data fusion. In particular, the present invention relates to a computer-implemented method and apparatus for assigning a mobile terminal in a road segment to a road user in the road segment. The present invention further relates to a computer program product, a computer-readable storage medium and a data carrier signal.Technical Background and ObjectOn many road sections such as intersections, exits or the like, an infrastructure for monitoring the road users there now exists. This infrastructure is implemented, for example, in the form of cameras or radar sensors. In the future, these stationary forms can also be supplemented by mobile devices such as drones.These infrastructure sensors can record the movement data such as position (latitude and longitude), speed, direction of movement or type (motor vehicle, cyclist, lorry, pedestrian, etc.) of the road users. For example, by image recognition of camera images. It is also possible to send this data to a backend such as a server. From the data, possible movement trajectories for all detected road users for the near future (for example up to 4 s) can be calculated in the backend.In addition, many road users have mobile terminals. Mobile terminals are portable communication devices which can be used for voice and data communication in a location-free manner, such as mobile telephones, smartphones, smart watches, netbooks, notebooks or tablets. These devices can determine their own position, for example as GNSS (global navigation satellite system) data. Within the scope of systems for warning vulnerable road users, this data can likewise be sent to a backend. As with the data of the infrastructure sensors, possible movement trajectories of the mobile terminals for the near future can be calculated from the GNNS data in the backend. In order to avoid accidents between road users, in particular vulnerable road users, a prediction is calculated in the backend from the calculated trajectories as to whether a dangerous situation, such as a collision, will occur for a road user in the near future. The affected road users can be warned in this case.The data of the infrastructure sensors offer the advantage that they provide an almost complete image of a road segment. In addition, the accuracy of the data with respect to the position of the road users is very good. The disadvantage of these data is that there is no direct possibility of informing the road user concerned about a danger.The data of the mobile terminals, on the other hand, offer the advantage that it is possible to send a warning exactly to the mobile terminal for which the risk was determined. This can then be displayed specifically to the road user concerned. However, it is disadvantageous that not all road users in a road segment generally have a mobile terminal or the data of which are not made available to a system for warning vulnerable road users. Thus, a participant in the system can only be warned of a collision if the possible accident partner also participates in the system. In addition, the accuracy of the position relating to mobile terminals is usually not very high and worse than that of the infrastructure sensors.Moreover, systematic deviations can occur in the data, such as, for example, a time delay in the data flow from the sensor to the backend, deviations in the conversion into a global coordinate system or a delayed reaction of sensors to changes (for example, by Kalman filters). In addition, the data is subject to statistical errors such as noise.In "High-performance spatiotemporal trajectory matching across heterogeneous data sources" (Xuri Gong, Zhou Huang, Yaoli Wang, Lun Wu, Yu Liu; In future generation computer systems, 105, 27.11.2019, 148-161, https: / / doi.org / 10.1016 / j.future.2019,11.027) and "Phone-vehicle trajectory matching framework based on ALPR and cellular signaling data" (Wei Wan, Ming Cai in IET Intelligent Transport Systems, 15, 25.11.2020, 1, 107-118, https: / / doi.org / 10.1049 / itr2.12008), it is disclosed how to achieve matching of spatiotemporal trajectories across heterogeneous data sets. For this purpose, distances between the data sets are calculated.DE 10 2019 208 424 A1 relates to a communication system having a motor vehicle, a vehicle-external portable communication adapter which can be reversibly coupled to a control device of the motor vehicle by means of a communication interface, and a vehicle-external coordination device. According to the invention, the communication adapter is configured to receive motor vehicle data describing a traffic behavior and / or a traffic environment of the motor vehicle from the control device, wherein the communication adapter has a transceiver unit and is configured to transmit the motor vehicle data to the vehicle-external coordinating device on the one hand by means of the transceiver unit and to receive warning data generated by the vehicle-external coordinating device and describing a traffic behavior and / or a traffic environment of at least one other motor vehicle on the other hand and to trigger at least one traffic-coordinating measure on the basis of the warning data.It is therefore desirable to have available a system which uses both data sources, for which purpose the data of the mobile terminals must be assigned to those of data of the infrastructure sensors. However, due to the low positional accuracy in the data of the mobile terminals, this can lead to incorrect assignments, as a result of which unnecessary or even missing warnings can occur.It is therefore the object of the present invention to provide a computer-implemented method and a device for assigning a mobile terminal in a road segment to a road user in the road segment, which eliminate at least one of the aforementioned disadvantages. It is a further object of the invention to provide a computer program product, a computer-readable storage medium and a data carrier signal.Disclosure of the InventionThe object is achieved according to the invention by the features of the main claims. Advantageous embodiments can be gathered from the dependent claims.According to a first aspect of the invention, a computer-implemented method for assigning a mobile terminal in a road segment to a road user in the road segment comprises a step in which a first data record is obtained which is assigned to the mobile terminal.A road section is a part of a road to be delimited or characterized by specific properties. A road section may correspond to an intersection or an exit road, for example. It may include, for example, roads, lanes, traffic lights, sidewalks, and other objects. It can be understood as a geographical sub-area of a map. Road users are, for example, motor vehicles, cyclists, trucks, pedestrians or the like. Mobile terminals are portable communication devices which can be used for voice and data communication in a location-free manner, such as mobile telephones, smartphones, smart watches, netbooks, notebooks or tablets.The first data record can correspond, for example, to a time series of data which permit conclusions about the position of the mobile terminal at the respective time. The first data record can be assigned to the mobile terminal, for example because it originally originates from the latter. The first data record can be present, for example, in the form of GNSS (global navigation satellite system) data.In a further step of the method, at least one second data record is obtained in which the road segment is at least partially detected. The at least one second data record can be present, for example, in the form of image data of a camera arranged in the road section and form a time series like the first data record. In other words, the at least one second data record can consist of an image series at different points in time, for example at intervals of 0.1 s.In an advantageous embodiment, the first data record and / or the at least one second data record has at least one of the following values: a time stamp, a position value, a speed value, an identifier, a movement direction value. These values are particularly useful for calculating a trajectory or for carrying out a unique assignment.According to the first aspect of the invention, the method has a step in which a first trajectory is calculated from the first dataset. If the first data record is present as GNSS data, this is particularly simple since time stamps and position are transmitted.The method further comprises a step in which a plurality of second trajectories is calculated from the at least one second dataset. For example, in image data of the road segment, a plurality of road users can be detected by means of known object recognition methods. For each of the detected road users, a position with time stamp can be determined, resulting in the trajectory of the road user.In an advantageous embodiment, the at least one second data set comprises image data and the plurality of second trajectories are calculated from the image data by object recognition. The skilled person is familiar with many known methods from the fields of computer vision, machine learning or pattern recognition.According to the first aspect of the invention, the method comprises a step in which deviations between the first trajectory and the plurality of second trajectories are calculated. For example, differences in positions at equal times may be calculated. For this purpose, it may be necessary to interpolate positions first between points in time if no data with an identical point in time are present for comparison. The deviation can then be calculated as a sum over the differences at the individual points in time.A weighted difference as the sum of the squares, for example of position, direction of movement and speed, is also possible. In this case, the differences in position, direction of movement and speed are squared, normalized and added up.The method further comprises a step in which the first trajectory is assigned to that second trajectory from the plurality of second trajectories which has a minimum deviation from the first trajectory.The method is calculated by a neural network trained for the road section, wherein the first data set and the at least one second data set serve as input for the neural network.In an advantageous embodiment, the method steps described above are carried out for a multiplicity of mobile terminals in the road segment. In other words, a plurality of first data sets are processed. This can be done in parallel or sequentially.The method described here can be realized, for example, by a neural network specifically trained for a road section, which implicitly corrects systematic errors. The first data record or a plurality of first data records and the at least one second data record serve as input for the neural network. The output then consists of an assignment of the first trajectory(s) to a second trajectory, for example as a pair of identifiers.In an advantageous embodiment, deviations are calculated only between the first trajectory and the second trajectories from the plurality of second trajectories if no previously defined exclusion criterion is fulfilled.Such exclusion criteria can be, for example:the traffic participant type does not match;for at least one time stamp in the first data record, the difference of the (interpolated) position to the (interpolated) position of a second trajectory is above a limit value;for at least one time stamp in the first data record, the difference of the (interpolated) speed to the (interpolated) speed of a second trajectory is above a limit value;for at least one time stamp in the first data record, the difference of the (interpolated) direction of movement to the (interpolated) direction of movement of a second trajectory is above a limit value;In other words, before the calculation of a deviation, it is checked whether the data on which the trajectories to be compared are based exhibit an excessively high deviation in one of their values.According to a second aspect of the invention, a computer program product comprises instructions which, when the program is executed by a computer, cause the computer to execute a computer-implemented method as described above. The computer program product is typically written in a programming language such as Python or C++.According to a third aspect of the invention, a computer readable storage medium comprises instructions which, when executed by a computer, cause the computer to execute a computer implemented method as described above. The computer readable storage medium is typically a nonvolatile memory such as a solid-state disk (SSD) or a flash memory.According to a fourth aspect of the invention, a data carrier signal transmits the computer program product as described above. The transmission can be effected in a cable-bound manner, for example by a CAN-BUS. Wireless transmissions using WLAN (wireless local area network), mobile radio such as 5G or 6G, or Bluetooth are also possible.According to a fifth aspect of the invention, a device for assigning a mobile terminal in a road segment to a road user in the road segment has an evaluation unit. The evaluation unit is designed to execute a method as described above.The evaluation unit is formed, for example, with a processor, a nonvolatile memory and a volatile memory. It has input and output devices and connections in order to ensure data communication with the other units which are necessary for carrying out a method as described above. The evaluation unit can be realized as a single component or as a distributed component. For example, a non-volatile memory can be implemented as a server.Furthermore, the device has at least one sensor unit which is communicatively connected to the evaluation unit. The sensor unit can consist of several sensors. These are usually arranged in a fixed position in the road section. The sensor unit can be realized as a single component or as a distributed component.The mobile terminal is communicatively connected to the evaluation unit. This can be implemented, for example, by mobile radio. In other words, the mobile terminal is able to make data available to the evaluation unit either directly or indirectly, for example via a server, by means of mobile radio or other wireless communication technologies such as Bluetooth.The device may be at least partially in a vehicle. Parts of some units can be used by components already present in the vehicle. For example, a central computer in the form of an HPC (high performance computer) can be used as the evaluation unit. Preferably, however, the evaluation unit is not arranged in a vehicle, but rather is designed as a central server.In an advantageous embodiment, the sensor unit has at least one sensor from the following group: a camera, a radar sensor, a lidar sensor. The data of these sensors are particularly suitable for detecting the road section with them and for determining movement trajectories of the road users located therein from them.Summary of the FiguresThe invention is explained in more detail below with reference to exemplary embodiments with the aid of figures. The figures show: FIG. 1 : An overview of a computer-implemented method for assigning a mobile terminal in a road segment to a road user in the road segment; FIG. 2 : shows an exemplary embodiment of a device for assigning a mobile terminal in a road segment to a road user in the road segment; FIG. 3 : shows a flow diagram of an exemplary embodiment of a computer-implemented method for assigning a mobile terminal in a road segment to a road user in the road segment; and FIG. 4 : a block diagram of the computer-implemented method from FIG. 3 and of the apparatus from FIG. 2.Detailed Description of the FiguresFIG. 1 shows an overview of a computer-implemented method 132 for assigning 144 a mobile terminal 116 in a road segment 100 to a road user 102, 104 in the road segment 100.In FIG. 1, a road section 100 is seen, which consists of a road with two lanes. Another track starts from one track. The direct environment of the lanes with sidewalks and vegetation (both not shown) may also belong to road segment 100.A first road user 102 and a second road user 104 travel on the road section 100. A camera 108 may capture the road segment 100 in image data 146. The camera 108 is communicatively connected to an evaluation unit 110 by cable. The evaluation unit 110 is in turn communicatively connected to a memory 112 in the form of a server, likewise by cable. The camera 108, the memory 112 and the evaluation unit 110 form a device 106.For a first point in time 118, the position and direction of movement for the first 102 and second road user 104 can be determined in the evaluation unit 110 by means of the image data 146 of the camera 108. Furthermore, a mobile terminal 116 sends its own position and direction of movement to the memory 112 by means of mobile radio. However, the position of the mobile terminal 116 does not seem to correspond to either the first 102 or the second road user 104. The cause is, for example, measurement errors, signal fluctuations or erroneous calibrations. At the first time 118, it is not clear whether the mobile terminal 116 should be assigned to the first 102 or second road user 104.Therefore, at a later second time 120, positions and movement directions of the road users 102, 104 and of the mobile terminal 116 are received again. It is still not clear only from the data at the second point in time 120 whether the mobile terminal 116 should be assigned to the first 102 or second road user 104.Only at a third point in time 122 is it clear that the mobile terminal 116 has to be assigned to the second road user 104. With the aid of deviations of the trajectories of the road users 102, 104 from the trajectory of the mobile terminal 116, an assignment 144 can thus be successful. There need not always be such a drastic difference in the trajectories as in FIG. 1.FIG. 2 shows an exemplary embodiment of a device 106 for assigning a mobile terminal 116 in a road segment 100 to a road user 102, 104 in the road segment 100.The device 106 has a sensor unit 124, which in turn has a camera 108 and a radar sensor. Furthermore, the device 100 has a memory 112, which can be designed as a server, for example, as in FIG. 1. Furthermore, the device 106 has an evaluation unit 110, which in turn has a processor 128 and a nonvolatile memory 130. Of course, the device can have further units, for example a volatile memory or a monitor.All units 110, 112, 124 are communicatively connected to one another in the device. The device 106 does not have to be realized as a single component. Rather, the individual units 110, 112, 124 can be realized in a distributed manner or even be part of other devices.FIG. 3 shows a flow diagram of an exemplary embodiment of a computer-implemented method 132 for assigning a mobile terminal 116 in a road segment 100 to a road user 102, 104 in the road segment 100.In a first obtaining step 134, a first data record 150 is obtained which is assigned to the mobile terminal 116. For example, this is data originating from the mobile terminal 116.A first trajectory 160 is then calculated by means of the first data set 150 in a first calculation step 136. The first trajectory 160 is the trajectory of the mobile terminal 116.In a second obtaining step 138, at least one second data set 146, 148 is obtained. A plurality of second data sets 146, 148 are possible, for example, when the road segment 100 is at least partially detected by a plurality of sensors 108, 126. By being covered, for example by a truck, road sections 100 cannot always be completely detected.A plurality of second trajectories 154, 156, 158 are then calculated by means of the at least one second data set 146, 148 in a second calculation step 140. The data of a single sensor 108, 126 can also provide a multiplicity of trajectories. For example, a plurality of road users 102, 104 can be detected in image data 146.In a third calculation step 142, a deviation between the first trajectory 160 and the plurality of second trajectories 159 is now calculated.Then, in an assignment step 144, that second trajectory 154, 156, 158 from the plurality of second trajectories 159 which has a minimum deviation from the first trajectory 160 can be assigned to the first trajectory 160FIG. 4 shows a block diagram of the computer-implemented method 132 of FIG. 3 and the apparatus 106 of FIG. 2.The sensor unit 124 has the camera 108 and the radar sensor 126. The camera 108 generates image data 146, and the radar sensor 126 generates radar data 148. Both sensors 108, 126 are arranged in the road section 100 in such a way that it is detected by their data 146, 148. For example, the sensors 108, 126 may be attached to a traffic light or be part of an RSU (road side unit). Non-stationary devices such as drones could also have corresponding sensors 108, 126.The image data 146 and radar data 146 are conducted to the evaluation unit 110. The evaluation unit 110 has the nonvolatile memory 130, on which a computer program product 152 is stored. Furthermore, the evaluation unit 110 has a processor 128, which is connected bidirectionally communicatively to the nonvolatile memory 130.If the computer program product 152 is executed on the processor 128, the steps of the method 132 from FIG. 3 are executed. In particular, the image data 146 and the radar data 146 are used to calculate second trajectories 159. The image data 146 and the radar data 146 are merged in such a way that three road users 102, 104 in the road segment 100 are identified and a second trajectory 159 is calculated for each. A first second trajectory 154 is determined solely on the basis of the image data 146. For example, because radar sensor 146 has not had a clear view of road user 102, 104.The image data 146 consists of a plurality of images of the road segment 100 at different points in time 118, 120, 122. Object detection is carried out in the images by a neural network and road users 102, 104 are identified. A position of the road users 102, 104 can then be determined, for example because the position of the camera 108 is calibrated such that this is possible. The first second trajectory 154 can then be calculated from the position data.The neural network will be trained using already labeled data. As input data for the neural network, a data point of a road user 102, 104 from the second data set (sensor data 146, 148) at a time 118, 120, 122 and a data point from the first data set 150 (data of the mobile terminal 116) at the same time 118, 120, 122 are selected. The output of the neural network indicates the probability of at least parts of the second data set 146, 148 and the first data set 150 originating from the same road user 102, 104.A second second trajectory 156 is calculated using both the image data 146 and the radar data 148. The calculation is performed in a similar manner to that for the first second trajectory 134. Likewise, a third second trajectory 158 is determined only on the basis of the radar data 148. A total of three road users 102, 104 could therefore be identified with the aid of sensors 108, 126, and their trajectories 154, 156, 158 could be determined.The mobile terminal 116 can likewise send data with a position and time indication as first data set 150 to a memory 112 on account of its built-in sensors, for example a GPS sensor and an acceleration sensor. The memory 112 can be designed, for example, as a server. The memory 112 stores the first data set 150 and is in bidirectional communicative connection with the evaluation unit 110.The evaluation unit 110 can request the first data set 150 from the memory 112 and then calculates a first trajectory 160. The first trajectory 160 correlates with the movement of the mobile terminal 116.Now, a deviation between the first trajectory 160 and each of the second trajectories 159 is calculated. This can be accomplished, for example, by forming the difference between the position for each time 118, 120, 122 and then adding up the differences produced. It is also possible to include further properties in the deviation calculation. A person skilled in the art is familiar with many methods of calculating a deviation, for example the (weighted) quadratic mean.Subsequently, the first second trajectory 154 is selected to be associated with the first trajectory 160 because it has the smallest difference from the first trajectory 160. This is visualized in FIG. 4 by a thick frame and gray background. In other words, the second second trajectory 156 and third second trajectory 158 have a greater deviation from the first trajectory 160 than the first second trajectory 154. Thus, it is assumed below that mobile terminal 116 is moving with road user 102, 104, to which first second trajectory 154 belongs.If collision probabilities between the road users 102, 104 are now calculated, a warning can be sent to the mobile terminal 116 which is assigned to the road users 102, 104 concerned in a further step. Because the method 132 reduces the susceptibility to errors in the assignment 144, fewer unnecessary or erroneous warnings are sent.List of reference characters100 Road section 102 First road user 104 Second road user 106 Device 108 Camera 110 Evaluation unit 112 Memory 116 Mobile terminal 118 First point in time 120 Second point in time 122 Third point in time 124 Sensor unit 126 Radar sensor 128 Processor 130 Nonvolatile memory 132 Method 134 First obtaining step 136 First calculating step 138 Second obtaining step 140 Second calculating step 142 Third calculating step 144 Assigning step 146 Image data 148 Radar data 150 First data set 152 Computer program product 154 First second trajectory 156 Second second trajectory 158 Third second trajectory 159 Second trajectories 160 First trajectory

Claims

Computer-implemented method (132) for assigning (144) a mobile terminal (116) in a road segment (100) to a road user (102, 104) in the road segment (100), the method (132) having the following steps: a) obtaining (134) a first dataset (150) which is assigned to the mobile terminal (116), b) obtaining (138) at least one second dataset (146, 148) in which the road segment (100) is at least partially detected, c) calculating (136) a first trajectory (160) from the first dataset (150), d) calculating (140) a plurality of second trajectories (159) from the at least one second dataset (146, 148), e) calculating (142) deviations between the first trajectory (160) and the plurality of second trajectories (159), and f) assignment (144) of the first trajectory (160) to that second trajectory from the plurality of second trajectories (159) which has a minimum deviation from the first trajectory (160), characterized in that the method (132) is calculated by a neural network trained for the road section (100), wherein the first data set (150) and the at least one second data set (146, 148) serve as input for the neural network.Computer-implemented method according to Claim 1, characterized in that the method steps (134,136,138,140,142,144) are carried out for a multiplicity of mobile terminals (116) in the road section (100).Computer-implemented method according to Claim 1 or 2, characterized in that the first data record (150) and / or the at least one second data record (146, 148) has at least one of the following values: a) a time stamp, b) a position value, c) a speed value, d) an identifier, e) a movement direction value.Computer-implemented method according to one of the preceding claims, characterized in that the at least one second data set (146, 148) has image data (146) and the plurality of second trajectories (159) are calculated from the image data (146) by means of object recognition.Computer-implemented method according to one of the preceding claims, characterized in that deviations are calculated only between the first trajectory (160) and the second trajectories (154, 156, 158) from the plurality of second trajectories (159) if no previously defined exclusion criterion is fulfilled.A computer program product (152) comprising instructions which, when the program is executed by a computer, cause the computer to execute a computer-implemented method (132) according to any preceding claim.A computer readable storage medium (112,130) comprising the computer program product (152) of claim 6.A data carrier signal carrying the computer program product (152) of claim 6.Device (106) for assigning (144) a mobile terminal (116) in a road section (100) to a road user (102, 104) in the road section (100), the device (106) comprising: a) an evaluation unit (110) designed to carry out a method (132) according to one of Claims 1 to 5, and b) at least one sensor unit (124) which is communicatively connected to the evaluation unit (110), wherein the mobile terminal (116) is communicatively connected to the evaluation unit (110).Device according to Claim 9, characterized in that the sensor unit (124) has at least one sensor from the following group: a) a camera (108), b) a radar sensor (126), c) a lidar sensor.

Images