A method for training multiple neural networks on a device

Abstract

The invention relates to a method (100) for training multiple neural networks (6) on a device (1) to solve a task, comprising the following steps: - Providing (101) a first set of training data, the first set of training data comprising sensor data of a first sensor (2) of the device (1) and labels specific for the task to be solved, - Training (102) a first neural network (6a) of the multiple neural networks (6) to solve the task based on the first set of training data, - Providing (103) a second set of training data, the second set of training data comprising sensor data of a second sensor (3) of the device (1), labels specific for the task to be solved, and an output of the first neural network (6a), - Training (104) a second neural network (6b) of the multiple neural networks (6) to solve the task based on the second set of training data. Furthermore, the invention relates to a computer program, an apparatus, and a storage medium for this purpose.

Description

[0001] R.413038

[0002] - 1 -

[0003] Description

[0004] Title

[0005] A method for training multiple neural networks on a device

[0006] The invention relates to a method for training multiple neural networks on a device to solve a task. Furthermore, the invention relates to a computer program, an apparatus, and a storage medium for this purpose.

[0007] State of the art

[0008] Many mobile systems are equipped with multiple sensors: cars on the roads with radars, lidars, cameras, LTE / 5G, Inertial Measurement Units (IMUs); smartphones with IMUs, magnetometers, gyroscopes, LTE / 5G; Automated Guided Vehicles (AGVs) with lidars, IMUs, WiFi / LTE / 5G, etc. For each of the mentioned sensor types, there are plenty of vendors and manufacturers. A crucial step in the utilization of every sensor is a proper calibration and validation / verification at the vendor side and at the OEM side after the integration into a device. The calibration is frequently left out because of the high costs associated with it.

[0009] Sensors play a key role in many use cases including localization. Here, different transport systems require a precise and robust positioning system to navigate in industries without human intervention, improve traffic on roads, etc.

[0010] Industries may operate multiple Automated Guided Vehicles (AGVs) from different manufacturers to support their industrial processes. To achieve the goal of precise indoor positioning, manufacturers may use different sensors within their AGVs that enable safe navigation. These sensors are calibrated in different laboratories and under a variety of conditions. After integration of sensors into a device, calibration shall be performed to calibrate the sensor relative to its new R.413038

[0011] - 2 - environment Many aspects may have an impact on the sensor data quality: close circuitry generating electromagnetic noise, the material of the device hull, closely located actuators, etc. These circumstances are unique for every product and eventually every sensor. With time, further environmental factors like atmospheric pressure, dirt or dust impact the measurement result.

[0012] The industrial environment, similar as the dense outdoor scenario, imposes several challenges on the propagation of wireless signals. These challenges include multi-path fading resulting in Multipath Components (MPCs), Non-Line-of- Sight (NLoS), and interference. Though the problem of interference in industries may be mitigated by limiting the number of devices operating in a same frequency band (controlled environment), but the problem of MPCs and NLoS still persists. Even small deviations in antenna placement on an end device and physical parameters of the frontend, which is quite common for mobile device frontends, result in large deviations in sensing data. The variation in amplitude and phase of each MPC can result in constructive or destructive interference that can introduce a large impact on the signal strength, coverage range, and data transfer rates. Thereby, making the signal less accurate to estimate the position of an AGV. Therefore, to achieve precise indoor positioning accuracy, data from various sources can be fused together to minimize the overall error of the system. This process is known as sensor fusion or data fusion. Data fusion is used across different applications, which include collision avoidance, activity recognition, pedestrian detection, Automated Emergency Steering (AES), Automated Emergency Braking (AEB), and indoor localization. The solutions employed in the literature for data fusion process include Kalman filters, Particle filters, and Machine Learning (ML) techniques. The individual configuration of weights in covariance matrices of filters or neuronal networks represents a “higher level calibration” process compared to the conventional calibration of sensors. However, this “higher level calibration” is performed manually.

[0013] Further, multi-modal nets based on the Deep Neural Network (DNN) that fuse the video data from camera with the IMU data with a goal to detect hazardous driving on the world’s road are known. The multi-modal net consists of four layers: vision layer, kinematics layer, data-fusion layer, and prediction layer. The vision layer is used to extract features from the video data. This includes detection of position of R.413038

[0014] - 3 - potential collision objects and classifying them into motor-vehicle (main cause of collision due to car, truck, bus) and non-motor (main cause of collision due to pedestrians and cyclist). The kinematics layer provides data from the IMU sensor. The data-fusion layer includes a fully connected layer, which trains on the video as well as the kinematics data and passes the results to the prediction layer, which computes the probabilities of collisions. The proposed model has recorded 91 .6% accuracy.

[0015] Further, a multi-modal Convolutional Neural Network (CNN) for activity recognition by fusing the data from multiple sensors is known. The model captures data of the same type from the sensors placed at various positions (smart watch, smart phone) on the human body and data from different sensor types (accelerometer, gyroscope, and magnetometer). The collected data are then fused together using a CNN to output the probability that a test signal from the sensors belongs to one of the ‘N’ activity classes. The application of data fusion for pedestrian detection by fusing thermal and RGB images is also possible. Features extracted from the thermal and colour images may be given as inputs to a Cross- Modality Attention Transformer (CAT), whose responsibility is to focus on the regions ignored by the one feature map but highlighted by the other feature map. The union of the proposals from both feature maps are fed to the Multimodal Fusion Transformer (MFT), which performs feature fusion to output the predictions.

[0016] An adaptive sensor fusion framework for pedestrian indoor navigation using sensors in smartphone is also known. In this context, WiFi Received Signal Strength (RSS) fingerprints may be fused with the data from IMU sensors in smartphone. The experimental area may be divided into several cells and at each location the WiFi fingerprint may be recorded along with the magnetometer readings, and the probabilities may be computed during offline process. During the online phase, the received WiFi signal is compared with the stored fingerprints. Using the data from IMU sensors, the position of the user may be computed through a Hidden Markov Model (HMM). Thereafter, the final position of the user may be revealed by fusing the WiFi fingerprint information and IMU sensors using a probabilistic approach. R.413038

[0017] - 4 -

[0018] Several state-of-the-art alternatives have been mentioned that focus on multimodal sensor data fusion. All these alternatives particularly use manual definition of weights for their fusing methods. The data from multiple sources / sensors include bias and noise caused by the individual sensor and dynamically changing environment. For example, different mount position of sensors as well as the height, weight, and movement style of AGVs from different manufacturers can yield irregular sensor data (e.g., IMU readings). Moreover, environmental factors such as presence of concrete walls, obstacles made with different materials and size / shape can affect the performance of various 5G / B5G / 6G wireless transceivers differently. The above studies have not considered the individual and dynamic changing environment factors. Moreover, it is not easy to calibrate the sensors reflecting such conditions because they are so diverse. Ignoring individual as well as dynamic changing environment factors leads to inaccurate results (e.g., localization results) and the situation becomes worse when multi-sensor AGVs from different manufacturers are operated in the same environment (applies also on multi-sensor vehicles in outdoor scenarios).

[0019] Disclosure of the invention

[0020] According to aspects of the invention a method with the features of claim 1 , a computer program with the features of claim 8, a data processing apparatus with the features of claim 9 as well as a computer-readable storage medium with the features of claim 10 are provided. Further features and details of the invention are disclosed in the respective dependent claims, the description and the drawings. Features and details described in the context to the inventive method also correspond to the inventive computer program, the inventive data processing apparatus as well as the inventive computer-readable storage medium, and vice versa in each case.

[0021] According to an aspect of the invention a method for training multiple neural networks on a device to solve a task is provided, the method comprising the following steps, wherein the steps may be performed repeatedly and / or in a specific order. The neural networks may be deep neural networks. The device may be a client device and particularly an Automated Guided Vehicle. However, the device may also be a vehicle, bicycle, smartphone, drone or a smart home R.413038

[0022] - 5 - security camera. The task to be solved may be an estimation of a position of the device. However, other tasks are also possible like obstacle detection for collision avoidance, personnel / pedestrian / human / animal detection for safety reasons, Automated Emergency Steering (AES) and Automated Emergency Braking (AEB).

[0023] In a first possible step, a first set of training data is provided, the first set of training data comprising sensor data of a first sensor of the device and labels specific for the task to be solved.

[0024] In a next possible step, a first neural network of the multiple neural networks is trained to solve the task based on the first set of training data.

[0025] In a next possible step, a second set of training data is provided, the second set of training data comprising sensor data of a second sensor of the device, labels specific for the task to be solved, and an output of the first neural network. The output of the first neural network may be an estimated position of the device if the task to be solved is a position estimation of the device.

[0026] The first and the second sensor may be sensors such as cameras, lidars, radar systems, and inertial measurement units (IMUs). It is also possible that the first and / or the second sensor is a wireless transceiver, wherein the position is estimated by means of the first or the second neural network based on distance measurements between the device and at least one base station if the task to be solved is a position estimation.

[0027] The sensor data may be real or synthetic data in each case, wherein the synthetic data may be a simulation or representation of actual measurements by the respective sensor.

[0028] In a next possible step, a second neural network of the multiple neural networks is trained to solve the task based on the second set of training data. Preferably, the second neural network is different from the first neural network. R.413038

[0029] - 6 -

[0030] The neural networks may each comprise an input layer, at least one hidden layer, and an output layer. The neural networks may form a two-stage Deep Neural Network.

[0031] The input to the first neural network may be the sensor data, for example a position of at least one base station and a distances from the device to the at least one base station, which may be computed by means of the first sensor, if the first sensor is a wireless transceiver, for example a 5G / B5G / 6G transceiver. The computing may be carried out via a measured time of flight between the base station and the first sensor. The first set of training data may be collected in advance in reference locations, which may involve synthetic and / or real-world individual measurement data by the first sensor. To create the real-world labelled training data, Automated Guided Vehicles as devices may be made to collect the distance measurements to the base stations with the first sensor in different reference locations.

[0032] The first neural network may learn a relationship between the provided labelled input training data and may optimize neuron weights to minimize its training loss by adjusting the weights. This adjustment of weights of the neurons may be done based on the learning experience of the sensor data obtained from the first sensor, which may be specific to each manufacturer of a device and an area of the industry where the device is operating. The architecture of the first neural network on every device may be the same. Only the data feeding to the first neural network may change from one device to the other. During a testing phase, when the first neural network on the device is exposed to a new set of measurements, from its local learning experience, the first neural network may solve the task, i.e. for example estimate the position of the device. Thereafter, an output of the first neural network, i.e. for example the estimated position, may be given as an input to the second neural network.

[0033] The second neural network may take a first input as the output of the first neural network, i.e. for example the estimated position of a device in X and Y coordinates, as well as a second input as the sensor data of the second sensor of the device, for example a distance computed from acceleration data of an I MU sensor as the second sensor. Similar as for the first neural network, a set of R.413038

[0034] - 7 - labelled training data may be collected in advance which may involve both synthetic as well as real-world measurement data. This labelled training data may be used to train the second neural network. To create the real-world labelled training data, the AGV as a device may be made to move along a reference trajectory taking reference movements (constant acceleration / deacceleration). During this movement, the AGV may record the positions estimated by the first neural network as well as the distances computed from IMU accelerations. This training data may be used to train the second neural network to produce the position of the AGV which is particularly a result of the data fusion. Similar as for the first neural network, the second neural network may learn a relationship between the provided labelled training data and try to minimize its training loss by adjusting the weights of its neurons.

[0035] In case more sensors are used, further neural networks may be applied using the same cascade structure. Thus, at least a third sensor and a respective third neural network may be used, which may receive an output of the second neural network and sensor data of the third sensor as an input.

[0036] The provided features may enable multi-sensor fusion in the training of the neural networks, which may improve the accuracy and robustness of the task solved by these networks. The ability to leverage data from multiple sensors on a device may allow the trained neural networks to incorporate information from diverse sources. This may lead to more accurate outputs regarding to task to be solved as the networks consider various inputs.

[0037] It is also possible that the method further comprises the following step:

[0038] Applying the trained first and second neural network in an inference phase to solve the task, wherein an input for the first neural network is provided by the first sensor of the device and an input for the second neural network is provided by the second sensor of the device and by an output of the first neural network.

[0039] By applying the trained first and second neural network a fusion of information from the two sensors may be enabled, including those with different measurement ranges and accuracy levels, resulting in a more robust and accurate solving of the task. R.413038

[0040] - 8 -

[0041] It is further possible that, if no output was provided by the first neural network in a current measuring interval of the second sensor of the device, a previous output of the second neural network is used as an input for the second neural network instead of the output of the first neural network. In other words, the proposed method may be carried out repeatedly, specifically periodically or cyclically, e.g., with a specific rate. Here, in a first cycle of carrying out the method, an output of the first neural network is provided to the second neural network, wherein in a second cycle of carrying out the method, no output of the first neural network is provided to the second neural network. For the latter case, a previous output of the second neural network is used as an input for the second neural network instead of the output of the first neural network.

[0042] The first and the second sensor may not provide their measurements at the same frequency rate. By means of this feature, the solving of the task may advantageously be performed with the frequency rate of the sensor with the highest frequency rate. By utilizing a previously calculated output, potential errors or inconsistencies may be avoided that might arise from sudden gaps in the sensor data of the first sensor. This feature may also have implications for scenarios where the first neural network may be experiencing temporary difficulties or is being recalibrated, such as during initial startup or after a software update. In these situations, using a previous output from the second neural network may ensure that the solving of the task is still performed until the issues are resolved.

[0043] It is possible that the method further comprises the following steps:

[0044] Providing at least two devices, the at least two devices comprising the first and the second neural network, the first and the second neural network being trained individually on each device according to the method according to the invention,

[0045] Providing a server, the server being in bilateral communication with the at least two devices, the bilateral communication particularly being an uplink and downlink 5G, B5G or 6G communication,

[0046] Receiving individual weights of the first and the second neural network from each device by the server, R.413038

[0047] - 9 -

[0048] Calculating new individual weights for the first and the second neural network based on the received individual weights by means of the server, particularly by performing an averaging of the received individual weights, Transmitting the calculated new individual weights from the server to the at least two devices.

[0049] The transmitted calculated new individual weights may then be implemented in the first and the second neural network of each device. This may be a federated learning approach, where learned weights are shared with the server, which then distributes the updated, calculated weights back to each device. The transmitting of the individual weights of the first and the second neural network from each device to the server may be performed regularly in defined intervals and / or initiated by a trigger like a rapid change in the environment or a rapid change in the individual weights of a device. By sharing knowledge across the at least two devices, a more comprehensive understanding of an environment of the devices may be developed and an overall accuracy may be improved. With regard to the implementation of the calculated new individual weights, every device may individually decide how to adapt the weights of its own neural networks with respect to the calculated new individual weights. The calculated new individual weights may also be used by new devices for initialization.

[0050] In another aspect of the invention, a computer program may be provided, in particular a computer program product, comprising instructions which, when the computer program is executed by a computer, cause the computer to carry out the method according to the invention. Thus, the computer program according to the invention can have the same advantages as have been described in detail with reference to a method according to the invention.

[0051] In another aspect of the invention, an apparatus for data processing may be provided, which is configured to execute the method according to the invention. As the apparatus, for example, a computer can be provided which executes the computer program according to the invention. The computer may include at least one processor that can be used to execute the computer program. Also, a nonvolatile data memory may be provided in which the computer program may be stored and from which the computer program may be read by the processor for being carried out. R.413038

[0052] - 10 -

[0053] According to another aspect of the invention a computer-readable storage medium may be provided which comprises the computer program according to the invention and / or instructions which, when executed by a computer, cause the computer to carry out the steps of the method according to the invention. The storage medium may be formed as a data storage device such as a hard disk and / or a non-volatile memory and / or a memory card and / or a solid state drive. The storage medium may, for example, be integrated into the computer.

[0054] Furthermore, the method according to the invention may be implemented as a computer-implemented method. Alternatively or additionally, at least one of the disclosed method steps may be computer-implemented and / or automated.

[0055] Further advantages, features and details of the invention will be apparent from the following description, in which embodiments of the invention are described in detail with reference to the drawings. In this context, the features mentioned in the claims and in the description may each be essential to the invention individually or in any combination. Showing:

[0056] Fig. 1 : A method, computer program, a storage medium and apparatus according to embodiments of the invention,

[0057] Fig. 2: A method with Automated Guided Vehicles as devices according to embodiments of the invention,

[0058] Fig. 3: A first and a second neural network according to embodiments of the invention,

[0059] Fig. 4: A system according to embodiments of the invention.

[0060] Fig. 1 shows a method 100, computer program 20, a storage medium 15 and apparatus 10 according to embodiments of the invention.

[0061] Fig. 1 particularly shows an embodiment of a method 100 for training multiple neural networks 6 on a device 1 to solve a task. In a first step 101 , a first set of R.413038

[0062] - 11 - training data is provided, the first set of training data comprising sensor data of a first sensor 2 of the device 1 , the first set of training data further comprising labels specific for the task to be solved. In a second step 102, a first neural network 6a is trained to solve the task based on the first set of training data. In a third step 103, a second set of training data is provided, the second set of training data comprising sensor data of a second sensor 3 of the device 1 and an output of the first neural network 6a, the second set of training data further comprising labels specific for the task to be solved. In a fourth step 104, a second neural network 6b is trained to solve the task based on the second set of training data.

[0063] In the present invention, a focus may be on a utilization of sensors like 5G / B5G / 6G transceivers that can receive the wireless signal from 5G / B5G / 6G radio heads and IM Us. These sensors are, however, only a few examples. The method according to embodiments of the invention may be extended to other sensors like lidar, radar, cameras, wheel encoders, gyroscopes, etc. As a device, AGVs may be used as autonomous transport systems for industrial manufacturing processes. However, further devices are also possible including vehicles, bicycles, smartphones, drones, smart home security cameras etc.

[0064] It may be assumed that the sensing of 5G / B5G / 6G signals as well as the sensing from IM Us at an AGV from a first manufacturer, most probably differ from that of the AGV from a second manufacturer, since their sensors most probably come from different vendors and even if they are calibrated, the calibration took place in different calibration laboratories. And, as mentioned above, the calibration of a sensor integrated into a device is frequently left out.

[0065] Considering sensing of 5G / B5G / 6G signals, the situation may become worse, when AGVs from multiple manufacturers are used in complex environments like industries. This may be due to the presence of heavy obstacles made with different materials (concrete, glass, wood), that block the Line-of-Sight (LoS) signal and give rise to Multi-Path Components (MPCs). Depending on the sensitivity of 5G / B5G / 6G transceiver from a specific manufacturer including unified placement of antennas, the channel parameters, e.g., Time of Arrival (ToA) and Angle of Arrival (AoA), interpreted may vary a lot. Under such circumstances, the achievable indoor localization accuracy for AGVs may differ R.413038

[0066] - 12 - from one manufacturer to the other and it may become a challenging task for industries to operate them in the same environment using the same software and the same parameter set The localization is just an example application of how the sensing data may be used. Further application examples could include obstacle detection for collision avoidance, personnel / human / animal detection for safety reasons, Automated Emergency Steering (AES) and Automated Emergency Braking (AEB) in automotive scenarios, etc.

[0067] A problem that may be identified is, given a multiple AGVs in an industrial scenario, where AGVs possess a variety of sensors from different manufacturers with at least one sensor being 5G / B5G / 6G transceiver, how to fuse the data coming from the sensors efficiently, such that the drawbacks of dealing with different sensors noise / bias can be minimized and the achievable indoor positioning accuracy for multiple AGVs can be maximized without need for a costly manual calibration of every sensor after their integration into an end product.

[0068] In the method according to embodiments of the invention, a new way of fusing multimodal data from multiple sensors is presented, with at least one sensor being a 5G / B5G / 6G transceiver, to handle individual properties of sensors without need for manual individual calibration of sensors after their integration into an end product. The method may be used in an application example of indoor positioning. The method according to embodiments of the invention may also handle the irregular readings of each sensor caused by the individual characteristics after the integration into an AGV as well as the dynamic changing environment and executes an efficient data fusion policy, which maximizes the achievable indoor positioning accuracy of multiple multi-sensor AGVs from different manufacturers in complex environments like industries.

[0069] The method according to embodiments of the invention may further be based on the collaborative learning technique known as Federated Learning (FL), implemented through two neural networks forming a two-stage Deep Neural Network (DNN). On every AGV, a two-stage neural network is deployed with a goal to estimate the AGVs position by fusing the data from 5G / B5G / 6G transceiver and IMU sensors. A reason for using a two-stage neural network may R.413038

[0070] - 13 - come from the fact that the output frequency rate of the first sensor, for example a 5G / B5G / 6G transceiver, may not match the second sensor, for example an IMU sensor. In general, the IMU sensor may have a higher frequency rate, allowing to output higher number of measurements than the 5G / B5G / 6G transceiver per unit time. Therefore, the two-stage neural network according to embodiments may handle the synchronization issue between these sensors in such a way that whenever there is a new measurement from the 5G / B5G / 6G transceiver, it may be fused with the IMU data to output the AGV position. In situations where there is no measurement from 5G / B5G / 6G transceiver, the IMU readings may be used to output the AGV position. It may be noted, that the number of neural networks may depend on the number of sensors used for an application of interest.

[0071] The first stage of the neural network may be used to handle the measurements from one of the sensors, i.e. the first sensor, for example a 5G / B5G / 6G transceiver. The 5G / B5G / 6G transceiver may provide the distance measurements to the base stations using the Time-of-Flight (ToF) / Two-Way Ranging (TWR) techniques similar to an Ultra-Wideband (UWB) transceiver. A sufficient set of labelled training data, created in advance may be used to train the first neural network. This labelled training data may include synthetic data as well as the real-world measurement data collected in reference locations from the 5G / B5G / 6G transceiver on every AGV. By this, every AGV may get sufficient training data to set the weights of the first neural network. These weights may reflect individual sensor characteristics. The training data may include positions of the base stations and the distances from the AGV to the base stations computed by the 5G / B5G / 6G transceiver. The neural network may learn a relationship between the provided inputs and adjusts the weights of its neurons in such a way that the neural network estimates the position of the AGV when it is exposed to the new set of measurements. Here, the adjustment of weights may be done based on the learnings from the sensor’s readings on the AGV, which is particularly specific to each manufacturer of AGV and the environment conditions.

[0072] For the creation of the reference data another sensor may be used to automate the process. For devices with wheels, odometers may represent an example. R.413038

[0073] - 14 -

[0074] Here, an AGV may collect sets of labelled training data by moving on a line: the exact location on a line may be provided by the odometer. A typical accuracy of a stepper motor used in AGVs may be below ±0.05°, which corresponds to an error of below 1 mm for a wheel of 15 cm in diameter.

[0075] The second stage of the neural network may be the place where the actual data fusion happens. The inputs to the second neural network may be the estimated position of an AGV in (X, Y) coordinates as well as the distance computed from the acceleration data of the IMU sensor. Similar as for the first neural network, a sufficient set of labelled training data, created in advance may be used to train the second neural network. This labelled data includes synthetic data as well as the real-world measurement data. Similar as for the first neural network, for real- world measurement data, initially, the AGVs may be made to move along a reference trajectory by making reference movements (move with constant acceleration / deacceleration). During this movement, the AGV may record the positions estimated by the first neural network as well as the distances computed from IMU accelerations. These training datasets (specific to each AGV) may be used to train the second neural network to produce the accurate position of the AGV which is a result of the data fusion.

[0076] The proposed 2-stage neural network may also handle the synchronization issue between the 5G / B5G / 6G transceiver and IMU sensor. The estimated position of an AGV in (X, Y) coordinates may be provided either from the 5G / B5G / 6G transceiver when there is a measurement available or from the output of the second neural network during the previous step when the measurement from the 5G / B5G / 6G transceiver is not available.

[0077] In short, every AGV may be trained to estimate the position from the learnings of its own sensor data. The learnings of each AGV may be used by other AGVs, which may improve the positioning accuracy of an entire fleet of AGVs even though they are from different manufacturers. Federated Learning may be used for sharing learning experiences among AGVs. Each AGV may share the weights of its two-stage neural network with a central federated localization server using the 5G / B5G / 6G uplink communication. The federated localization server may fuse the received weights by a calculation, (e.g., performing averaging, however, R.413038

[0078] - 15 - other methods are also possible) and forwards the newly derived weights to the entire fleet of AGVs using the 5G / B5G / 6G downlink communication. Every AGV may individually decide how to adapt the weights of its own neural network neurons with respect to the global weights. These newly received weights may not be specific to any AGV or environment. In fact, they may represent the learnings of the entire fleet of AGVs and the dynamic changing environment. These weights may be used by new AGVs for initialization.

[0079] Further spin-off out of the FL approach may be represented in the amount of the collected weights with results of individual training. The weights may be collected at a central entity, i.e. the server, and may be stored along with the specification of the installed sensors and further characteristics of an AGV. For a large database of weights, new possibilities for learning of the dependences among the sensor model, mounting position, AGV model, etc. may be beneficial. The time for the individual training of new AGVs may be minimized or can become obsolete in case the corresponding parameter set of a similar AGV is found in the database along with the pre-recorded weights.

[0080] The proposed solution may be useful when an application (e.g., precise indoor positioning service) used for a large number of similarly equipped devices relies on a multimodal sensor fusion with a variety of sensor hardware from different manufacturers, calibrated under a different condition.

[0081] A two-stage neural network 6 as shown in Fig. 3 may be used to solve the task according to the invention, i.e. for example estimate the position of an AGV, and may be deployed on every device 1 . The neural networks 6a, 6b may each comprise an input layer 61 a, 61 b, at least a first hidden layer 62a, 62b and a second hidden layer 63a, 63b, and an output layer 64a, 64b. The input layer of the first neural network 6a may receive different sensor data Xi, yi, di , ..., y&, d6from the first sensor 2 and may produce an output X5G / B5G / 6G, y5G / B5G / 6G. This output, as well as sensor data dimufrom the second sensor 3 may be an input for the second neural network 6b, which may produce an output Xftnai, ytinai.

[0082] A method 200 with regard to an Automated Guided Vehicle embodiment is shown in Fig. 2. In a first step 201 , a first neural network may be trained with R.413038

[0083] - 16 - labelled data involving synthetic as well as the real-world measurement data collected by every AGV in advance in reference locations. In a second step 202, a second neural network may be trained with labelled data involving synthetic as well as the real-world measurement data collected by every AGV in advance along a reference trajectory by taking reference movements. In a third step 203, the resulting two- stage neural network may be deployed on an entire fleet of AGVs with a same number of inputs, hidden, and output layers. In a fourth step 204, a testing phase may be performed, where the two-stage neural networks on each AGV may be exposed with a new set of measurements. In a fifth step 205, it may be evaluated if at a current timestep, the measurements from both the 5G / B5G / 6G transceiver as well as the I MU are available. If not, then, according to a sixth step 206 (path “N” for “No”), due to the lack of 5G / B5G / 6G measurement at the current timestep, the execution of first neural network may be skipped, and the control may jump directly to the second neural network. According to a subsequent seventh step 207, the inputs to the second neural network may be the output of the second neural network at the previous timestep and the distance data computed from the IMU accelerations. If the measurements from both the 5G / B5G / 6G transceiver as well as the IMU are available at the current timestep in step 205, then, according to a subsequent eighth step 208 (path “Y” for “Yes”), the 5G / B5G / 6G receiver measurements may be passed to the first neural network which may output the initial position estimate of the AGV. According to a subsequent ninth step 209, the initial estimated position by the first neural network may be given as input to the second neural network along with the distance measurements computed from IMU accelerations to estimate the final position of the AGV, which may be the result of data fusion. In a tenth step 210, the weights of the two-stage neural network on each AGV may be shared with the federated localization server using uplink 5G / B5G / 6G communication. In an eleventh step 211 , the federated localization server may fuse the received weights and distributes the new set of weights to the entire fleet of AGVs using downlink 5G7B5G / 6G communication. In a twelfth step 212, each AGV may decide how to adapt the weights of its own neurons with respect to the global weights. In a thirteenth step 213, the AGVs may position themselves. R.413038

[0084] - 17 -

[0085] The input to the first neural network may be the sensor data, for example a position of at least one base station and a distances from the device to the at least one base station, which may be computed by the first sensor, if the first sensor is a wireless transceiver, for example a 5G / B5G / 6G transceiver. The first set of training data may be collected in advance in reference locations, which may involve synthetic and / or real-world individual measurement data by the first sensor. To create the real-world labelled training data, Automated Guided Vehicles as devices may be made to collect the distance measurements to the base stations with the onboard 5G / B5G / 6G transceiver in different reference locations.

[0086] The first neural network may learn a relationship between the provided labelled input training data and may optimize neuron weights to minimize its training loss by adjusting the weights. This adjustment of weights of the neurons may be done based on the learning experience of the data obtained from the first sensor, which may be specific to each manufacturer of a device and an area of the industry where the device is operating. The architecture of the first neural network on every device may remain the same. Only the data feeding to the first neural network may change from one device to the other. During a testing phase, when the first neural network on the device is exposed to a new set of measurements, from its local learning experience, the first neural network may solve the task, i.e. for example estimate the position of the device. Thereafter, an output of the first neural network, i.e. for example the estimated position, may be given as an input to the second neural network.

[0087] The second neural network may take inputs as the output of the first neural network, i.e. for example the estimated position of a device in X and Y coordinates, as well as the sensor data of the second sensor of the device, for example a distance computed from acceleration data of an I MU sensor as the second sensor. Similar as for the first neural network, a set of labelled training data may be collected in advance which may involve both synthetic as well as real-world measurement data. This labelled training data may be used to train the second neural network. To create the real-world labelled training data, the AGV as a device may be made to move along a reference trajectory taking reference movements (constant acceleration / deacceleration). During this movement, the R.413038

[0088] - 18 -

[0089] AGV may record the positions estimated by the first neural network as well as the distances computed from I MU accelerations. These datasets may be used to train the second neural network to produce the accurate position of the AGV which is particularly a result of the data fusion. Similar as for the first neural network, the second neural network may learn a relationship between the provided labelled training data and tries to minimize its training loss by adjusting the weights of its neurons.

[0090] In case more sensors are used, further neural networks may be applied using the same cascade structure.

[0091] In general, a 5G / B5G / 6G transceiver may not provide the measurements at the same frequency rate as that of an I MU sensor. Therefore, in the situations when there is no estimated position measurement from the 5G / B5G / 6G transceiver, then simply the output of the second neural network during the previous step may be used as the input to the second neural network. The accuracy of the final estimated position may depend on the sensitivity of the sensors being used onboard the AGV, which might vary from manufacturer to manufacturer. To benefit from the learnings and maximize the achievable positioning accuracy across the entire fleet of the AGVs, one may make use of Federated Learning (FL). Each AGV may share the weights from its learned data with the federated localization server using a 5G / B5G uplink communication. The federated localization server may fuse the received weights (e.g., performing averaging, however, other methods are also possible) and forward the newly derived weights to the entire fleet of AGVs using the 5G / B5G / 6G downlink communication. Every AGV may individually decide how to adapt the weights of its own neural network with respect to the global weights. In addition, these weights may be used by new AGVs for initialization.

[0092] A system 50 according to embodiments of the invention is shown in Fig. 4. The system 50 may comprise several devices 1 which in turn may comprise a first sensor 2 and a second sensor 3. The first sensor 2 may be a 5G / B5G / 6G transceiver and the second sensor 3 may be an IMU sensor. By means of the first sensor 2, a respective distance may be calculated towards different base stations 7. On the devices 1 , a respective two-stage neural network 6 according R.413038

[0093] - 19 - to the invention may be deployed. Individual weights w1 , w2 of the respective two-stage neural networks 6 may be shared with a server 4, which may be a federated localization server. The server may be connected to a database 5, which may store the shared individual weights and may also save previously shared individual weights.

[0094] Localization is just an example application of how the current invention may be used to solve real-life challenging problems. Further, application examples could include obstacle detection for collision avoidance, personnel / human / animal detection for safety reasons, Automated Emergency Steering (AES) and Automated Emergency Braking (AEB), etc. AGVs may also be equipped with a variety of other sensors like lidar, radar, camera, and wheel encoders. AGV is just an example of the device. The invention may also be extended to other devices including vehicles, bicycles, smartphones, drones, smart home security cameras, etc.

[0095] The above explanation of the embodiments describes the present invention in the context of examples. Of course, individual features of the embodiments can be freely combined with each other, provided that this is technically reasonable, without leaving the scope of the present invention.

Claims

R.413038- 20 -Claims1 . A method (100) for training multiple neural networks (6) on a device (1) to solve a task, comprising the following steps:Providing (101) a first set of training data, the first set of training data comprising sensor data of a first sensor (2) of the device (1) and labels specific for the task to be solved,Training (102) a first neural network (6a) of the multiple neural networks (6) to solve the task based on the first set of training data, Providing (103) a second set of training data, the second set of training data comprising sensor data of a second sensor (3) of the device (1), labels specific for the task to be solved, and an output of the first neural network (6a),Training (104) a second neural network (6b) of the multiple neural networks (6) to solve the task based on the second set of training data.

2. The method (100) of claim 1 , characterized in that the task to be solved is an estimation of a position of the device (1) and the output of the first neural network (6a) is an estimated position of the device (1).

3. The method (100) of claim 2, characterized in that the first sensor (2) is a wireless transceiver, wherein the position is estimated by means of the first neural network (6a) based on distance measurements between the device (1) and at least one base station (7).

4. The method (100) of any one of the preceding claims 2 or 3, characterized in that the device (1) is an Automated Guided Vehicle.

5. The method (100) of any one of the preceding claims, characterized in that the method (100) further comprises the following step:Applying the trained first and second neural network (6a, 6b) in an inference phase to solve the task, wherein an input for the first neural network (6a) is provided by the first sensor (2) of the deviceR.413038- 21 -(1) and an input for the second neural network (6b) is provided by the second sensor (3) of the device (1) and by an output of the first neural network (6a).

6. The method (100) of claim 5, characterized in that, if no output was provided by the first neural network (6a) in a current measuring interval of the second sensor (3) of the device (1), a previous output of the second neural network (6b) is used as an input for the second neural network (6b) instead of the output of the first neural network (6a).

7. The method (100) of any one of the preceding claims, characterized in that the method (100) further comprises the following steps:Providing at least two devices (1), the at least two devices (1) comprising the first and the second neural network (6a, 6b), the first and the second neural network (6a, 6b) being trained individually on each device (1) according to the method (100) of one of the preceding claims,Providing a server (4), the server (4) being in bilateral communication with the at least two devices (1), Receiving individual weights of the first and the second neural network (6a, 6b) from each device (1) by the server (4), Calculating new individual weights for the first and the second neural network (6a, 6b) based on the received individual weights by means of the server (4),Transmitting the calculated new individual weights from the server (4) to the at least two devices (1).

8. A computer program (20), comprising instructions which, when the computer program (20) is executed by a computer (10), cause the computer (10) to carry out the method (100) of any one of the preceding claims.

9. A data processing apparatus (10), comprising means for carrying out the method (100) of any one of claims 1 to 7.R.413038- 22 -10. A computer-readable storage medium (15) comprising instructions which, when executed by a computer (10), cause the computer (10) to carry out the steps of the method (100) of any one of claims 1 to 7.

Images