Apparatus and method for sensor selective data fusion

Abstract

Disclosed is a device (319, 408) for performing data fusion configured to: receive (601) a respective first vector (313, 314, 315, 316, 317, 318, 416, 417, 418, 419, 420, 421, 422) from each of a plurality of input devices (307, 308, 309, 310, 311, 312, 401, 402, 403, 404, 405, 406, 407); based on the first vectors, select (602) a sub-set (403, 405, 407) of the plurality of input devices; receive (603) a respective second vector (425, 426, 427) from each input device of the sub-set (403, 405, 407) of the plurality of input devices; and process (604) the second vectors using a first permutation-invariant deep learning model (428) to form an output (320, 429). This may allow the device to perform data fusion whilst respecting a predefined limitation on communication constraints and allow the device to form an output even if the data from some input devices is blocked.

Description

[0001] APPARATUS AND METHOD FOR SENSOR SELECTIVE DATA FUSION

[0002] FIELD OF THE INVENTION

[0003] This invention relates to data processing in a network, in particular to data fusion in a communications network.

[0004] BACKGROUND

[0005] Data fusion is the process of integrating multiple data points or sets of data perceived by multiple data sources. The data sources may, for example, be sensors. The aim of a data fusion process is to produce more consistent, accurate and useful information than that which can be provided by any individual sensor.

[0006] In two possible implementations of data fusion set-ups, the viewpoints of the sensors and the sensed phenomenon are static, or the viewpoints are changing and / or the sensed phenomenon is dynamic. Fixed cameras shooting a garage door, as illustrated in Figure 1 (a), is a static set-up, whereas static or moving cameras shooting a moving crowd, as illustrated in Figure 1 (b), is a dynamic one. In Figure 1 (a), four cameras each shoot a different area 1-4 of the garage door. Information from each image can be fused to obtain an output of the data- fusion process.

[0007] In order to fuse their information, sensors share their respective data, or features extracted from their respective data.

[0008] To process the data perceived by the sensors, machine learning techniques may be used. In some situations, an output may be formed with regard to a task. Machine learning is the process of using a computer algorithm to learn from given data, for example to perform a certain task.

[0009] An important category of tasks in machine learning is classification. In classification tasks, an algorithm decides which class a data sample given as input belongs to. An example is the classification of images. The task may be to determine if an image shows a cat, a dog, an elephant or a penguin. In this example, each animal corresponds to one of the four classes.

[0010] Deep neural networks (DNN) are efficient tools to solve such classification tasks. A DNN takes one or more data points (for example, one or more images) as input. The DNN generally outputs one number per class and the classification result is the class with the largest output value. DNNs are trained using a training data set, comprising data labelled with their corresponding classes. This is referred to as deep learning (DL). An optimisation algorithm, typically a stochastic gradient descent algorithm, is used to find the neural network parameters. The actual classification accuracy of the DNN in terms of the percentage of correctly classified data points is determined using a second labelled data set, known as the validation data set.

[0011] Figure 2 schematically illustrates an example of a data fusion architecture 200. To exemplify the use of such an architecture, consider a classification task. The classification task is one of single-class classification, where each data point (or set of data points in the case of data fusion with multiple inputs) is associated to exactly one class.

[0012] The set of input data to classify is a set of sensor data 201-206, denoted by xi, X2, ... , XN.with N being the number of sensors. These sensory data are indexed by sensor number. Each sensor can locally process their data, hi 207-212 is the local processing function of the sensor i. The output of each local processing function is a vector, shown at 213-218 for sensors 1-N respectively.

[0013] The example considered here is a centralized fusion model f, meaning that after processing their data, each sensor sends a feature vector to a central device, which can be referred to as the fusion center, f is the fusion model at the fusion center 219. Thus, the overall classification model to output classification result 220 is described by: classification result = f(hi(xi), h2(x2), ... , IIN(XN)).

[0014] Data fusion in a telecommunications context generally means that transmissions between the sensors and the fusion center are made through wireless channels. This entails two constraints. Firstly, the degradation or even blocking of some transmissions by wireless channels, and secondly, the need to minimize the bandwidth used.

[0015] With the above formalism, the first constraint can be expressed in the following way: it is desirable for the fusion model to perform correctly even if some of the data sources (x) are blocked.

[0016] The second constraint can be expressed in the following way: given a limitation of the total transmission size, defined as ‘total transmission size = sum of the size of all hi(xi)s’, it is also desirable to achieve the best classification result using the fusion model f. Deep learning approaches in data fusion are generally centralized fusion models and can be formulated as described above. Thus, even if they are not specifically designed to be used in a telecommunications context, a transmission size can be deduced for the models used.

[0017] Most of the deep learning approaches focus on data fusion without external constraints. The techniques only aim to implement a neural network that maximizes the classification performance, without considering communication constraints.

[0018] It is desirable to develop a method which overcomes such problems.

[0019] SUMMARY

[0020] According to one aspect, there is provided a device for performing data fusion, the device being configured to: receive a respective first vector from each of a plurality of input devices; based on the respective first vectors, select a sub-set of the plurality of input devices; receive a respective second vector from each input device of the sub-set of the plurality of input devices; and process the respective second vectors using a first permutation-invariant deep learning model to form an output.

[0021] As the device receives second vectors from only a sub-set of the input devices, this may, for example, allow the device to perform data fusion whilst respecting a predefined limitation on the communication constraints, such as a total transmission size of data sent between the input devices and the device for performing data fusion. This may also allow the device to form an output even if the data from one or more of the input devices is blocked.

[0022] A model is permutation-invariant when the output of the model does not depend on the order of the inputs. In this case, the output of the first permutation-invariant deep learning model does not depend on the order in which the respective second vectors from the selected subset of input devices are input to the model. This may allow the first permutation-invariant deep learning model to form the same output irrespective of the input order of the data from the input devices.

[0023] The device may be configured to select the sub-set of the plurality of input devices in dependence on an output of a selection model implemented by the device. The selection model may be configured to receive the respective first vectors as input. This may allow the device to select a sub-set of the input devices to receive second vectors from, which are used to form the output. This may allow the device to limit the total transmission size, whilst forming a reliable output. The selection model may be a second permutation-invariant deep learning model. In this case, the output of the selection model does not depend on the order in which the respective first vectors are input to the selection model. This may allow the model to form the same output irrespective of the input order of the data from the input devices.

[0024] Each respective first vector may comprise a number of coordinates. All of the first vectors may have the same number of coordinates. The selection model may be configured to select the sub-set of the plurality of input devices by applying a max pooling operation over each coordinate of the respective first vectors. This may allow the selection model to select an appropriate sub-set of input devices for a data fusion task.

[0025] An output of the max pooling operation may be a single vector having a same size as each respective first vector. This may allow the selection model to select which of the input devices it will use for the step of forming the output, i.e. which of the input devices the first permutationinvariant deep learning model will process second vectors from.

[0026] The device may be further configured to send a signal to each input device of the sub-set of the plurality of input devices. The signal may comprise a request for the respective second vector from each input device of the sub-set of the plurality of input devices. This may allow the device to communicate to the sub-set of input devices that they have been selected. The input devices of the selected sub-set can then send their respective second vectors to the device.

[0027] Each respective first vector may have a first size. Each respective first vector may have the same first size. Each respective second vector may have a second size. Each respective second vector may have the same second size. The second size may be larger than the first size. This may allow the device to only receive large vectors from the selected sub-set of input devices, whilst also allowing smaller first vectors from each of the input devices to be used in making the decision of which input devices should be selected to form the output.

[0028] A sum of the first sizes of the respective first vectors and the second sizes of the respective second vectors may define a total transmission size of data received by the device for forming the output. The total transmission size can therefore be directly computed from the sum of the first sizes and the second sizes. The device may be configured to respect a predefined limitation on the total transmission size. The predefined limitation may be defined by a user of the device. This may be done by configuring the input devices to output first and second vectors of a predefined size.

[0029] The respective first vectors may be respective outputs of a first data processing model implemented by each respective input device of the plurality of input devices. The first data processing model may receive as input a data sample, such as an image. Each of the plurality of input devices may implement the same first data processing model. This may allow each input device to extract relevant data from its respective input data sample.

[0030] The respective second vectors may be respective outputs of a second data processing model implemented by each respective input device of the plurality of input devices. The second data processing model may receive as input a data sample, such as an image. Each of the plurality of input devices may implement the same second data processing model. This may allow each input device to extract relevant data from its respective input data sample. The selected subset of the plurality of input devices send their respective second vectors to the device for input to the first permutation-invariant deep learning model.

[0031] Each of the plurality of input devices may be configurable to allow the first and second data processing models to output respective first vector of a predetermined size and / or a respective second vector of a different predetermined size. This may allow a user to set a desired size of the first vectors, second vectors and / or the total transmission size.

[0032] The first data processing model may be a different model to the second data processing model. The first and second data processing models may comprise one or more functions. Models may be considered to be different if they have the same architecture, but different weights. The first and second data processing models may be configured to output vectors of different sizes. The second data processing model may output a larger vector than the first data processing model.

[0033] Each of the input devices may comprise a sensor. The sensors may be, for example, camera sensors. This may allow each input device to capture an image to be used in the data fusion process. Each of the input devices may comprise a sensor of a different type. Each of the input devices may comprise the same type of sensor for collecting single-modal data. Alternatively, each input device may receive its input data from another source.

[0034] The respective first vectors and the respective second vectors may be feature vectors derived from respective input images. Each of the plurality of input devices may receive a respective input image as input. The respective input images may be processed by the first data processing model and the second data processing model. The respective input images may be processed by the second data processing model at each input device of the selected subset of input devices.

[0035] The respective input images may each represent a view of a static scene or a dynamic scene. A static scene is one where the viewpoint and objects in the scene are not moving or changing. A dynamic scene is one where the viewpoint and / or objects in the scene are moving or changing. The approach described herein may therefore be used to form an output from static or more complex dynamic input data.

[0036] In other implementations, the data sample input to the input devices may be of another data type (for example, text). Any types of data may be used. Preferably, the data sample input to each of the input devices is single-modal (i.e. each data sample is of the same data type).

[0037] The output of the device may be a classification result. For example, the output may classify an input image into one of a plurality of classes. The device may also be used for other types of task. The device may conveniently be used to form an output relating to any task utilizing data fusion.

[0038] According to a second aspect, there is provided a method for performing data fusion, the method comprising: receiving a respective first vector from each of a plurality of input devices; based on the respective first vectors, selecting a sub-set of the plurality of input devices; receiving a respective second vector from each input device of the sub-set of the plurality of input devices; and processing the respective second vectors using a first permutation-invariant deep learning model to form an output.

[0039] As second vectors are received from only a sub-set of the input devices, this may, for example, allow the device to respect a predefined limitation on the communication constraints, such as a total transmission size of data sent between the input devices and the device for performing data fusion. This may also allow the process to form an output even if the data from one or more of the input devices is blocked.

[0040] The plurality of input devices may be devices in a communications network. The communications network may be a wireless network. T ransmissions between the input devices and the device for performing data fusion may be made through wireless channels. Each of the input devices and / or the device for performing data fusion may be located at a node of the communications network, where each node is a connection point in the communications network and can act as an endpoint for data transmission or redistribution. One or more of the input devices and / or the device for performing data fusion may be base stations in the communications network. One or more of the input devices may comprise a respective sensor for collecting data samples for processing by a respective input device.

[0041] According to a further aspect, there is provided a computer readable storage medium having stored thereon computer readable instructions that, when executed at a computer system, cause the computer system to perform the method as set forth above. The computer system may comprise one or more processors. The computer readable storage medium may be a non-transitory computer readable storage medium.

[0042] BRIEF DESCRIPTION OF THE FIGURES

[0043] The present disclosure will now be described by way of example with reference to the accompanying drawings.

[0044] In the drawings:

[0045] Figures 1 (a) illustrates an example of a static set-up, with four static cameras shooting a static garage door.

[0046] Figure 1(b) schematically illustrates a dynamic set-up, with moving or static cameras shooting a crowd.

[0047] Figure 2 shows an example of a known centralised data fusion model.

[0048] Figure 3 schematically illustrates an example of a permutation-invariant deep learning data fusion architecture.

[0049] Figure 4 schematically illustrates another example of a permutation-invariant deep learning data fusion architecture.

[0050] Figure 5 shows an example of a max pooling layer of the selection model.

[0051] Figure 6 shows an example of a method of performing data fusion in accordance with embodiments of the present invention.

[0052] Figure 7 illustrates the building of a dynamic data fusion dataset.

[0053] Figure 8 shows experimental results obtained using methods in accordance with embodiments of the present invention compared with alternative data fusion methods.

[0054] DETAILED DESCRIPTION

[0055] Figure 3 shows an example of an architecture for a data fusion system 300. A set of input data from which to form an output using data fusion is the set of input data xi, X2, ... , XN, shown at 301-306 respectively. There are multiple items of input data in the set. In the exemplary system of Figure 3, there are N items of input data in the set, where N>1.

[0056] Any types of input data may be used, such as images, text and sound recordings. Preferably, the data samples input to each of the input devices are single-modal (i.e. each data sample in the set is of the same data type). However, multi-modal data (i.e. more than one type of data in the set) may also be used.

[0057] The data fusion system 300 comprises multiple input devices 307-312 and a device 319 at which the data fusion is performed. The device 319 is referred to herein as the fusion center.

[0058] The fusion center 319 and each of the input devices 307-312 may comprise a processor and a memory. The memory stores in a non-transient way code that is executable by the processor to implement the device or respective node in the manner described herein. The fusion center and each of the input devices may also comprise a transceiver for transmitting and receiving data to and from the device. The devices may be part of a communications network. The network may be a wireless network or a wired network. The fusion center and each of the input devices may be located at a node of the network, where each node is a connection point in the communications network and can act as an endpoint for data transmission or redistribution. One or more of the nodes may be base stations in the communications network.

[0059] The data fusion system 300 implements multiple models, such as deep learning networks, as will be described in more detail below.

[0060] Each input device 307-312 may measure its respective input data 301-306, for example using a respective sensor located at each input device, or may receive its respective input data from another source which makes the measurement.

[0061] Each input device 307-312 can locally process their respective input data 301-306. Each input device is configured to implement a respective first data processing model which receives the input data as input. Each respective first data processing model may comprise one or more functions. Each respective first data processing model may be a deep learning network, such as a DNN.

[0062] During inference, the items of input data 301-306 are each input into to the first data processing model at their respective input device 307-312. The output of the first data processing model implemented at the input devices 307-312 is a respective first vector, as indicated at 313-318. Each input device is preferably configured to implement the same first data processing model. At each input device, the output of the first data processing model is a first vector having size K1 . There are N such first vectors in total output at this stage of the process: one for each of the N input devices. Therefore, the first data processing model is implemented N times, i.e. once at each of the N input devices.

[0063] In other embodiments, for example in the case of multi-modal input data, one or more of the input devices may be configured to implement a different first data processing model to one or more other input devices. In this case, each of the respective first data processing models preferably outputs a vector of the same size as the other first processing model(s), for example K1.

[0064] The N vectors of size K1 are then fed into a selection model implemented by the fusion center 319. The selection model may be a neural network, such as a DNN.

[0065] The selection model can classify the set of input data using the N vectors of size K1. Based on the N vectors, the selection model selects a sub-set of the input devices. An example of how this may be done will be described below with reference to Figure 5. The number of selected input devices is N2.

[0066] The selected sub-set of input devices then each implement a respective second data processing model. Each input device is preferably configured to implement the same second data processing model. In other embodiments, for example in the case of multi-modal input data, one or more of the sub-set of input devices may be configured to implement a different second data processing model to one or more other input devices of the sub-set.

[0067] The output at each input device of the selected sub-set is a vector of size K2 (these vectors are not explicitly shown in Figure 3). In the preferred implementation, K2>K1. These N2 vectors of size K2 are then fed into a classification network implemented by the fusion center 319. Although each input device is capable of implementing a respective second data processing model, preferably, the respective second data processing model may only be run at a respective input device if that input device is part of the selected sub-set. Therefore, preferably, only the selected sub-set of input devices each output a respective second vector during the inference stage. In one example, the inputs to the first data processing model and the second data processing model are images recorded by a respective sensor at each input device. The first processing data model and the second data processing model may each perform feature extraction. In this case, the output vectors from the first and second data processing models are feature vectors for the respective input image.

[0068] In some implementations, the only difference between the first data processing model and the second data processing model may be the size of the vectors that are output. For example, they may have essentially the same architecture, but may be configured to form an output vector having a different size.

[0069] The first data processing model implemented at each of the input devices may output a respective first vector having a first size (for example, K1). Each respective first vector may have the same first size.

[0070] The second data processing model implemented at each input device of the selected sub-set may output a respective second vector having a second size (for example, K2). Each respective second vector may have the same second size. The second size may be larger than the first size. This may allow the device to only receive large vectors from the selected sub-set of input devices for forming the final output (for example, a classification result), whilst also allowing smaller vectors from each of the input devices to be used in making the decision as to which input devices should be selected to provide their larger vectors for forming the output.

[0071] A sum of the first sizes of the respective first vectors and the second sizes of the respective second vectors may define a total transmission size of data received by the device for forming the output. The total transmission size can therefore be directly computed from the sum of the first sizes and the second sizes.

[0072] The device may be configured to respect a predefined limitation on the total transmission size. The predefined limitation may be defined by a user of the device. This may be done by configuring the plurality of input devices to output first and second vectors of a predefined size.

[0073] Therefore, the input to the selection model at the fusion center 319 comprises multiple vectors of size K1 from each of the input devices. The input to the classification model comprises vectors of size K2 from each input device of the selected sub-set of input devices. The classification network implemented at the fusion center 319 then classifies the set of input data 301-306 using the N2 vectors of size K2. The output of the classification network is a classification result, shown at 320 in Figure 3.

[0074] The operation of the data fusion system and its associated models will now be described in more detail.

[0075] The exemplary data fusion system 400 schematically illustrated in Figure 4 is a system for performing data fusion in a classification task, particularly one of single-class classification, where each data point (or set of data points in the case of data fusion with multiple inputs) is associated with exactly one class. This is done for ease of description and explanation of the fundamental ideas. Generalizations to more complex problems and different network structures are also possible.

[0076] As shown in more detail in Figure 4, the data fusion system 400 comprises a plurality of input devices 401-407 and a fusion center 408. Each input device 401-407 comprises a sensor. In this example, the sensors are cameras which capture respective input images. The input images may be different parts of a larger image or may capture different views of the same object. Therefore, each item of input data of the set of input data is an image captured by a respective camera at a respective input device.

[0077] In this exemplary implementation, the method is implemented to perform classification for a set of images corresponding to a dynamic data fusion set-up.

[0078] All sensors 401-407 locally process their data with a respective first local data processing model and output a feature vector 416-422 respectively. In this example, as the data is single- modal (i.e. all images), each sensor processes their input data using the same first data processing model. All of the sensors 401-407 send their respective feature vector 416-422 to the fusion center 408.

[0079] In this example, the transmission of data is made through wireless channels, as shown at 423. However, the techniques described herein may alternatively be implemented in a wired network. In some implementations, the component devices may be contained within a single larger device and may optionally share a single processor. A selection model 424 implemented at the fusion center 408 selects a sub-set of sensors based on the feature vectors 416-422. One example of how this may be done will be described later, with reference to Figure 5.

[0080] In this example, the selection model 424 selects sensors 403, 405 and 407. The fusion center 408 sends back a signal, indicated by the dashed arrows, from the selection model 424 to the selected sensors 403, 405 and 407, to let them know that they have been selected. The signal may comprise a request for a second feature vector from each sensor of the sub-set of the plurality of sensors.

[0081] The selected sensors 403, 405, 407 then locally process their data with a second local data processing model and obtain feature vectors 425, 426, 427 respectively. Each feature vector 425, 426, 427 is larger than each feature vector 416-422. In this example, as the data is single- modal, each of the selected sub-set of sensors processes their input data using the same second data processing model.

[0082] The selected sensors 403, 405, 407 send this larger vector 425, 426, 427 to the fusion center 408. A classification model 428 implemented at the fusion center 408 proceeds to perform classification based on these large feature vectors 425, 426, 427. The classification model may have any suitable architecture known in the art, but is preferably a DNN. The classification result output from the classification model 428 is shown at 429.

[0083] In a preferred implementation, the selection model 424 comprises a max pooling layer and performs a max pooling operation over the coordinates of the first vectors received from the input devices. The max pooling operation extracts a maximum value of a respective coordinate for each of the first vectors. As shown in Figure 5, for six vectors 501-506 each having size K (in this example, K=4), for each coordinate i from 1 to K, this layer selects the largest coordinate i among the N input vectors 501-506. Doing so for each coordinate, it selects K numbers from at most K sensors and thus outputs a vector of size K.

[0084] For example, as shown in Figure 5, the max pooling layer takes the maximum value of coordinate 1 from the possible values indicated at 507-512 for the vectors 501-506 respectively. In this example, the maximum value of coordinate 1 is 13.2. Therefore, the max pooling layer chooses this value for coordinate 1 , 514, of output vector 513.

[0085] This layer allows for fusion of the data from the input sensors into a single vector. Furthermore, this max pooling layer can also be seen as a selection process. These at most K input devices, and only these, can then be used to provide second vectors for the classification task. Thus, the selection model at the fusion center makes an implicit selection of at most K sensors during fusion of the data from the multiple input devices in the system.

[0086] By selecting input devices that have sent first vectors having the largest coordinates, this may allow the selection model to select the most appropriate sub-set of input devices for the data fusion task.

[0087] Therefore, each respective first vector may comprise a number of coordinates. All of the first vectors may have the same number of coordinates. The selection model may be configured to select the sub-set of the plurality of input devices by applying a max pooling operation over each coordinate of the respective first vectors.

[0088] An output of the max pooling operation may be a single vector having a same size as each respective first vector. This may allow the selection model to select which of the input devices it will use for the step of forming the output, i.e. which of the input devices the first permutationinvariant deep learning model will receive and process second vectors from at the fusion center.

[0089] The max pooling layer placed at the beginning of the selection model 424 makes the model permutation-invariant and thus makes the overall architecture of the data fusion system permutation-invariant.

[0090] Both the selection model 424 and the classification model 428 can be trained for the task using suitable training data. Any suitable training method can be applied.

[0091] In the above implementation, the task is one of single-class classification, where each data point (or set of data points in the case of data fusion with multiple inputs) is associated with exactly one class. However, the technique may also be used for more complex problems and can be applied to other outputs besides classification results.

[0092] In the examples described above, the selection model 424 and the classification model 428 are deep neural networks (DNNs). However, other model architectures may be used.

[0093] In the preferred implementation, the selection model 424 and the classification model 428 are permutation-invariant deep learning models. This choice of implementation confers the advantage that the procedure becomes applicable to dynamic data fusion set-ups. This procedure can explicitly tackle the transmission size for the data fusion process, as the sizes of the first and second vectors sent by each input device can be adjusted to respect a predefined limitation on the transmission size.

[0094] Another advantage of the present procedure is that it is very general and can be applied in numerous set-ups (for example, for tasks other than classification, for single-modal or multimodal input data, and for dynamic or static set-ups).

[0095] Figure 6 shows a flow chart summarizing an example of a method for performing data fusion in accordance with embodiments of the present invention. At step 601, the method comprises receiving a respective first vector from each of a plurality of input devices. At step 602, the method comprises, based on the respective first vectors, selecting a sub-set of the plurality of input devices. At step 603, the method comprises receiving a respective second vector from each input device of the sub-set of the plurality of input devices. At step 604, the method comprises processing the respective second vectors using a first permutation-invariant deep learning model to form an output.

[0096] The following describes experiments performed to evaluate the performance of the approach described herein.

[0097] The experiments were restricted to a single-modal dynamic data fusion set-up.

[0098] Figure 7 illustrates the building of an exemplary dataset to work on. The MNIST (Modified National Institute of Standards and Technology) database is a large database of handwritten digits that is commonly used for training various image processing systems. There are 10 classes and each image belongs to a class between 0 and 9. To build a dataset for dynamic data fusion, each image, such as the image shown at 701 , was slightly modified by enlarging it, as shown at 702. Random parts of this image were then taken, as shown at 703 and 704. The number of parts of the image corresponds to the number of sensors (input devices) used.

[0099] In this example, the sensors are cameras. Each camera perceives only part of the context. By using the data from every sensor, the aim is to classify the image.

[0100] Two other data fusion procedures were used for comparative purposes. The first is a classical procedure with no selection step. In this procedure, there is only a single step of classification where every sensor sends their locally processed features to the fusion center, where it fuses them and outputs the classification results. This procedure is referred to as ‘no selection’.

[0101] The other data fusion procedure randomly selects a sub-set of sensors as a first step. There is no transmission during this step. Then, the no selection procedure is applied to this randomly-selected sub-set of sensors.

[0102] To compare these different procedures, the size of the transmitted data during a procedure was fixed at 180 (total number of coordinates for the first and second vectors). Having the same transmission size, these data fusion procedures were used to classify the sensor images.

[0103] In the experiments, there were 30 sensors and thus 30 parts of images (referred to below as quadrants).

[0104] For every procedure, similar permutation-invariant deep learning models were used to focus the comparison on the used procedure.

[0105] The configurations used were as follows. Configuration A used the two-step selection approach described herein, with 4 selected quadrants and 15 features per quadrant. Configuration B used the two-step selection approach described herein, with 3 selected quadrants and 30 features per quadrant. Configuration C used no selection, with 6 features per quadrant. Configuration D used random selection, with 27 selected quadrants and 7 features per quadrant.

[0106] The results are shown in Figure 8. The training accuracy (Train Acc), training loss (Train Loss), test accuracy (Test Acc) and test loss (Test Loss) are quantified for each configuration. These results demonstrate that the two-step procedure used in configurations A and B outperforms the two other procedures by at least 3.06% in accuracy. Therefore, given a fixed transmission size, the procedure described herein outperforms the other procedures used. For a fixed performance level, the procedure uses a smaller bandwidth than the other procedures. The quantity that is optimized is the fraction performance / transmission size.

[0107] In summary, the proposed approach for data fusion in a telecommunications context is advantageous for both of performance and transmission size, as well as in terms of generalization to multiple data fusion set-ups. Previous deep learning methods proceed to device selection while performing the data fusion. However, making the selection at the fusion center level does not reduce the transmission size. In contrast, the method described herein allows only a specific sub-set of sensors to transmit their larger second vectors to the fusion center for forming the output. Thus, the method may allow for reduced data transmission when applied in telecommunications context.

[0108] The applicant hereby discloses in isolation each individual feature described herein and any combination of two or more such features, to the extent that such features or combinations are capable of being carried out based on the present specification as a whole in the light of the common general knowledge of a person skilled in the art, irrespective of whether such features or combinations of features solve any problems disclosed herein, and without limitation to the scope of the claims. The applicant indicates that aspects of the present invention may consist of any such individual feature or combination of features. In view of the foregoing description it will be evident to a person skilled in the art that various modifications may be made within the scope of the invention.

Claims

CLAIMS1. A device (319, 408) for performing data fusion, the device being configured to: receive (601) a respective first vector (313, 314, 315, 316, 317, 318, 416, 417, 418, 419, 420, 421 , 422) from each of a plurality of input devices (307, 308, 309, 310, 311 , 312, 401 , 402, 403, 404, 405, 406, 407); based on the respective first vectors, select (602) a sub-set (403, 405, 407) of the plurality of input devices; receive (603) a respective second vector (425, 426, 427) from each input device of the sub-set (403, 405, 407) of the plurality of input devices; and process (604) the respective second vectors using a first permutation-invariant deep learning model (428) to form an output (320, 429).

2. The device (319, 408) as claimed in claim 1 , wherein the device is configured to select the sub-set (403, 405, 407) of the plurality of input devices in dependence on an output of a selection model (424) implemented by the device, wherein the selection model is configured to receive the respective first vectors (416, 417, 418, 419, 420, 421 , 422) as input.

3. The device (319, 408) as claimed in claim 2, wherein the selection model (424) is a second permutation-invariant deep learning model.

4. The device (319, 408) as claimed in claim 2 or claim 3, wherein each respective first vector comprises a number of coordinates and wherein the selection model is configured to select the sub-set of the plurality of input devices by applying a max pooling operation over each coordinate of the respective first vectors.

5. The device (319, 408) as claimed in claim 4, wherein an output of the max pooling operation is a single vector having a same size as each respective first vector.

6. The device (319, 408) as claimed in any preceding claim, wherein the device is further configured to send a signal to each input device of the sub-set of the plurality of input devices, the signal comprising a request for the respective second vector from each input device of the sub-set of the plurality of input devices.

7. The device (319, 408) as claimed in any preceding claim, wherein each respective first vector (313, 314, 315, 316, 317, 318, 416, 417, 418, 419, 420, 421 , 422) has a first size andeach respective second vector (425, 426, 427) has a second size, the second size being larger than the first size.

8. The device (319, 408) as claimed in claim 7, wherein a sum of the first sizes of the respective first vectors (313, 314, 315, 316, 317, 318, 416, 417, 418, 419, 420, 421 , 422) and the second sizes of the respective second vectors (425, 426, 427) defines a total transmission size of data received by the device (319, 408) for forming the output (320, 429).

9. The device (319, 408) as claimed in any preceding claim, wherein the respective first vectors (313, 314, 315, 316, 317, 318, 416, 417, 418, 419, 420, 421 , 422) are respective outputs of a first data processing model implemented by each respective input device (307, 308, 309, 310, 311 , 312, 401 , 402, 403, 404, 405, 406, 407) of the plurality of input devices.

10. The device (319, 408) as claimed in any preceding claim, wherein the respective second vectors (425, 426, 427) are respective outputs of a second data processing model implemented by each respective input device (403, 405, 407) of the plurality of input devices.

11. The device (319, 408) as claimed in any preceding claim, wherein the respective first vectors (416, 417, 418, 419, 420, 421 , 422) and the respective second vectors (425, 426, 427) are feature vectors derived from respective input images (409, 410, 411 , 412, 413, 414, 415).

12. The device (319, 408) as claimed in claim 11 , wherein the respective input images (409, 410, 411 , 412, 413, 414, 415) each represent a view of a static scene or a dynamic scene.

13. The device (319, 408) as claimed in any preceding claim, wherein the output (320, 429) of the device is a classification result.

14. A method (600) for performing data fusion, the method comprising: receiving (601) a respective first vector (313, 314, 315, 316, 317, 318, 416, 417, 418, 419, 420, 421 , 422) from each of a plurality of input devices (307, 308, 309, 310, 311 , 312, 401 , 402, 403, 404, 405, 406, 407); based on the respective first vectors, selecting (602) a sub-set (403, 405,407) of the plurality of input devices; receiving (603) a respective second vector (425, 426, 427) from each input device of the sub-set of the plurality of input devices; and processing (604) the respective second vectors using a first permutation-invariant deep learning model (428) to form an output (320, 429).

15. A computer-readable storage medium having stored thereon computer-readable instructions that, when executed at a computer system, cause the computer system to perform the method of claim 14.

Images