Apparatus and method for sensor selective data fusion

Abstract

A device (319, 408) for performing data fusion is configured to: receive (601) a respective first vector (313, 314, 315, 316, 317, 318, 416, 417, 418, 419, 420, 421, 422) from each of a plurality of input devices (307, 308, 309, 310, 311, 312, 401, 402, 403, 404, 405, 406, 407); selecting (602) a subset (403, 405, 407) of the plurality of input devices according to the first vector; receiving (603) a respective second vector (425, 426, 427) from each input device of the subset (403, 405, 407) of the plurality of input devices; the second vector is processed (604) using a first permutation-invariant deep learning model (428) to form an output (320, 429). In this way, the device can perform data fusion while complying with predefined restrictions on communication constraints, and the device can form an output even if the data of some input devices is blocked.

Description

Technical Field

[0001] The present application relates to data processing in a network, and in particular to data fusion in a communication network. Background Art

[0002] Data fusion is the process of combining multiple data points or sets of data sensed by multiple data sources. For example, the data sources can be sensors. The purpose of the data fusion process is to produce more consistent, accurate, and useful information than any single sensor can provide.

[0003] In two possible implementations of data fusion settings, the sensor's viewpoint and perceived phenomena are static, or the viewpoint is constantly changing and / or the perceived phenomena are dynamic. A fixed camera photographing a garage door as shown in Figure 1(a) is a static setting, while a static or moving camera photographing a moving crowd as shown in Figure 1(b) is a dynamic setting. In Figure 1(a), four cameras photograph different areas 1 to 4 of the garage door. The information of each image can be fused to obtain the output of the data fusion process.

[0004] To fuse the information of sensors, these sensors share their respective data or share the features extracted from their respective data.

[0005] To process the data sensed by the sensors, machine learning techniques can be used. In some cases, an output can be formed for a certain task. Machine learning is the process of using computer algorithms to learn from given data, for example to perform a certain task.

[0006] An important category of tasks in machine learning is classification. In classification tasks, the algorithm decides to which class an input data sample belongs. Image classification is an example. The task could be to determine whether an image shows a cat, dog, elephant, or penguin. In this example, each animal corresponds to one of the four categories.

[0007] Deep neural networks (DNNs) are an efficient tool for solving such classification tasks. DNNs take one or more data points (e.g., one or more images) as input. DNNs typically output a number for each category, and the classification result is the category with the largest output value. DNNs are trained using a training dataset that includes data labeled with corresponding categories. This is called deep learning (DL). An optimization algorithm (usually a stochastic gradient descent algorithm) is used to find the neural network parameters. The actual classification accuracy of the DNN (i.e., the percentage of correctly classified data points) is determined by a second labeled dataset (called a validation dataset).

[0008] Figure 2 An example of a data fusion architecture 200 is shown schematically. To illustrate the use of such an architecture, consider a classification task. The classification task is one of a single-class classification task, i.e., each data point (or a group of data points in the case of data fusion with multiple inputs) is associated with exactly one class.

[0009] A set of input data to be classified is a set of sensor data 201 to 206, represented as x1, x2, ... N , where N is the number of sensors. These sensor data are indexed by the sensor number. Each sensor can process its own data locally. i 207 to 212 are local processing functions of sensor i. The output of each local processing function is a vector, as shown in 213 to 218 corresponding to sensors 1 to N respectively.

[0010] The example considered here is a centralized fusion model f, that is, each sensor sends the feature vector to a central device after processing its own data, which can be called a fusion center. f is the fusion model in the fusion center 219. Therefore, the overall classification model of the output classification result 220 is described as follows:

[0011] Classification result = f(h1(x1),h2(x2),…,h N (x N )).

[0012] Data fusion in the telecommunications field usually refers to the transmission between sensors and fusion centers through wireless channels. This brings two constraints. First, some transmissions over wireless channels will degrade or even be interrupted; second, the bandwidth used needs to be minimized.

[0013] According to the above form, the first constraint can be expressed as follows: Even if some data sources (x i ) is blocked, and we hope that the fusion model can run correctly.

[0014] The second constraint can be expressed in the following way: considering the limitation of the total transmission size (defined as “total transmission size = the sum of the sizes of all hi(xi)”), it is also desirable to use the fusion model f to obtain the best classification result.

[0015] Deep learning methods in data fusion are usually centralized fusion models that can be formulated as described above. Therefore, even if these methods are not specifically designed for the telecommunications domain, the transmission size can be derived for the model used.

[0016] Most deep learning approaches focus on data fusion without external constraints. These techniques simply aim to implement a neural network that maximizes the classification performance without considering communication constraints.

[0017] It is desirable to develop a method to address these issues. Summary of the invention

[0018] According to one aspect, a device for performing data fusion is provided. The device is configured to: receive a corresponding first vector from each input device of a plurality of input devices; select a subset of the plurality of input devices based on the corresponding first vector; receive a corresponding second vector from each input device in the subset of the plurality of input devices; and process the corresponding second vector using a first permutation invariant deep learning model to form an output.

[0019] Since the device receives the second vectors only from a subset of the input devices, the device can perform data fusion while adhering to predefined limits on communication constraints (e.g., the total transfer size of data sent between the input devices and the device for performing data fusion), etc. In this way, the device can form an output even if data from one or more input devices is blocked.

[0020] A model is a permutation invariant model when the output of the model does not depend on the order of the inputs. In this case, the output of the first permutation invariant deep learning model does not depend on the order in which the corresponding second vectors of the selected subset of input devices are input into the model. Thus, the first permutation invariant deep learning model can form the same output regardless of the order in which the data of the input devices are input.

[0021] The device may be configured to select the subset of the plurality of input devices based on an output of a selection model implemented by the device. The selection model may be configured to receive the corresponding first vector as input. Thus, the device may select a subset of input devices to receive a second vector therefrom for forming an output. Thus, the device may limit the total transmission size while forming a reliable output.

[0022] The selection model may be a second permutation invariant deep learning model. In this case, the output of the selection model does not depend on the order in which the corresponding first vectors are input into the selection model. In this way, the model can form the same output regardless of the input order of the data of the input device.

[0023] Each corresponding first vector may include multiple coordinates. All first vectors may have the same number of coordinates. The selection model may be used to select the subset of the multiple input devices by performing a maximum pooling operation on each coordinate of the corresponding first vector. In this way, the selection model may select a suitable subset of input devices for the data fusion task.

[0024] The output of the maximum pooling operation may be a single vector having the same size as each corresponding first vector. In this way, the selection model may select which input device's data is used in the step of forming the output, i.e., which input device's second vector is to be processed by the first permutation invariant deep learning model.

[0025] The device may also be configured to send a signal to each input device in the subset of the plurality of input devices. The signal may include a request for the corresponding second vector of each input device in the subset of the plurality of input devices. In this way, the device may communicate with the selected subset of input devices. The input devices in the selected subset may then send their corresponding second vectors to the device.

[0026] Each corresponding first vector may have a first size. Each corresponding first vector may have the same first size. Each corresponding second vector may have a second size. Each corresponding second vector may have the same second size. The second size may be greater than the first size. In this way, the device may receive only the large vectors of a selected subset of input devices, while also using the smaller first vectors of each input device to determine which input devices' second vectors should be selected to form an output.

[0027] The sum of the first size of the corresponding first vector and the second size of the corresponding second vector may define a total transmission size of data received by the device for forming the output. Thus, the total transmission size may be calculated directly from the sum of the first size and the second size. The above device may be configured to comply with a predefined limit on the total transmission size. The predefined limit may be defined by a user of the device. This may be achieved by configuring the input device to output the first vector and the second vector of predefined sizes.

[0028] The corresponding first vector may be a corresponding output of a first data processing model implemented by each corresponding input device of the plurality of input devices. The first data processing model may receive a data sample (e.g., an image) as input. Each input device of the plurality of input devices may implement the same first data processing model. In this way, each input device may extract relevant data from its corresponding input data sample.

[0029] The corresponding second vectors may be corresponding outputs of a second data processing model implemented by each corresponding input device of the plurality of input devices. The second data processing model may receive a data sample (e.g., an image) as input. Each of the plurality of input devices may implement the same second data processing model. In this way, each input device may extract relevant data from its corresponding input data sample. A selected subset of the plurality of input devices sends their corresponding second vectors to the above-mentioned device for input into the first permutation invariant deep learning model.

[0030] Each of the plurality of input devices may be configured such that the first data processing model and the second data processing model output a respective first vector of a predetermined size and / or a respective second vector of a different predetermined size. In this way, a user may set a desired size of the first vector, the second vector, and / or the total transfer size.

[0031] The first data processing model may be a different model from the second data processing model. The first data processing model and the second data processing model may include one or more functions. If the models have the same architecture but different weights, the models may be considered different. The first data processing model and the second data processing model may be used to output vectors of different sizes. The vector output by the second data processing model may be larger than the vector of the first data processing model.

[0032] Each input device may include a sensor. For example, the sensor may be a camera sensor. In this way, each input device may capture an image to be used in the data fusion process. Each input device may include a different type of sensor. Each input device may include the same type of sensor for collecting single-modal data. Alternatively, each input device may receive its own input data from other sources.

[0033] The corresponding first vector and the corresponding second vector may be feature vectors extracted from the corresponding input image. Each input device of the plurality of input devices may receive the corresponding input image as input. The corresponding input image may be processed by the first data processing model and the second data processing model. The corresponding input image may be processed by the second data processing model in each input device in the selected subset of input devices.

[0034] The corresponding input images may represent views of a static scene or a dynamic scene, respectively. A static scene is a scene in which the viewpoint and objects in the scene do not move or change. A dynamic scene is a scene in which the viewpoint and / or objects in the scene are moving or changing. Thus, the methods described herein may be used to form outputs from static input data or more complex dynamic input data.

[0035] In other implementations, the data samples input to the input devices may be of other data types (e.g., text). Any type of data may be used. The data samples input to each input device are preferably unimodal (i.e., each data sample is of the same data type).

[0036] The output of the device may be a classification result. For example, the output may classify the input image into one of a plurality of categories. The device may also be used for other types of tasks. The device may be conveniently used to form outputs associated with any task utilizing data fusion.

[0037] According to a second aspect, a method for performing data fusion is provided. The method comprises: receiving a corresponding first vector from each input device of a plurality of input devices; selecting a subset of the plurality of input devices based on the corresponding first vector; receiving a corresponding second vector from each input device in the subset of the plurality of input devices; and processing the corresponding second vector using a first permutation invariant deep learning model to form an output.

[0038] Since the second vectors are only received from a subset of the input devices, the apparatus can comply with predefined limits on communication constraints, such as the total transfer size of data sent between the input devices and the apparatus for performing data fusion. In this way, the process can generate an output even if data from one or more input devices is blocked.

[0039] The plurality of input devices may be devices in a communication network. The communication network may be a wireless network. Transmissions between the input devices and the device for performing data fusion may be performed via a wireless channel. Each input device and / or the device for performing data fusion may be located in a node of the communication network, wherein each node is a connection point in the communication network and may be used as an endpoint for data transmission or redistribution. One or more input devices and / or the device for performing data fusion may be a base station in the communication network. One or more input devices may include a corresponding sensor for collecting data samples for processing by the corresponding input device.

[0040] According to another aspect, a computer-readable storage medium storing computer-readable instructions is provided. When the computer-readable instructions are executed by a computer system, the computer system performs the above method. The computer system may include one or more processors. The computer-readable storage medium may be a non-transitory computer-readable storage medium. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] The present application is described below by way of examples with reference to the accompanying drawings.

[0042] In the attached picture:

[0043] Figure 1(a) shows an example of a static setup, where four static cameras capture a static garage door.

[0044] FIG1( b ) schematically shows a dynamic setup, where a moving or static camera captures a crowd.

[0045] Figure 2 An example of a known centralized data fusion model is shown.

[0046] Figure 3 An example of a permutation-invariant deep learning data fusion architecture is shown schematically.

[0047] Figure 4 Another example of a permutation-invariant deep learning data fusion architecture is shown schematically.

[0048] Figure 5 An example of selecting the max pooling layer of the model is shown.

[0049] Figure 6 An example of a method for performing data fusion provided by an embodiment of the present application is shown.

[0050] Figure 7 The construction of a dynamic data fusion dataset is shown.

[0051] Figure 8 The experimental results obtained by comparing the method provided in the embodiment of the present application with an alternative data fusion method are shown. DETAILED DESCRIPTION

[0052] Figure 3 An example of the architecture of the data fusion system 300 is shown. A set of input data to form an output using data fusion is x1, x2, ... x N , as shown in 301 to 306 respectively. There are multiple input data in this group. Figure 3 In the exemplary system, there are N items of input data in this group, where N>1.

[0053] Any type of input data may be used, for example, images, text, and audio recordings. The data samples input to each input device are preferably unimodal data samples (i.e., each data sample in the set is of the same data type). However, multimodal data (i.e., more than one type of data in the set) may also be used.

[0054] The data fusion system 300 includes a plurality of input devices 307 to 312 and a device 319 for performing data fusion. The device 319 is referred to herein as a fusion center.

[0055] The fusion center 319 and each input device 307 to 312 may include a processor and a memory. The memory stores code in a non-transient manner, which can be executed by the processor to implement the above-mentioned devices or corresponding nodes in the manner described in this article. The fusion center and each input device may also include a transceiver, which is used to send data to the above-mentioned devices and receive data from the above-mentioned devices. These devices can be part of a communication network. The network can be a wireless network or a wired network. The fusion center and each input device can be located in a node of the network, wherein each node is a connection point in the communication network and can be used as an endpoint for data transmission or redistribution. One or more of the nodes can be a base station in the communication network.

[0056] The data fusion system 300 implements a variety of models, such as a deep learning network, which will be described in detail below.

[0057] Each input device 307 to 312 may measure its respective input data 301 to 306, for example, using a respective sensor located in each input device; or may receive its respective input data from another source that makes measurements.

[0058] Each input device 307 to 312 can locally process its corresponding input data 301 to 306. Each input device is used to implement a corresponding first data processing model, which receives the input data as input. Each corresponding first data processing model may include one or more functions. Each corresponding first data processing model may be a deep learning network, such as a DNN.

[0059] During the inference process, each item of input data 301 to 306 is respectively input into the first data processing model in its corresponding input device 307 to 312. The output of the first data processing model implemented by the input device 307 to 312 is the corresponding first vector, as shown in 313 to 318.

[0060] Each input device is preferably used to implement the same first data processing model. In each input device, the output of the first data processing model is a first vector of size K1. At this stage of the process, there are N such first vectors in the total output: one for each of the N input devices. Thus, the first data processing model is implemented N times, i.e. once for each of the N input devices.

[0061] In other embodiments, for example, in the case of multimodal input data, one or more input devices may be used to implement a first data processing model that is different from one or more other input devices. In this case, each corresponding first data processing model preferably outputs a vector having the same size (e.g., K1) as the other one or more first processing models.

[0062] Then, the N vectors of size K1 are input into the selection model implemented by the fusion center 319. The selection model can be a neural network, for example, a DNN.

[0063] The selection model can use N vectors of size K1 to classify this set of input data. Based on the N vectors, the selection model selects a subset of input devices. Figure 5 An example is given to illustrate how to make the selection. The number of input devices selected is N2.

[0064] Then, the selected input device subset realizes the corresponding second data processing model respectively.Each input device is preferably used to realize the same second data processing model.In other embodiments, for example, in the case of multimodal input data, one or more input devices in the input device subset can be used to realize the second data processing model different from one or more other input devices in the subset.

[0065] The output of each input device in the selected subset is a vector of size K2 (these vectors are not in Figure 3 ). In a preferred implementation, K2>K1. Then, these N2 vectors of size K2 are input into the classification network implemented by the fusion center 319. Although each input device can implement the corresponding second data processing model, it is preferred that if the corresponding input device is part of the selected subset, then the input device can only run on the corresponding second data processing model. Therefore, it is preferred that only the selected subset of input devices output the corresponding second vectors respectively in the inference stage.

[0066] In one example, the inputs of the first data processing model and the second data processing model are images recorded by corresponding sensors in each input device. The first data processing model and the second data processing model can perform feature extraction respectively. In this case, the output vectors of the first data processing model and the second data processing model are feature vectors of the corresponding input images.

[0067] In some implementations, the only difference between the first data processing model and the second data processing model may be the size of the output vector. For example, the first data processing model and the second data processing model may have substantially the same architecture, but may be used to form output vectors having different sizes.

[0068] The first data processing model implemented by each input device may output a corresponding first vector having a first size (eg, K1). Each corresponding first vector may have the same first size.

[0069] The second data processing model implemented by each input device in the selected subset can output a corresponding second vector having a second size (e.g., K2). Each corresponding second vector can have the same second size. The second size can be larger than the first size. In this way, the above-mentioned device can only receive the large vector of the selected input device subset to form the final output (e.g., classification result), while also using the smaller vectors of the individual input devices to determine which input devices should be selected to provide larger vectors to form the output.

[0070] The sum of the first size of the corresponding first vector and the second size of the corresponding second vector may define a total transmission size of data received by the device for forming an output. Thus, the total transmission size may be directly calculated from the sum of the first size and the second size.

[0071] The above device can be configured to comply with a predefined limit on the total transfer size. The predefined limit can be defined by a user of the device. This can be achieved by configuring the plurality of input devices to output the first vector and the second vector of predefined size.

[0072] Thus, the input to the selection model in the fusion center 319 includes a plurality of vectors of size K1 for each input device. The input to the classification model includes a vector of size K2 for each input device in the selected subset of input devices.

[0073] Then, the classification network implemented by the fusion center 319 uses N2 vectors of size K2 to classify a set of input data 301 to 306. The output of the classification network is the classification result, such as Figure 3 As shown in 320.

[0074] The operation of the data fusion system and its related models are introduced in detail below.

[0075] Figure 4 The exemplary data fusion system 400 shown in the schematic diagram is a system for performing data fusion in a classification task, which is specifically one of a single-class classification task, i.e., each data point (or a group of data points in the case of data fusion with multiple inputs) is associated with exactly one class. This is done to facilitate description and explanation of the basic idea. It can also be generalized to more complex problems and different network structures.

[0076] like Figure 4As shown in detail, the data fusion system 400 includes a plurality of input devices 401 to 407 and a fusion center 408. Each input device 401 to 407 includes a sensor. In this example, the sensor is a camera that captures a corresponding input image. The input image can be a different part of a larger image or can capture different views of the same object. Therefore, each input data in the set of input data is an image captured by a corresponding camera in a corresponding input device.

[0077] In this exemplary implementation, the above method is used to perform classification on a set of images corresponding to a dynamic data fusion setting.

[0078] All sensors 401 to 407 process their data locally using the corresponding first local data processing model and output feature vectors 416 to 422 respectively. In this example, since the data is single-modal data (i.e., all are images), each sensor processes their input data using the same first data processing model. All sensors 401 to 407 send their corresponding feature vectors 416 to 422 to the fusion center 408.

[0079] In this example, the data transmission is performed via a wireless channel, as shown in 423. However, the techniques described herein may also be implemented in a wired network. In some implementations, the component devices may be included in a single larger device and may also optionally share a single processor.

[0080] The selection model 424 implemented by the fusion center 408 selects a subset of sensors based on the feature vectors 416 to 422. Figure 5 Give an example of how to make a selection.

[0081] In this example, the selection model 424 selects sensors 403, 405, and 407. The fusion center 408 sends a signal from the selection model 424 back to the selected sensors 403, 405, and 407, as shown by the dashed arrows, to let these sensors know that they are selected. The signal may include a request for a second feature vector for each sensor in the subset of the plurality of sensors.

[0082] Then, the selected sensors 403, 405, 407 process their data locally using the second local data processing model and obtain feature vectors 425, 426, 427, respectively. Each feature vector 425, 426, 427 is larger than each feature vector 416 to 422. In this example, since the data is unimodal data, each sensor in the selected sensor subset processes their input data using the same second data processing model.

[0083] The selected sensors 403, 405, 407 send larger vectors 425, 426, 427 to the fusion center 408. The classification model 428 implemented by the fusion center 408 continues to perform classification based on these large feature vectors 425, 426, 427. The classification model can have any suitable architecture known in the art, but is preferably a DNN. The classification result output from the classification model 428 is shown as 429.

[0084] In a preferred implementation, the selection model 424 includes a maximum pooling layer, and performs a maximum pooling operation on the coordinates of the first vectors received from the input device. The maximum pooling operation extracts the maximum value of the corresponding coordinates of each first vector. Figure 5 As shown, for six vectors 501 to 506 of size K (in this example, K=4), for each coordinate i from 1 to K, this layer selects the largest coordinate i among the N input vectors 501 to 506. This is done for each coordinate, and this layer selects K numbers from at most K sensors, thereby outputting a vector of size K.

[0085] For example, Figure 5 As shown, the max pooling layer takes the maximum value of coordinate 1 from the possible values ​​corresponding to vectors 501 to 506, as shown in 507 to 512, respectively. In this example, the maximum value of coordinate 1 is 13.2. Therefore, the max pooling layer selects this value for coordinate 1 of output vector 513 (as shown in 514).

[0086] This layer fuses the data from the input sensors into a single vector. In addition, this max pooling layer can also be seen as a selection process. These at most K input devices (and only these devices) can then provide the second vector for the classification task. Therefore, when fusing data from multiple input devices in the system, the selection model in the fusion center implicitly selects at most K sensors.

[0087] By selecting the input device that has sent the first vector with the largest coordinate, the selection model can select the most appropriate subset of input devices for the data fusion task.

[0088] Thus, each corresponding first vector may include a plurality of coordinates. All first vectors may have the same number of coordinates. The selection model may be used to select a subset of the plurality of input devices by performing a maximum pooling operation on each coordinate of the corresponding first vector.

[0089] The output of the max pooling operation may be a single vector having the same size as each corresponding first vector. In this way, the selection model may select which input device's data is used in the step of forming the output, i.e., which input device's second vector is received and processed by the first permutation invariant deep learning model in the fusion center.

[0090] The max pooling layer placed at the beginning of the selection model 424 makes the model permutation invariant, thereby making the overall architecture of the data fusion system permutation invariant.

[0091] Both the selection model 424 and the classification model 428 can be trained for the task using appropriate training data. Any suitable training method can be used.

[0092] In the implementation above, the task is one of single-class classification, i.e., each data point (or a group of data points in the case of data fusion with multiple inputs) is associated with exactly one class. However, the technique can also be used for more complex problems and can be applied to outputs other than classification results.

[0093] In the above example, the selection model 424 and the classification model 428 are deep neural networks (DNNs). However, other model architectures may also be used.

[0094] In a preferred implementation, the selection model 424 and the classification model 428 are permutation invariant deep learning models. The benefit of choosing this implementation is that the process is applicable to dynamic data fusion settings.

[0095] This process can explicitly address the transfer size issue during data fusion, as the size of the first and second vectors sent by each input device can be adjusted to comply with the predefined limit on the transfer size.

[0096] Another advantage of our process is that it is very general and can be applied in a variety of settings (e.g., tasks other than classification, unimodal or multimodal input data, and dynamic or static settings).

[0097] Figure 6 6 is a flowchart summarizing an example of a method for performing data fusion provided by an embodiment of the present application. At step 601, the method includes: receiving a corresponding first vector from each input device of a plurality of input devices. At step 602, the method includes: selecting a subset of the plurality of input devices according to the corresponding first vector. At step 603, the method includes: receiving a corresponding second vector from each input device in the subset of the plurality of input devices. At step 604, the method includes: processing the corresponding second vector using a first permutation invariant deep learning model to form an output.

[0098] The following describes the experiments conducted to evaluate the performance of the approach described in this paper.

[0099] These experiments are limited to a single-modality dynamic data fusion setting.

[0100] Figure 7 The construction of an exemplary data set to be processed is shown. The Modified National Institute of Standards and Technology (MNIST) database is a large database of handwritten digits that is commonly used to train various image processing systems. There are 10 categories, and each image belongs to a category between 0 and 9. In order to construct a data set for dynamic data fusion, each image (for example, the image shown in 701) is slightly modified after being enlarged, as shown in 702. Then, parts of the image are randomly extracted, as shown in 703 and 704. The number of parts of the image corresponds to the number of sensors (input devices) used.

[0101] In this example, the sensors are cameras. Each camera can only perceive a portion of the environment. By using the data from each sensor, the image is classified.

[0102] For comparison purposes, two other data fusion processes are used. The first is a classical process without a selection step. In this process, there is only one classification step, where each sensor sends its locally processed features to the fusion center, which fuses these features and outputs the classification result. This process is called "no selection".

[0103] Another data fusion process is to randomly select a subset of sensors as the first step. No transmission is performed in this step. Then, the non-selection process is applied to this randomly selected subset of sensors.

[0104] In order to compare these different processes, the size of the transmission data in one process is fixed to 180 (the total number of coordinates of the first vector and the second vector). In the case of the same transmission size, these data fusion processes are used to classify the sensor image.

[0105] In the experiment, there are 30 sensors and thus the image has 30 parts (hereafter referred to as quadrants).

[0106] For each process, a similar permutation-invariant deep learning model was used to focus the comparison on the process used.

[0107] The configurations used are as follows. Configuration A uses the two-step selection method described in the paper and selects 4 quadrants with 15 features per quadrant. Configuration B uses the two-step selection method described in the paper and selects 3 quadrants with 30 features per quadrant. Configuration C uses no selection and selects 6 features per quadrant. Configuration D uses a random selection method and selects 27 quadrants with 7 features per quadrant.

[0108] The results are as follows Figure 8 As shown. For each configuration, the training accuracy (Train Acc), training loss (Train Loss), test accuracy (Test Acc), and test loss (Test Loss) are quantified. These results show that the two-step process used in configuration A and configuration B is at least 3.06% higher in accuracy than the other two processes. Therefore, when the transfer size is fixed, the process described in this article outperforms the other processes used. When the performance level is fixed, the bandwidth used by this process is smaller than that used by other processes. The optimization goal is to maximize the ratio of performance to transfer size.

[0109] In summary, the data fusion method proposed in this application for use in the telecommunications field has advantages in terms of performance and transmission size, and can also be extended to a variety of data fusion settings.

[0110] Previous deep learning approaches perform device selection while performing data fusion. However, performing the selection at the fusion center level does not reduce the transmission size. In contrast, the approach described in this paper only allows a specific subset of sensors to send their larger second vectors to the fusion center to form the output. Therefore, when applied in the telecommunications field, this approach can reduce data transmission.

[0111] Applicants hereby disclose separately each individual feature described herein and any combination of two or more such features. With the common knowledge of those skilled in the art, such features or combinations can be implemented as a whole according to this specification, without considering whether such features or combinations of features can solve any problem disclosed herein; and without limiting the scope of the claims. Applicants point out that aspects of the present application may include any such individual features or combinations of features. In view of the above description, it will be obvious to those skilled in the art that various modifications can be made within the scope of the present application.

Claims

1. A device (319, 408) for performing data fusion, characterized in that The device is used for: receiving (601) a respective first vector (313, 314, 315, 316, 317, 318, 416, 417, 418, 419, 420, 421, 422) from each input device of a plurality of input devices (307, 308, 309, 310, 311, 312, 401, 402, 403, 404, 405, 406, 407); selecting (602) a subset (403, 405, 407) of the plurality of input devices based on the corresponding first vectors; receiving (603) a respective second vector (425, 426, 427) from each input device in the subset (403, 405, 407) of the plurality of input devices; The corresponding second vector is processed (604) using a first permutation invariant deep learning model (428) to form an output (320, 429).

2. The device (319, 408) according to claim 1, characterized in that The device is used to select the subset (403, 405, 407) of the multiple input devices based on the output of a selection model (424) implemented by the device, wherein the selection model is used to receive the corresponding first vectors (416, 417, 418, 419, 420, 421, 422) as input.

3. The device (319, 408) according to claim 2, characterized in that The selected model (424) is a second permutation invariant deep learning model.

4. The device (319, 408) according to claim 2 or 3, characterized in that Each corresponding first vector includes a plurality of coordinates, and the selection model is used to select the subset of the plurality of input devices by performing a maximum pooling operation on each coordinate of the corresponding first vector.

5. The device (319, 408) according to claim 4, characterized in that The output of the max pooling operation is a single vector having the same size as each corresponding first vector.

6. The device (319, 408) according to any of the preceding claims, characterized in that The device is further for sending a signal to each input device of the subset of the plurality of input devices, wherein the signal comprises a request for the corresponding second vector for each input device of the subset of the plurality of input devices.

7. The device (319, 408) according to any of the preceding claims, characterized in that Each corresponding first vector (313, 314, 315, 316, 317, 318, 416, 417, 418, 419, 420, 421, 422) has a first magnitude, and each corresponding second vector (425, 426, 427) has a second magnitude that is greater than the first magnitude.

8. The device (319, 408) according to claim 7, characterized in that The sum of the first size of the corresponding first vector (313, 314, 315, 316, 317, 318, 416, 417, 418, 419, 420, 421, 422) and the second size of the corresponding second vector (425, 426, 427) defines a total transmission size of data received by the device (319, 408) for forming the output (320, 429).

9. The device (319, 408) according to any of the preceding claims, characterized in that The corresponding first vectors (313, 314, 315, 316, 317, 318, 416, 417, 418, 419, 420, 421, 422) are corresponding outputs of the first data processing model implemented by each corresponding input device (307, 308, 309, 310, 311, 312, 401, 402, 403, 404, 405, 406, 407) of the multiple input devices.

10. The device (319, 408) according to any of the preceding claims, characterized in that The respective second vectors (425, 426, 427) are respective outputs of a second data processing model implemented by each respective input device (403, 405, 407) of the plurality of input devices.

11. The device (319, 408) according to any of the preceding claims, characterized in that The corresponding first vectors (416, 417, 418, 419, 420, 421, 422) and the corresponding second vectors (425, 426, 427) are feature vectors extracted from the corresponding input images (409, 410, 411, 412, 413, 414, 415).

12. The device (319, 408) according to claim 11, characterized in that The corresponding input images (409, 410, 411, 412, 413, 414, 415) represent views of a static scene or a dynamic scene, respectively.

13. The device (319, 408) according to any of the preceding claims, characterized in that The output (320, 429) of the device is a classification result.

14. A method (600) for performing data fusion, characterized in that: The method comprises: receiving (601) a respective first vector (313, 314, 315, 316, 317, 318, 416, 417, 418, 419, 420, 421, 422) from each input device of a plurality of input devices (307, 308, 309, 310, 311, 312, 401, 402, 403, 404, 405, 406, 407); selecting (602) a subset (403, 405, 407) of the plurality of input devices based on the corresponding first vectors; receiving (603) a respective second vector (425, 426, 427) from each input device of the subset of the plurality of input devices; The corresponding second vector is processed (604) using a first permutation invariant deep learning model (428) to form an output (320, 429).

15. A computer-readable storage medium storing computer-readable instructions, characterized in that: The computer readable instructions, when executed by a computer system, cause the computer system to perform the method according to claim 14 .

Images