A multi-device fusion perception method and apparatus

Abstract

The embodiment of the application provides a kind of multi-device fusion perception method and device, it is related to artificial intelligence AI technical field, solve the problem of big power consumption overhead caused by continuous calculation and inter-device communication in distributed technology.The method is executed according to the training model in electronic device, the method comprises: the original data collected by electronic device is extracted, and feature vector is obtained;Electronic device determines the weight corresponding to feature vector;When weight reaches threshold value, electronic device sends feature vector and the calculation result of weight to central device, and the calculation result is used for central device to determine perception target.When weight does not reach threshold value, electronic device determines not to send feature vector and the calculation result of weight to central device.The embodiment of the application is used for the process of inter-device communication in AI.

Description

TECHNICAL FIELD

[0001] Embodiments of the present application relate to the field of artificial intelligence (AI) technology, and in particular to a multi-device fusion perception method and device. BACKGROUND

[0002] Accurate perception of user activities, states, and context environment information is the basis for providing personalized and accurate services to users. Current mobile devices, wearable devices, and Internet of Things (IoT) devices are integrated with rich sensors and have strong computing capabilities, enabling the collection of visual, auditory, motion sensor, signal strength, and other modal signals for computing of human and environmental perception targets. With the continuous development of distributed technology, data interconnection and capability assistance between devices become possible, and perception capabilities can be realized not only by a single device but also by multiple devices to utilize multi-device perception from different perspectives and form a more comprehensive and three-dimensional understanding of the perception object.

[0003] If continuous perception is performed using mobile devices and wearable devices, the mobile devices and wearable devices are required to continuously collect data, continuously compute, and continuously communicate the computation results to achieve centralized / distributed fusion computing. Unlike vehicle-mounted and home multi-sensor fusion perception, the power of mobile devices, wearable devices, and other power-free devices is very valuable, and continuous computation and inter-device communication will cause non-negligible power consumption overhead, so low-power design is particularly important for mobile device fusion perception. SUMMARY

[0004] Embodiments of the present application provide a multi-device fusion perception method and device, which can solve the problem of large power consumption overhead caused by continuous computation and inter-device communication in distributed technology.

[0005] To achieve the above object, embodiments of the present application adopt the following technical solutions:

[0006] In a first aspect, a multi-device fusion perception method is provided, which is executed according to a trained model in an electronic device. The method includes: performing feature extraction on collected raw data by the electronic device (perception device) to obtain a feature vector; determining a weight corresponding to the feature vector by the electronic device; when the weight reaches a threshold value, sending the feature vector and a calculation result of the weight to a center device by the electronic device, the calculation result being used by the center device to determine a perception target; and when the weight does not reach the threshold value, determining by the electronic device not to send the feature vector and the calculation result of the weight to the center device. The perception target can be a detection target in a person or an environment.

[0007] Therefore, in the fusion perception method of the application, for a plurality of perception devices establishing a communication connection with the center device, the perception devices that do not meet the threshold condition can be prevented from reporting the calculation result of the feature vector and the weight to the center device through the screening of the threshold. The threshold is obtained by model training, and the perception device can determine whether to report the calculation result according to the model deployed in the perception device. When the calculation result is reported, the corresponding feature can be considered as a feature with high relevance to the perception target, and when the calculation result is not reported, the corresponding feature can be considered as a feature that does not affect the center device to determine the type of the perception target. In this way, the type of the perception target can be determined while reducing the communication overhead between the perception device and the center device. Meanwhile, on the center device side, the feature vector of the perception device that does not report the calculation result can be set to 0 during fusion calculation, so that the center device does not need to infer the perception target according to the calculation result of all the communication-connected perception devices, and the fusion calculation overhead on the center device side can be reduced.

[0008] In a possible design, the electronic device performs feature extraction on the collected raw data to obtain the feature vector, including: the electronic device takes the raw data as the input of a first neural network to obtain the feature vector, the first neural network including a multi-layer perceptron (MLP), or a convolutional neural network (CNN), or a deep neural network (DNN). The first neural network can be understood as an implementation of the feature extraction module in the application. That is, the electronic device can implement feature extraction through the first neural network, and the raw data can be data such as acceleration, angular velocity, and direction collected by a sensor in the electronic device. These data can be used to extract the feature vector through the first neural network, so as to subsequently calculate the weight and perform fusion calculation to infer the perception target according to the feature vector.

[0009] In a possible design, the electronic device determines the weight corresponding to the feature vector, including: the electronic device takes the feature vector as the input of a second neural network, and calculates the feature vector through the second neural network to obtain the weight corresponding to the feature vector. The second neural network includes a hidden layer, a fully connected layer, and an activation function. The weight can be understood as the degree of relevance of the feature vector corresponding to the weight to the perception target. The greater the weight, the greater the degree of relevance of the corresponding feature to the perception target. Therefore, when the weight is higher than a threshold, it can be considered that the corresponding feature has high relevance to the perception target, and can be reported to the center device for identifying the perception target. In this way, the features that are not higher than the threshold are not reported to the center device, and the communication overhead between the perception device and the center device can be reduced.

[0010] In a possible design, the threshold value is obtained by adding an L1 sparse representation regularization constraint to a calculation result of the threshold value during training of the training model; and when a regularization term of the L1 sparse representation regularization constraint is increased, the threshold value is also increased accordingly. That is, when the model is trained, the perception device can report as few features as possible to the center device when the reportable features can be inferred to the perception target. This requires obtaining a suitable threshold value. The present application can obtain a suitable threshold value by adding an L1 sparse representation regularization constraint during training, so that the features corresponding to the weights higher than the threshold value are reported to the center device, and the features corresponding to the weights lower than the threshold value are not reported, thereby reducing the communication overhead between the perception device and the center device.

[0011] In a second aspect, a multi-device fusion perception method is provided. The method is performed according to a training model in a center device, the center device communicates with a plurality of electronic devices (perception devices). The method comprises: determining, by the center device, whether a calculation result is received from the plurality of electronic devices, the calculation result being determined by an electronic device according to a feature vector corresponding to raw data collected by the electronic device and a weight corresponding to the feature vector; when the center device receives the calculation result from at least one of the plurality of electronic devices, performing, by the center device, feature fusion calculation according to the calculation results of the plurality of electronic devices to obtain a fusion result, the fusion result being used to represent a perception target of a user or an environment; and when the center device determines that the calculation result sent by any of the plurality of electronic devices is not received, setting, by the center device, all elements of the feature vector corresponding to the any electronic device to 0.

[0012] That is, for the center device side, not every perception device that establishes a communication connection with the center device needs to report a calculation result, which can reduce the communication overhead between the perception device and the center device. For the perception device that does not report a calculation result, the center device can set all elements of the feature vector corresponding to the perception device that does not report a calculation result to 0 when performing feature fusion calculation, or the center device can not consider the perception device that does not report a calculation result when performing feature fusion calculation, which can reduce the calculation overhead of the center device side.

[0013] In a possible design, the method further comprises: when the center device determines that the calculation result is not received from the plurality of electronic devices, the center device does not perform feature fusion calculation. It can be understood that the method can be periodically executed. If the center device does not receive the calculation result of all perception devices that establish a communication connection with the center device, the center device does not need to perform feature fusion calculation in the current period. This can reduce the calculation overhead of the center device side.

[0014] In a possible design, the center device performs data fusion calculation according to the calculation results of the plurality of electronic devices, to obtain a fusion result, including: the center device taking the calculation results of the plurality of electronic devices as inputs of a neural network, performing data fusion calculation by using the neural network, and obtaining the fusion result output by the neural network; wherein the neural network includes a multi-layer perceptron (MLP), or a convolutional neural network (CNN), or a deep neural network (DNN). The fusion result can be understood as a type of a final determined perception target. For example, the type of the perception target can represent an action of getting on a vehicle or an action of getting off a vehicle.

[0015] In a third aspect, an electronic device is provided, including a trained model, the model including: a feature extraction module, configured to perform feature extraction on collected raw data, to obtain a feature vector; a feature selection module, configured to determine a weight corresponding to the feature vector; the feature selection module is further configured to send the feature vector and a calculation result of the weight to a center device when the weight reaches a threshold value, the calculation result being used by the center device to determine a perception target; and when the weight does not reach the threshold value, the feature selection module is configured to determine not to send the feature vector and the calculation result of the weight to the center device.

[0016] In a possible design, the electronic device performs feature extraction on the collected raw data to obtain a feature vector, including: the electronic device taking the raw data as inputs of a first neural network, to obtain the feature vector, the first neural network including a multi-layer perceptron (MLP), or a convolutional neural network (CNN), or a deep neural network (DNN).

[0017] In a possible design, the electronic device determines a weight corresponding to the feature vector, including: the electronic device taking the feature vector as inputs of a second neural network, and performing calculation on the feature vector by using the second neural network, to obtain the weight corresponding to the feature vector. The second neural network includes a hidden layer, a fully connected layer, and an activation function.

[0018] In a possible design, the threshold value is obtained by adding an L1 sparse representation regularization constraint to a calculation result of the threshold value obtained in a training process when training the trained model; and when a regularization term of the L1 sparse representation regularization is increased, the threshold value is also increased accordingly.

[0019] In a fourth aspect, a center device is provided, the center device comprising a trained model, the center device being in communication with a plurality of electronic devices, the model comprising: a determination module configured to determine whether a computation result is received from the plurality of electronic devices, the computation result being determined by an electronic device according to a feature vector corresponding to raw data collected by the electronic device and a weight corresponding to the feature vector; and a feature fusion module configured to, when the center device receives the computation result from at least one of the plurality of electronic devices, perform feature fusion computation according to the computation results of the plurality of electronic devices to obtain a fusion result, the fusion result being used to represent a perception target of a user or an environment; wherein, when the center device determines that the computation result sent by any of the plurality of electronic devices is not received, the feature fusion module is configured to set all elements of the feature vector corresponding to the any of the plurality of electronic devices to 0.

[0020] In a possible design, the feature fusion module is further configured to: when the center device determines that the computation result is not received from the plurality of electronic devices, the center device does not perform the feature fusion computation.

[0021] In a possible design, the feature fusion module is configured to: take the computation results of the plurality of electronic devices as inputs of a neural network, perform data fusion computation by using the neural network to obtain a fusion result output by the neural network; and wherein the neural network comprises a multi-layer perceptron (MLP), or a convolutional neural network (CNN), or a deep neural network (DNN).

[0022] In a fifth aspect, an embodiment of the present application provides a computer-readable storage medium, comprising computer instructions, when the computer instructions are run on an electronic device, causing the electronic device to perform the multi-device fusion perception method in the first aspect and any possible implementation manner.

[0023] In a sixth aspect, an embodiment of the present application provides a computer program product, when the computer program product is run on a computer, causing the electronic device to perform the multi-device fusion perception method in the first aspect and any possible implementation manner.

[0024] In a seventh aspect, an embodiment of the present application provides a computer-readable storage medium, comprising computer instructions, when the computer instructions are run on an electronic device, causing the electronic device to perform the multi-device fusion perception method in the second aspect and any possible implementation manner.

[0025] In an eighth aspect, an embodiment of the present application provides a computer program product, when the computer program product is run on a computer, causing the electronic device to perform the multi-device fusion perception method in the second aspect and any possible implementation manner.

[0026] In a ninth aspect, an embodiment of the present application provides a system, which can include a central device and at least one electronic device in any possible implementation manner of any of the above aspects. The electronic device and the central device can perform the multi-device fusion perception method in any of the above aspects and any possible implementation manner. BRIEF DESCRIPTION OF DRAWINGS

[0027] Figure 1 A network architecture diagram of multi-device fusion computing is provided for an embodiment of the present application.

[0028] Figure 2 An architecture diagram of a perception device and a central device is provided for an embodiment of the present application.

[0029] Figure 3 A structure diagram within a device of a perception device and a central device is provided for an embodiment of the present application.

[0030] Figure 4 A flow diagram of training a model is provided for an embodiment of the present application.

[0031] Figure 5 A flow diagram of online running of a model is provided for an embodiment of the present application.

[0032] Figure 6 A flow diagram of training a model is provided for an embodiment of the present application.

[0033] Figure 7 A flow diagram of online running of a model is provided for an embodiment of the present application.

[0034] Figure 8 A structure diagram of an electronic device is provided for an embodiment of the present application.

[0035] Figure 9 A structure diagram of an intelligent device is provided for an embodiment of the present application.

[0036] Figure 10 A structure diagram of a central device is provided for an embodiment of the present application. DETAILED DESCRIPTION

[0037] For ease of understanding, some of the example-related concepts of the embodiments of the present application are described for reference. As shown below:

[0038] Artificial neural network: an operation model composed of a large number of nodes (or neurons) connected to each other. Each node represents a specific output function, called activation function. The connection between each two nodes represents the weighted value of the connected signal, which can be called weight, which is equivalent to the memory of the artificial neural network.

[0039] Hidden layer: layers other than the input layer and the output layer can be hidden layers. The hidden layer does not directly receive signals from the outside world, nor does it directly send signals to the outside world.

[0040] Multilayer perceptron (MLP): a feedforward artificial neural network model that maps multiple input data sets to a single output data set.

[0041] Convolutional neural network (CNN): a class of feedforward neural networks containing convolutional calculations and having a deep structure, which is one of the representative algorithms of deep learning. Convolutional neural networks have representation learning capabilities and can perform shift-invariant classification on input information according to their hierarchical structure.

[0042] Recurrent neural network (RNN): a class of recursive neural networks that take sequence data as input, perform recursion in the evolution direction of the sequence, and connect all nodes in a chain.

[0043] The embodiments of the present application are used for data communication between devices under distributed technology to determine the perception target in the character or environment. The present application trains a model, distributes the trained model to each device, and determines whether to send data to the central device according to the model, which can reduce the communication overhead between devices and the subsequent fusion calculation overhead.

[0044] At present, for the communication between devices under the existing distributed technology, in one way, the data of each sensor in multiple sensors is filtered and corrected to obtain the corrected data of multiple sensors. The inter-sensor fusion weight of data fusion between different sensors in multiple sensors is calculated. Based on the corrected data and the inter-sensor fusion weight, secondary weighted fusion is performed to obtain the final fusion result. Although this scheme calculates the weight of each feature participating in fusion perception, each feature is weighted to participate in fusion calculation. However, the present application selects the features participating in perception according to the weight of the features, and discards other features that do not participate in perception. The adaptation of the upper fusion network to the feature absence is considered during model training.

[0045] In another way, the plurality of functionalities can be offloaded from the application processor to the sensor hub processor by configuring the sensor hub processor of the sensor hub; and one or more context-aware applications can be executed using one or more sensors coupled to the sensor hub processor. However, this solution is managed by the central hub to manage the activities of each sensor, including the activities of managing power consumption, but this solution does not involve the method of protecting the power consumption management. The present application is to manage the activities of each sensing device autonomously, to determine whether to report the features, and the central device does not need to issue instructions to each sensing device.

[0046] In another way, the dynamic data collected by each sensor can be monitored and read in real time; the dynamic data is matched with the set simple event generation rule, if the matching is successful, a start signal is sent out to generate a simple event; the simple event is read to generate a simple event set, and the simple event set is filtered according to the set event filtering rule; the filtered simple event set is aggregated according to the set event aggregation rule to generate a complex event. It can be understood that this solution is to match the dynamic data with the set simple event generation rule, and if the matching is successful, a start signal is sent out. However, this solution deals with events and does not involve model calculation. The present application deals with the original data collected by the sensor, the sensing device extracts features from the data through the feature extraction network, and the central device fuses the features through the fusion network to obtain the sensing target. Moreover, the existing solution filters the events through the rules, and for the original data, the existing technology cannot use the rules to screen the original data, while the present application screens the features extracted from the original signal through the feature selection module, and the feature selection module is obtained by training with the model, which can select the features associated with the sensing target.

[0047] In another way, a multi-sensor selective measurement data fusion estimation algorithm is provided, which selects part of valuable data in each sensor measurement data and submits it to the fusion center, and the fusion of measurement data is performed at the fusion center, and then the target is tracked and estimated based on the fused measurement data. However, this algorithm is to detect the target object by image, to obtain the target position by the current upper layer fusion, to transmit the target position to the bottom layer sensors, to let the sensors predict the position of the target object, to intercept the picture near the target position and upload it to the center node for fusion calculation. It can be understood that this scheme needs the center node to transmit the target position to the bottom layer sensor nodes, and the sensor nodes calculate their own inspection positions according to the received target position. However, the present application does not need the center device to issue parameters to the sensing devices of the sub-network, but the sensing devices complete the feature selection independently. Moreover, the existing scheme is to predict the target position of each sensor node according to the target position issued by the center node, and to intercept part of the data near the predicted target position in the collected data and upload it to the center node. However, in the present application, each sensing device will first process the collected data as a whole to extract features, and then select the feature selection module to be activated according to the features, and report the overall features obtained by the activated sensing devices.

[0048] Therefore, as described above, in order to reduce the communication overhead between devices and the subsequent fusion calculation overhead, the present application proposes a fusion perception method based on multi-device low power consumption, which aims to enable each sensing device to independently process the collected raw data, extract features, and perform feature selection, and only report the features with high relevance to the perception target to the center device, so that the center device can perform upper layer fusion calculation based on the features with high relevance, thereby reducing the communication overhead between devices and the subsequent fusion calculation overhead.

[0049] Based on the method proposed in the present application, the present application provides a network architecture, as shown in Figure 1 The network architecture can be applied in the AI field, for example, specifically in the automatic driving field of the AI field. The network architecture can include a plurality of sensing devices 10, which include a center device 11. The center device 11 can be a device with stronger computing power in the plurality of sensing devices 10, for example, the center device 11 can be a mobile phone, a computer or a tablet computer, and the sensing devices 10 other than the center device can be a smart watch. For example, in the automatic driving field, the mobile phone receives the sensing data sent by the smart watch, and the mobile phone performs fusion calculation based on its own sensing data and the sensing data sent by the smart watch to obtain the current perception target of the person or the environment. As shown in Figure 2As shown, for example, by the mobile phone and the watch, the acceleration and the change in direction are perceived at different parts of the human body, and after the mobile phone performs fusion calculation, the human posture and action can be finely perceived. The human posture and action may be, for example, the drunk walking of the human, or the self-driving behavior, or the getting on and off the vehicle behavior, and the like.

[0050] The network architecture is applied, as shown, Figure 3 In the present application, as shown, a neural network can be trained first, which includes a plurality of bottom sub-networks and a top fusion network. The communication mode between the sub-networks and the fusion network can be Bluetooth communication, wireless Internet (Wi-Fi), and the like. For some edge devices in the network architecture, such as monitoring cameras, and the like, the fourth generation mobile communication technology (4G) communication can also be used.

[0051] Each sub-network corresponds to a perception device, and each perception device includes at least one sensor for collecting raw data of a human or an environment. The sub-networks and the fusion network can deploy the trained model, the model in the sub-network is used to extract features and calculate feature weights, and according to the feature weights, it is determined whether to report the features and the weighted features to the fusion network. The model in the fusion network is used to perform fusion calculation according to the weighted features to determine the perception target of the human or the environment.

[0052] As shown, Figure 3 The model in each sub-network can include a feature extraction module and a feature selection module, the feature extraction module is used to extract features according to the raw data collected by the sensor, and the feature selection module is used to calculate the weight of the extracted features. When the model of the sub-network is trained offline, the L1 sparse representation regularization constraint can be added to the weight of each feature output by the feature selection module, which is used to compress the number of activated feature selection modules. After the model is trained, the fusion network can be deployed on the center device, and the sub-networks can be deployed on the corresponding perception devices; the sub-networks report the extracted features and the weighted features to the center device; only the devices with activated feature selection modules report the features and the weighted features to the center device for subsequent fusion calculation, that is, the selective communication between the center network and the perception devices. The center network also starts the calculation only when the reported features are received, that is, the devices corresponding to the feature selection modules that are not activated do not report the features and the weighted values, which can reduce the communication overhead between the devices. When the center device starts the calculation, the data corresponding to the sub-networks that are not reported can be set to 0 for calculation, which can reduce the subsequent fusion calculation overhead. The center network can include a gated recurrent unit (GRU) network, which is a time sequence fusion model, and is used to implement the fusion calculation in the present application.

[0053] The offline training process for the sub-networks and fusion networks in this application will be described below. This offline training process can be executed on a personal computer (PC). The PC can train the model based on the raw data collected by the central device and the raw data collected by each sensing device.

[0054] like Figure 4 As shown, the offline training process of the model in this application may include:

[0055] 401. Construct an artificial neural network consisting of multiple lower-level sub-networks and an upper-level fusion network. Each sub-network corresponds to a sensing device and is responsible for extracting features from the data collected by that device. The fusion network is also a sensing device and is responsible for extracting features from the data collected by the central device.

[0056] 402. Each sensing device extracts features from the data collected by this device.

[0057] Each sensing device's sub-network may include a feature extraction module and a feature selection module. The feature extraction module is responsible for extracting features from the data collected by the corresponding device. Each sensing device may include multiple sensors for collecting data, and the feature extraction module can extract features based on the data collected by these multiple sensors.

[0058] For example, the feature extraction module may include a multi-layer neural network, which can be a HiddenLayer. The input to this multi-layer neural network is the raw data collected by the sensor. The raw data passes through multiple HiddenLayers to obtain output values, and the output value of the last Hidden Layer is used as a feature vector. This feature vector represents the features extracted from the data collected by this device. The multi-layer Hidden Layer can be one of several types of networks, such as MLP, CNN, and RNN.

[0059] 403. Each sensing device determines the importance of the extracted features to the sensing target.

[0060] For any sensing device, its feature selection module is used to evaluate the importance of the features extracted by the feature extraction module to the sensing target.

[0061] Exemplarily, the feature selection module can be a multi-layer neural network, the input of which is the feature vector output by the feature extraction module, and the output of which is a weight value of a feature after calculation by a multi-layer Hidden Layer, a full connection layer and a ReLu (rectified linear units) activation layer. The weight value is used to represent the importance of the feature to the perception target, or in other words, a score of the feature.

[0062] 404. An L1 sparse representation regularization constraint is added to the output of the feature selection module in each perception device, so as to compress the number of activated feature selection modules by penalizing the activation layer of the feature selection module.

[0063] The number axis can be understood as a training threshold value used to determine whether the feature selection module is activated. When the weight value is higher than or equal to the threshold value, it is determined that the corresponding feature selection module is activated; when the weight value is lower than the threshold value, it is determined that the corresponding feature selection module is not activated, and the corresponding weight value can be set to 0.

[0064] When the feature selection module is activated, the corresponding perception device needs to report the feature to the center device; when the feature selection module is not activated, the corresponding perception device does not report the feature to the center device, i.e., does not send communication signaling to the center device. Thus, the communication overhead between the perception device and the center device can be reduced, and the calculation overhead of the center device for fusion calculation can also be reduced.

[0065] Exemplarily, the sparsity of the output of the feature selection module can be adjusted by adjusting the size of the sparse representation regularization constraint. The larger the value of the regularization term, the higher the threshold value used to determine whether to activate the feature selection module, and the more feature selection modules are not activated, and the higher the correlation of the activated feature selection module to the corresponding input feature and the perception target.

[0066] 405. Each perception device inputs the weighted feature to the center device (upper layer fusion network) after weighting the output feature with the weight generated by the feature selection module.

[0067] That is, the input of the center device is the feature weighted by the feature and the weight by the multiple groups of sub-networks (multiple perception devices).

[0068] 406. The center device outputs a classification result of the perception target according to the weighted features reported by the multiple perception devices.

[0069] The center device can include a feature fusion calculation module for obtaining a classification result of the perception target according to the reported weighted features. The classification result is used to represent the type of the current human or environmental perception target, for example, the type is an action of getting on a vehicle or an action of getting off a vehicle, etc.

[0070] Exemplarily, the upper fusion network can select one of a plurality of networks, such as an MLP, a CNN, or an RNN. For the central device, the sub-network corresponding to the inactive feature selection module does not report the weighted features, and the central device defaults that the features of the sub-network are a feature vector with all 0s. When the central device performs fusion calculation, the central device defaults that the features corresponding to the sub-network are 0.

[0071] Before the training starts, that is, in step 401, 407, offline collection of data of a plurality of devices can be performed, and data labeling is completed. That is, data collection and data labeling occur before training. The data labeling completes data labeling of the model corresponding to the sub-network and completes data labeling of the model corresponding to the fusion network.

[0072] The plurality of sub-networks and the upper fusion network form a whole network, and the perception model is trained through labeled data.

[0073] After the training is completed, the model of the fusion network can be deployed on the central device, and the model corresponding to the sub-network can be deployed on the perception device to implement online inference of the model. As shown in Figure 5 The online inference process can include the following steps.

[0074] 501. The feature extraction module of each perception device extracts features from the data collected by the device and calculates the weight of the features through the feature selection module.

[0075] 502. Each perception device compares the weight with a threshold.

[0076] 503. When the weight is greater than or equal to the threshold, the perception device reports the features weighted by the weight to the central device; when the weight is less than the threshold, the perception device sets the weight value to 0 and determines not to send communication signaling to the central device, that is, does not report the features.

[0077] In the present application, the communication mode of the perception device for data reporting is not limited. The reported data needs to be aligned through a synchronous clock, that is, the clock of each perception device and the central device needs to be synchronized, so that the central device receives the features reported by each perception device at the same time.

[0078] 504. After the central device receives the features reported by each perception device, the central device starts data fusion calculation and infers the perception target.

[0079] If no perception device reports the features, the central device does not start fusion calculation. If only part of the perception devices report the features, the feature fusion calculation module in the central device can be used to fill 0s in the features of the perception devices that do not report the features, and then perform fusion calculation.

[0080] In this way, by threshold screening whether the feature selection module is activated, the features reported by the perception device with the reporting action can be important for the perception target, or the reported features have a high correlation with the perception target. Therefore, sending the features with high correlation to the center device can also enable the center device to infer the perception target. In this way, when part of the perception devices do not report features, the communication overhead between the perception devices and the center device can be reduced. At the same time, for the center device, without calculating the features of all the perception devices that establish a communication connection, the overhead of the fusion calculation of the center device can also be reduced.

[0081] The following exemplary scenario is used to illustrate the model training of the PC end and the implementation of the actual application after the training.

[0082] For example, the scenario is to identify the user's boarding and alighting actions through the user's mobile phone and watch.

[0083] It can be understood that in this scenario, the perception devices are the mobile phone and the watch, and the mobile phone and the watch each include a data acquisition module, a feature extraction module, and a feature selection module; the center device is the mobile phone, which includes a feature fusion calculation module; and the mobile phone and the watch each include a communication module for transmitting instructions and data between the watch and the mobile phone.

[0084] In this scenario, after the mobile phone and the watch perception task (the task of inferring the perception target) is started, on the mobile phone and the watch, the data acquisition module is used to acquire the raw data of the acceleration sensor and the gyroscope sensor of the device, and send the raw data to the feature extraction module. The feature extraction module is used to extract features from the raw data, and the feature weight is evaluated by the feature selection module. When the weight meets the threshold requirement, the feature selection module is used to multiply the feature by the weight and report it to the feature fusion calculation module, and the feature fusion calculation module is used to infer the current action category of the user (the classification result of the perception target). Among them, the features of the mobile phone are locally transmitted to the feature fusion calculation module of the mobile phone, and the features of the watch are remotely transmitted to the mobile phone through the communication module.

[0085] According to the above scenario, when training the model, as shown in Figure 6 the training process can include:

[0086] 601, data acquisition.

[0087] a) The mobile phone and the watch acquire the raw data of the acceleration sensor and the gyroscope sensor in the mobile phone and the watch at a frequency of 50 Hz.

[0088] That is, the mobile phone includes an acceleration sensor and a gyroscope sensor, and the mobile phone includes an acceleration sensor and a gyroscope sensor.

[0089] Taking a watch as an example, the original data corresponding to the watch can include the azimuth pitch of the watch relative to the direction of gravity, the linear acceleration (excluding gravity acceleration) in three directions in the local coordinate system of the watch: acc_x, acc_y, acc_z, the angular velocity omega_x, omega_y, omega_z in three directions, a total of 8 dimensions of original data. Among them, the collection process can include signal processing processes such as noise reduction, normalization, gravity acceleration extraction, and device azimuth calculation, which are provided by the hardware driver layer.

[0090] b) The mobile phone and the watch select a time window with a length of 2s, slide with a step of 1s, collect the original data collected in the time window, splice the 8 dimensions of original data collected in the 2s time window into an 8*100 matrix, and take the matrix as the input data x (input original data x) of each round of calculation.

[0091] 602, feature extraction.

[0092] The PC end can extract the features (or feature maps) of the mobile phone and watch input data x through the 1D-CNN network.

[0093] Among them, the one-dimensional convolutional neural network (1D-CNN) is a type of CNN network, and the 1D-CNN network can be trained in a 2-layer convolution+pooling manner.

[0094] For example, the first layer can select 32 8*10 filters to convolve the input data x with a step stride=1, the activation function is ReLu, and 1*2 average pooling is performed to obtain a 32*45 feature map (feature map).

[0095] The second layer selects 64 16*5 filters to convolve the input data x with a step stride=1, the activation function is ReLu, and 1*4 average pooling is performed to obtain a 64*10 feature map.

[0096] After 2-layer convolution+pooling training, the feature map output by the mobile phone is denoted as v 1 ij The feature map output by the watch is denoted as v 2 ij .

[0097] 603, feature weight calculation.

[0098] a) The PC end calculates the weights of the mobile phone and the watch according to the feature maps of the mobile phone and the watch.

[0099] For example, the weight of the mobile phone or watch in the time window is denoted as c t When t = 1, c t may be expressed as:

[0100] denotes the linear superposition weight, and the c t The calculation method can be understood as linear superposition of the matrix elements of the output matrix of the 1D-CNN.

[0101] When t = 1, c1 denotes the weight of the feature corresponding to the mobile phone; when t = 2, c2 denotes the weight of the feature corresponding to the watch.

[0102] b) During training, an L1 sparse representation regularization term for the weight is added to the loss function;

[0103] l attn = λ∑ t |c t |, l attn denotes the L1 sparse representation regularization term.

[0104] The present application can adjust the L1 sparse representation regularization term in the present application by adjusting the value of λ, so as to adjust the threshold in the present application.

[0105] The loss function described above can be understood as the loss function of the entire model trained in the present application, and each weight in the model can be calculated by optimizing the loss function. The loss function in the present application can include: cross-entropy between the predicted value and the true value output by the feature fusion calculation module, L2 regularization term of the full connection layer output by the feature fusion calculation module, and L1 regularization term of the weight output by the feature selection module.

[0106] c) The feature output is: the feature weight multiplied by the feature u t = c t v t ij .

[0107] For example, when t = 1, the feature weight corresponding to the mobile phone multiplied by the feature is denoted as: u1 = c1 v 1 ij ;

[0108] When t = 2, the feature weight corresponding to the watch multiplied by the feature is denoted as: u2 = c2 v 2 ij .

[0109] If the weight c1=0, the feature weight of the mobile phone multiplied by the value of the feature u1 is also 0, and the mobile phone does not send the feature (u1) of the mobile phone to the feature fusion calculation module of the mobile phone through the local; on the contrary, when c1≠0, the mobile phone needs to send the feature (u1) to the feature fusion calculation module of the mobile phone through the local.

[0110] If the weight c2=0, the feature weight of the watch multiplied by the value of the feature u2 is also 0, and the watch does not send the feature u2 to the feature fusion calculation module of the mobile phone through the communication module of the watch; on the contrary, when c2≠0, the watch needs to send the feature (u2) to the feature fusion calculation module of the mobile phone through the communication module of the watch.

[0111] 604, Feature fusion calculation.

[0112] a) The PC splices the feature u1 output by the mobile phone and the feature u2 output by the watch into a 128x10 feature map, denoted as u ij ;

[0113] b) The input vector u ij is multiplied by the weight matrix w, and after softmax normalization, the prediction of the probability of each category is obtained

[0114] wherez n =∑ ij w n,ij u ij , wherein, represents the predicted value, which can be understood as the possibility of the occurrence of the nth perceived target; z n represents an intermediate quantity.

[0115] c) Based on the result of softmax, a loss function of multi-classification is defined by Cross Entropy. Cross Entropy is used to describe the difference between the predicted value and the true value.

[0116] d) In the training, the loss function adds L2 regularization to the fully connected layer to constrain the fully connected layer and avoid overfitting.

[0117] 605, Training.

[0118] a) In the user's getting on and off and other behavior actions, the data of the watch, mobile phone acceleration sensor and gyroscope sensor are collected, the data is sliced for 2s, and the class labels of the 2s actions are manually marked, which are getting on, getting off and other types respectively.

[0119] b) training the above 1D-CNN model using the collected dataset, the training process using stochastic gradient descent (SGD) to train the neural network parameters.

[0120] After the model is trained:

[0121] 1) Deploy the trained data collection module, feature extraction module, and feature selection module with the corresponding mobile phone and watch devices; deploy the feature fusion calculation module with the mobile phone; deploy the communication module on the mobile phone and watch, which can transmit instructions and data to each other;

[0122] 2) After the start of the boarding and alighting identification task, the mobile phone notifies the mobile phone and watch to start data collection, feature extraction, and feature selection; at the same time, the mobile phone and watch perform clock alignment;

[0123] 3) The mobile phone and watch collect the data of the acceleration sensor and gyroscope sensor of the device for the past 2 seconds every 1 second, extract the features, and calculate the feature weights through the respective feature selection modules. If the feature weight is not zero, the mobile phone and watch report the feature multiplied by the feature weight to the feature fusion calculation module of the mobile phone; among them, the mobile phone reports locally, and the watch reports to the mobile phone through the communication module; if the feature weight is zero, the feature is not reported;

[0124] 4) The mobile phone performs a round of inspection every 1 second. If the reported feature is received, the data fusion calculation is started, and the perception target is inferred. If no device reports data, the subsequent fusion calculation is not continued. If only the mobile phone reports the feature and the watch does not report the feature, the mobile phone fills the value of the feature of the watch with 0 and continues the subsequent fusion calculation; if only the watch reports the feature and the mobile phone does not report the feature, the mobile phone fills the value of the feature of the mobile phone with 0 and continues the subsequent fusion calculation.

[0125] It can be understood that the model provided in the present application includes a fusion network of the center device and a sub-network of each perception device, and multiple networks are trained together; the sub-network of the perception device includes a feature extraction module and a feature selection module; during training, a sparse representation regularization constraint is added to each feature weight; the input of the upper fusion network is the feature after sparse, and some perception devices do not report the feature, which does not affect the fusion effect, because the reported feature is a feature with high relevance to the perception target. Moreover, each perception device independently judges the association between the feature and the perception target, and only when a key feature is generated, the center device is reported for feature-level fusion; the center device also starts fusion calculation only when the feature is reported, which can reduce the communication overhead between the perception device and the center device, and also reduce the calculation overhead on the center device side.

[0126] According to the above description, the present application provides a multi-device fusion perception method, which can be applied in the field of AI, for example, in the field of automatic driving, etc.

[0127] The method is performed according to a trained model in an electronic device, and the threshold value in the following is obtained when L1 sparse representation regularization constraint is added when training the trained model; wherein when the regularization term of the L1 sparse representation regularization constraint is raised, the threshold value is also raised accordingly.

[0128] As shown in Figure 7 When the training is completed to obtain the trained model, and the model is deployed in the electronic device and the center device, the method includes:

[0129] 701. The electronic device extracts features from the collected raw data to obtain a feature vector.

[0130] The electronic device has the same meaning as the perception device in the above, for example, the electronic device can be a mobile phone or a watch, etc.

[0131] In some embodiments, the electronic device can take the raw data as the input of the first neural network to obtain the feature vector, and the first neural network can include MLP, or CNN or DNN.

[0132] Among them, the raw data is the corresponding raw data of a person or an environment collected by each sensor in the electronic device, and the sensor can be an acceleration sensor or a gyroscope sensor, etc., and the raw data can be acceleration, azimuth angle and angular velocity, etc.

[0133] One possible implementation of step 701 can refer to the implementation of step 602 described above.

[0134] It should be noted that steps 701-707 can be periodically executed.

[0135] 702. The electronic device determines the weight corresponding to the feature vector.

[0136] The electronic device takes the feature vector as the input of the second neural network, and calculates the feature vector through the second neural network to obtain the weight corresponding to the feature vector. The second neural network includes Hidden Layer, fully connected layer and activation function. Then, step 703 or 704 is executed.

[0137] One possible implementation of step 702 can refer to the implementation of step 603 described above.

[0138] 703. When the weight reaches the threshold value, the electronic device sends the feature vector and the calculation result of the weight to the center device, and the calculation result is used for the center device to determine the perception target.

[0139] The threshold is obtained by L1 sparse representation regularization constraint, and thus, when the weight corresponding to the feature vector of the electronic device is greater than the threshold, it can be understood that the feature corresponding to the feature vector has a higher correlation with the perception target, and is more helpful for the central device to determine the type of the perception target.

[0140] The calculation result can be the result of the multiplication of the feature vector and the weight. For example, the calculation result has the same meaning as u t in step 603 in the foregoing.

[0141] 704. When the weight does not reach the threshold, the electronic device determines not to send the calculation result of the feature vector and the weight to the central device.

[0142] When the weight corresponding to the feature vector of the electronic device does not reach the threshold, it can be understood that the feature corresponding to the feature vector has a lower correlation with the perception target, and the calculation result of the feature vector and the weight can not be sent to the central device, and the determination of the type of the perception target by the central device is not affected when the calculation result is not reported.

[0143] 705. The central device determines whether the calculation result is received from the plurality of electronic devices, the calculation result being determined by the electronic device according to the feature vector corresponding to the collected original data and the weight corresponding to the feature vector. Then, step 706 or step 707 is performed.

[0144] In the present application, the electronic device needs to determine whether the calculation result of the feature needs to be reported to the central device according to the model, and thus, the situation that all electronic devices do not report can occur. Therefore, before performing feature fusion calculation, the central device determines whether the calculation result of the plurality of electronic devices is received, and if it is determined that the calculation result is received, the central device determines that the feature fusion calculation needs to be started.

[0145] 706. When the central device receives the calculation result from at least one of the plurality of electronic devices, the central device performs feature fusion calculation according to the calculation result of the plurality of electronic devices to obtain a fusion result, and the fusion result is used to represent the perception target of the user or the environment.

[0146] When the central device determines that the calculation result sent by any of the plurality of electronic devices is not received, the central device sets all elements of the feature vector corresponding to any of the electronic devices to 0.

[0147] In some embodiments, the central device can take the calculation result of the plurality of electronic devices as the input of the neural network, perform data fusion calculation by using the neural network, and obtain the fusion result output by the neural network.

[0148] The neural network includes MLP, or CNN, or deep DNN.

[0149] One possible implementation of step 706 can refer to step 604 described above.

[0150] 707、When the center device determines that no calculation result is received from the plurality of electronic devices, the center device does not perform feature fusion calculation.

[0151] That is, if the center device determines that none of the plurality of electronic devices that establish a communication connection with the center device reports a calculation result, the center device determines that no feature fusion calculation needs to be performed in the current period.

[0152] Therefore, in this application, when the center device establishes a communication connection with the plurality of electronic devices, in each period, according to the screening of the threshold, not every electronic device can send the calculation result of the feature and the weight to the center device, which can reduce the communication overhead between the electronic device and the center device. The center device can also not need to perform fusion calculation according to the calculation result reported by each electronic device, which can also reduce the calculation overhead of the center device side.

[0153] It can be understood that, in order to implement the above functions, the electronic device and / or the center device comprises corresponding hardware and / or software modules for executing each function. The algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in hardware or a combination of hardware and computer software. Whether a certain function is implemented in hardware or computer software driven hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application in conjunction with the embodiments, but such implementation should not be considered beyond the scope of the present application.

[0154] The present embodiment can divide the functional modules of the electronic device and / or the center device according to the above method examples, for example, each functional module can be divided according to each function, or two or more functions can be integrated in one processing module. The integrated module can be implemented in the form of hardware. It should be noted that the division of modules in the present embodiment is illustrative, and is only a logical functional division. Actual implementation can have another division manner.

[0155] In the case of dividing each functional module according to each function, Figure 8 A possible composition schematic diagram of the electronic device 80 (perception device) involved in the above embodiments is shown, as shown in the figure, the electronic device 80 can include a feature extraction unit 801, a feature evaluation unit 802 and a feature reporting unit 803. Figure 8

[0156] The feature extraction unit 801 can be used to support the electronic device 80 to perform steps 501, 701, etc. described above, and / or for other processes of the technology described herein.​

[0157] The feature evaluation unit 802 can be configured to support the electronic device 80 to perform the step 502, the step 702, etc. described above, and / or other procedures of the techniques described herein.

[0158] The feature reporting unit 803 can be configured to support the electronic device 80 to perform the step 503, the step 703, the step 704, etc. described above, and / or other procedures of the techniques described herein.

[0159] It should be noted that all relevant content of each step involved in the above method embodiments can be cited from the function description of the corresponding function module, which will not be repeated here.

[0160] The electronic device 80 provided in the embodiment is configured to perform the multi-device fusion perception method described above, and thus can achieve the same effects as the implementation method described above.

[0161] In the case of using integrated units, the electronic device 80 can include a processing module, a storage module, and a communication module. The storage module can include programs and data corresponding to the model stored in the electronic device 80, and the communication module can include the functions of the feature reporting unit 803 described above. The processing module can be configured to control and manage the actions of the electronic device 80, for example, can be configured to support the electronic device 80 to perform the steps performed by the feature extraction unit 801 and the feature evaluation unit 802 described above. The storage module can be configured to support the electronic device 80 to store program codes and data, etc., such as the model trained in the present application. The communication module can be configured to support the electronic device 80 to communicate with other devices, such as the center device.

[0162] The processing module can be a processor or a controller. It can realize or execute various exemplary logical blocks, modules, and circuits described in combination with the disclosure of the present application. The processor can also be a combination of computing functions, such as a combination of one or more microprocessors, a combination of digital signal processing (DSP) and microprocessors, etc. The storage module can be a memory. The communication module can be a device for interacting with other electronic devices, such as a radio frequency circuit, a Bluetooth chip, a Wi-Fi chip, etc.

[0163] In one embodiment, when the processing module is a processor, the storage module is a memory, and the communication module is a transceiver, the electronic device involved in the embodiment can be a smart device with the structure as shown in the figure. Figure 9

[0164] In the case of dividing each function module according to each function, Figure 10 ​A possible component diagram of the center device 100 (one of the perception devices) involved in the above embodiment is shown in FIG. 10, which shows that the center device 100 can include a determination unit 1001 and a feature fusion unit 1002. Figure 10

[0165] The determination unit 1001 can be configured to support the center device 100 to perform the above step 705 and the like, and / or other procedures of the techniques described herein.

[0166] The feature fusion unit 1002 can be configured to support the center device 100 to perform the above step 706, step 707 and the like, and / or other procedures of the techniques described herein.

[0167] It should be noted that all relevant content of each step involved in the above method embodiments can be cited from the function description of the corresponding functional module, which will not be repeated here.

[0168] The center device 100 provided by the embodiment can be used to perform the multi-device fusion perception method described above, and thus can achieve the same effects as the implementation method described above.

[0169] In the case of integrated units, the center device 100 can include a processing module, a storage module and a communication module.

[0170] The processing module includes a determination module (with the same function as the determination unit 1001) and a feature fusion module (with the same function as the feature fusion unit 1002), the storage module can include programs and data corresponding to the model stored in the center device 100, and the communication module can be used to receive the data sent by the feature reporting unit 803. The processing module can be used to control and manage the actions of the center device 100, for example, it can be used to support the center device 100 to perform the steps performed by the determination unit 1001 and the feature fusion unit 1002. The storage module can be used to support the center device 100 to store program codes and data, etc., such as the model trained in the present application. The communication module can be used to support the center device 100 to communicate with other devices, such as other perception devices.

[0171] The processing module can be a processor or a controller. It can realize or execute various exemplary logic blocks, modules and circuits described in combination with the disclosure of the present application. The processor can also be a combination of computing functions, such as one or more microprocessor combinations, DSP and microprocessor combinations, etc. The storage module can be a memory. The communication module can be a radio frequency circuit, a Bluetooth chip, a Wi-Fi chip, etc. and other electronic devices interacting with devices.

[0172] ​In an embodiment, when the processing module is a processor, the storage module is a memory, and the communication module is a transceiver, the central device involved in the embodiment can be an intelligent device with the structure shown in FIG. 1. Figure 9

[0173] The embodiment of the present application also provides an electronic device / central device, which includes one or more processors and one or more memories. The one or more memories are coupled with the one or more processors, and the one or more memories are used to store computer program codes. The computer program codes include computer instructions, and when the one or more processors execute the computer instructions, the electronic device performs the steps of the related method to implement the multi-device fusion perception method in the above embodiment.

[0174] The embodiment of the present application also provides a computer storage medium, which stores computer instructions. When the computer instructions run on an electronic device, the electronic device performs the steps of the related method to implement the multi-device fusion perception method in the above embodiment.

[0175] The embodiment of the present application also provides a computer program product. When the computer program product runs on a computer, the computer performs the steps of the related method to implement the multi-device fusion perception method performed by the electronic device in the above embodiment.

[0176] In addition, the embodiment of the present application also provides a device, which can be a chip, a component or a module. The device can include a processor and a memory connected with each other. The memory is used to store computer execution instructions. When the device runs, the processor can execute the computer execution instructions stored in the memory, so that the chip performs the multi-device fusion perception method performed by the electronic device in the above method embodiments.

[0177] The electronic device, the computer storage medium, the computer program product or the chip provided in the embodiment are used to execute the corresponding method provided above, and thus the beneficial effects achieved by the electronic device, the computer storage medium, the computer program product or the chip can refer to the beneficial effects of the corresponding method provided above, which will not be described herein again.

[0178] Another embodiment of the present application provides a system, which can include the central device and the at least one electronic device, and can be used to implement the multi-device fusion perception method.

[0179] Through the description of the above embodiments, those skilled in the art can understand that, for the convenience and brevity of description, only the division of the above functional modules is taken as an example for illustration. In actual application, the above functions can be completed by different functional modules according to needs, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. ​

[0180] In several embodiments provided in the present application, it should be understood that the disclosed apparatus and method can be implemented by other manners. For example, the apparatus embodiments described above are merely illustrative, for example, the division of the modules or units is merely a logical function division, and actual implementation can have another division manner, for example, a plurality of units or components can be combined or integrated into another apparatus, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the units or components shown or discussed can be indirect coupling or communication connection through some interfaces, apparatuses or units, and can be electrical, mechanical or other forms.

[0181] The units described as separate components can or can not be physically separated, and the components shown as units can be one physical unit or multiple physical units, that is, can be located in one place or distributed to multiple different places. Part or all of the units can be selected according to actual needs to achieve the purpose of the embodiment scheme.

[0182] In addition, the functional units in each embodiment of the present application can be integrated in one processing unit, or each unit can be physically present separately, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of a software functional unit.

[0183] The integrated unit, if realized in the form of a software functional unit and sold or used as an independent product, can be stored in a readable storage medium. Based on such understanding, the technical solutions of the embodiments of the present application essentially or the parts that make contributions to the prior art or the whole or part of the technical solutions can be embodied in the form of a software product, which is stored in a storage medium and includes a plurality of instructions for causing an apparatus (which can be a single-chip microcomputer, a chip, etc.) or a processor to execute all or part of the steps of the method described in the embodiments of the present application. The foregoing storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and various storage medium that can store program codes.

[0184] The above is merely a specific implementation of the present application, but the protection scope of the present application is not limited thereto, and any person skilled in the art can easily think of changes or replacements within the technical scope disclosed in the present application, which should be covered in the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A multi-device fusion perception method, characterized in that, The method is executed according to a training model in an electronic device, the training model is implemented based on a constructed artificial neural network, the artificial neural network comprises a plurality of bottom-layer sub-networks and a top-layer fusion network, each of the sub-networks corresponds to one of the electronic devices, and the fusion network corresponds to a center device; and the method comprises: The electronic device extracts features from collected original data to obtain a feature vector; The electronic device determines a weight corresponding to the feature vector, the weight being used to represent a correlation degree of the feature vector with a perception target; When the weight reaches a threshold value, the electronic device sends a calculation result obtained by performing weighted processing on the feature vector and the weight to the center device, the calculation result being used by the center device to determine a perception target; When the weight does not reach the threshold value, the electronic device determines not to send the calculation result obtained by performing weighted processing on the feature vector and the weight to the center device.

2. The method of claim 1, wherein, The electronic device extracts features from collected original data to obtain a feature vector comprises: The electronic device takes the original data as an input of a first neural network to obtain the feature vector, the first neural network comprising a multi-layer perceptron (MLP), a convolutional neural network (CNN), or a deep neural network (DNN).

3. The method according to claim 1 or 2, characterized in that, The electronic device determines a weight corresponding to the feature vector comprises: The electronic device takes the feature vector as an input of a second neural network, and obtains the weight corresponding to the feature vector by performing calculation on the feature vector through the second neural network; The second neural network comprises a hidden layer, a fully connected layer, and an activation function.

4. The method according to claim 1 or 2, characterized in that, The threshold value is obtained by adding an L1 sparse representation regularization constraint when training the training model; When a regularization term of the L1 sparse representation regularization constraint is increased, the threshold value is increased.

5. A multi-device fusion perception method, comprising: The method is executed according to a training model in a center device, the training model is implemented based on a constructed artificial neural network, the artificial neural network comprising a plurality of bottom-layer sub-networks and a top-layer fusion network, each of the sub-networks corresponding to one of electronic devices, and the fusion network corresponding to the center device; The center device communicates with a plurality of electronic devices, and the method comprises: The center device determines whether to receive a calculation result from the plurality of electronic devices, the calculation result being determined by the electronic device by performing weighted processing on a feature vector corresponding to collected original data and a weight corresponding to the feature vector, the weight being used to represent a correlation degree of the feature vector with a perception target; When the center device receives the calculation result from at least one of the plurality of electronic devices, the center device performs feature fusion calculation according to the calculation results of the plurality of electronic devices to obtain a fusion result, the fusion result being used to represent a perception target of a user or an environment; When the center device determines that no calculation result sent by any of the plurality of electronic devices is received, the center device sets all elements of the feature vector corresponding to the any electronic device to 0.

6. The method of claim 5, wherein, The method further comprises: When the center device determines that no calculation result is received from the plurality of electronic devices, the center device does not perform feature fusion calculation.

7. The method according to claim 5 or 6, characterized in that, The center device performs data fusion calculation according to the calculation results of the plurality of electronic devices to obtain a fusion result. The center device takes the calculation results of the plurality of electronic devices as inputs of a neural network, and performs data fusion calculation by using the neural network to obtain the fusion result output by the neural network. The neural network includes a multi-layer perceptron (MLP), a convolutional neural network (CNN), or a deep neural network (DNN).

8. An electronic device, comprising: The electronic device includes a trained model implemented based on a constructed artificial neural network, the artificial neural network including a plurality of bottom-layer sub-networks and a top-layer fusion network, each of the sub-networks corresponding to one of the electronic devices, and the fusion network corresponding to the center device. The model includes: a feature extraction module configured to perform feature extraction on collected raw data to obtain a feature vector; a feature selection module configured to determine a weight corresponding to the feature vector, the weight being used to represent a correlation degree of the feature vector with a perception target; When the weight reaches a threshold value, the feature selection module is further configured to send a calculation result obtained by performing weighted processing on the feature vector and the weight to the center device, the calculation result being used by the center device to determine the perception target.

9. The electronic device of claim 8, wherein, When the weight does not reach the threshold value, the feature selection module is configured to determine not to send the calculation result obtained by performing weighted processing on the feature vector and the weight to the center device. The electronic device performs feature extraction on collected raw data to obtain a feature vector, including:

10. The electronic device of claim 8 or 9, wherein, The electronic device takes the raw data as inputs of a first neural network to obtain the feature vector, the first neural network including a multi-layer perceptron (MLP), a convolutional neural network (CNN), or a deep neural network (DNN). The electronic device determines a weight corresponding to the feature vector, including: The electronic device takes the feature vector as inputs of a second neural network, and obtains the weight corresponding to the feature vector by performing calculation on the feature vector by using the second neural network.

11. The electronic device of claim 8 or 9, wherein, The second neural network includes a hidden layer, a fully connected layer, and an activation function. The threshold value is obtained by adding an L1 sparse representation regularization constraint during training of the trained model.

12. A central device, characterized by When the regularization term of the L1 sparse representation regularization constraint is increased, the threshold value is increased. The center device includes a trained model implemented based on a constructed artificial neural network, the artificial neural network including a plurality of bottom-layer sub-networks and a top-layer fusion network, each of the sub-networks corresponding to one of the electronic devices, and the fusion network corresponding to the center device. The center device communicates with a plurality of electronic devices, and the model includes: a determination module configured to determine whether a calculation result is received from the plurality of electronic devices, the calculation result being determined by the electronic device by performing weighted processing on a feature vector corresponding to collected raw data and a weight corresponding to the feature vector, the weight being used to represent a correlation degree of the feature vector with a perception target. The feature fusion module is configured to perform feature fusion calculation according to the calculation results of the plurality of electronic devices to obtain a fusion result when the central device receives the calculation result from at least one of the plurality of electronic devices, and the fusion result is used to represent a perception target of a user or an environment. When the central device determines that the calculation result sent by any one of the plurality of electronic devices is not received, the feature fusion module is configured to set all elements of the feature vector corresponding to the any one of the plurality of electronic devices to 0.

13. The central device of claim 12, wherein, The feature fusion module is further configured to: When the central device determines that the calculation result is not received from the plurality of electronic devices, the central device does not perform feature fusion calculation.

14. The central device according to claim 12 or 13, characterized in that, The feature fusion module is configured to: use the calculation results of the plurality of electronic devices as inputs of a neural network, perform data fusion calculation by using the neural network, and obtain the fusion result output by the neural network; The neural network includes a multi-layer perceptron (MLP), a convolutional neural network (CNN), or a deep neural network (DNN).

15. A computer-readable storage medium, characterized in that, The computer program product includes computer instructions, and when the computer instructions are run on the electronic device, the electronic device performs the method in any one of claims 1-4.

16. A computer program product, characterised in that, When the computer program product is run on the computer, the electronic device performs the method in any one of claims 1-4.

17. A computer-readable storage medium, characterized in that, The computer program product includes computer instructions, and when the computer instructions are run on the electronic device, the electronic device performs the method in any one of claims 5-7.

18. A computer program product, characterised in that, When the computer program product is run on the computer, the electronic device performs the method in any one of claims 5-7.

Images