Systems and methods for robust and highly generalized failure sensor data detection for monitoring applications

Abstract

A method includes receiving sensor data from a sensor group including real-time localization sensors for performing localization of an object, and repeating, for multiple sensors, determining temporal dependencies of sensor data from the same sensor, and determining spatial dependencies between sensor data from different sensors in the sensor group. The method includes generating an indicator indicating whether sensor data associated with individual sensors in the sensor group is faulty sensor data, including inputting the spatial dependencies and the temporal dependencies into a convolutional neural network (CNN), and recovering faulty sensor data, including combining a version of the indicator and a version of the sensor data associated with the indicator to generate combined faulty data. The method includes inputting the combined faulty data into a multi-attention transformer neural network, and performing localization using the recovered non-faulty sensor data to find a localization of the object.

Description

[0001] introduce

[0002] This disclosure relates to the analysis of sensor data collected to monitor various conditions or parameters, and more particularly to the detection and recovery of faulty sensor data.

[0003] Sensors providing parametric data are used to monitor a wide variety of objects, systems, environments, and scenarios. This parametric data includes data for navigation, vehicle performance, biological health monitoring, traffic, weather, industrial machinery, residential, commercial and industrial buildings, computing and mobile devices, and many other applications. Faulty sensor data can occur due to varying environmental conditions around the sensor that may interfere with its operation (such as terrain obstructing wireless transmissions from the sensor), and sensors may have faulty data in many different environmental conditions or domains. When faulty sensor data occurs, it can be detected using machine learning techniques. Therefore, it is desirable to provide systems and methods for more robust faulty sensor data detection across a large number of varying domains without requiring excessively large training datasets. Furthermore, other desirable features and characteristics of this disclosure will become apparent from the following detailed description and appended claims, taken in conjunction with the accompanying drawings and the foregoing introduction. Summary of the Invention

[0004] In one example implementation, a method includes: receiving sensor data from a sensor group including real-time positioning sensors for performing object localization via at least one processor; and repeatedly determining, via at least one processor, the temporal dependency of sensor data from the same sensor in the sensor group and the spatial dependency between sensor data from different sensors in the sensor group for multiple sensors. The method further includes: generating, via at least one processor, an indicator indicating whether sensor data associated with each sensor in the sensor group is faulty sensor data, including inputting spatial and temporal dependencies into a convolutional neural network (CNN), recovering faulty sensor data into non-faulty sensor data, and combining a version of the indicator and a version of the sensor data associated with the indicator to generate combined faulty data. The method also includes: inputting the combined faulty data into a multi-attention transformer neural network, and performing localization using non-faulty sensor data to locate the object.

[0005] In another example implementation, the non-faulty sensor data is distance readings, and localization includes performing trilateration using the localization and distance readings of real-time localization sensors by applying a least-squares function or a density-based spatial clustering (DBSCAN) function with noise.

[0006] In another example implementation, the method includes formatting spatial dependencies into a spatial diagonal matrix, wherein the spatial dependency values ​​in the spatial diagonal matrix indicate the dependencies between each available sensor pair in the sensor group, and inputting the spatial diagonal matrix into a CNN.

[0007] In another example implementation, the method includes formatting the time dependencies into a time diagonal matrix, wherein the time dependency values ​​in the time diagonal matrix show the dependency between each available pair of different sample time points of a single sensor in the sensor group, and repeating this process for multiple sensors, and feeding the time diagonal matrix into a CNN.

[0008] In another example implementation, the CNN comprises a sequence of multiple CNN blocks to define a number of iterations and outputting intermediate features at each CNN block. These CNN blocks are repeating neural network blocks with the same neural network structure. The output of each block is an intermediate feature, and each output is a three-dimensional vector. This three-dimensional vector includes a first channel for a number of samples acquired over time from a group of sensors, a second channel for a number of sensors in the group, and a third channel, which is a feature channel. The feature channel has indicators in the form of multiple feature values ​​that collaboratively form a feature map and indicate a feature distribution. The CNN is trained such that the different output feature distributions from the CNN indicate fault data from different sensors.

[0009] In another example implementation, the method includes training a CNN by inputting indicators from the CNN output into both a fault sensor data classifier neural network and a domain adaptive neural network (DA-NN). The fault sensor data classifier neural network operates using only labeled training data, while the DA-NN operates using both labeled and unlabeled training data.

[0010] In another example implementation, training includes: the DA-NN generating dynamic weighting factors for a domain-adaptive loss used to modify the weights of the CNN, and the dynamic weighting factors depending on the amount of distributional difference, which is the difference between the source domain and the target domain.

[0011] In another example implementation, the amount of distributional difference is determined using the maximum mean difference (MMD).

[0012] In one example implementation, the amount of distribution difference is determined using a radial basis function (RBF) kernel.

[0013] In another example implementation, the method includes modifying the weights of the CNN using both the ground value loss from the fault sensor data classifier neural network and the domain adaptive loss from the DA-NN.

[0014] In another example embodiment, a system includes a memory, a processor circuitry forming at least one processor communicatively coupled to the memory, and arranged to operate in such a way as to receive sensor data from a sensor group including real-time positioning sensors for performing object positioning, repeatedly determining temporal dependencies of sensor data from the same sensors in the sensor group for multiple sensors, and determining spatial dependencies between sensor data from different sensors in the sensor group, and generating indicators indicating whether sensor data associated with each sensor in the sensor group is faulty sensor data, which includes inputting the spatial and temporal dependencies into a convolutional neural network (CNN). The at least one processor is further configured to: operate by recovering faulty sensor data into non-faulty sensor data, and includes combining a version of an indicator and a version of sensor data associated with the indicator to generate combined fault data, and inputting the combined fault data into a multi-attention transformer neural network to perform a monitoring task including using non-faulty sensor data, and pre-training a CNN including providing indicators to both a faulty sensor data classifier neural network and a domain adaptive neural network (DA-NN), the faulty sensor data classifier neural network and the domain adaptive neural network (DA-NN) being arranged to improve the CNN's generalization across domains that have the same set of sensors but have different environments between domains that affect the same set of sensors.

[0015] In another example implementation, the monitoring task involves health monitoring using multiple different types of sensors in a sensor array, which includes at least sensors for heart monitoring and human movement to collaboratively generate a health score.

[0016] In another example implementation, the monitoring task involves at least one of the following: navigation, vehicle performance and maintenance, biological health monitoring, traffic, weather, industrial machinery, residential, commercial and industrial buildings, utility systems and appliances, computers, computing devices and mobile devices, industrial machinery, environmental conditions, safety, occupancy, noise levels, light intensity, water usage, gas detection, smoke detection, vibration, structural integrity, waste management, alarm systems, asset tracking, worker safety, resource allocation, manufacturing processes, energy efficiency, cooling systems, machine condition, and equipment certification.

[0017] In another example implementation, at least one sensor in the sensor group has multiple dependencies, including both time-dependent and space-dependent dependencies.

[0018] In another example implementation, the CNN includes multiple blocks, which are repeated neural network blocks for several iterations. Each block has, in sequence: a spatial feature embedding layer that embeds spatial dependencies using: (1) intermediate features from the output of a previous block, or (2) sensor data values ​​used to form spatial dependencies and spatial enrichment features; a first subtractor for calculating a first difference between (1) the value of the intermediate feature or sensor data from the output of the previous block and (2) the value of the spatial enrichment feature; a first or more convolutional layers using the first difference; a first rectified linear unit (ReLU) layer using the results from the first or more convolutional layers and for forming a first ReLU result; a temporal feature embedding layer that receives temporal dependencies and embeds the temporal dependencies using the first ReLU result to generate temporal enrichment features; a second subtractor that calculates a second difference between the first ReLU result and the temporal enrichment features; a second or more convolutional layers using the temporal enrichment features; and a second ReLU layer using the results of the second or more convolutional layers and outputting the next intermediate feature.

[0019] In another example implementation, at least one non-transitory computer-readable medium includes instructions thereon that, when executed by a computing device, cause the computing device to operate.

[0020] In another example implementation, the instructions cause the computing device to operate in such a way as to generate an adjacency matrix by the following formula. normalization results Where D is the dihedral matrix of the sum of the connection weights of each dependency value, and It is a matrix of spatial or temporal dependency values. The connection weights are each the sum of the spatial or temporal dependency values ​​of the edges connected to each sensor value.

[0021] In another example implementation, the instructions cause the computing device to operate by multiplying a normalized adjacency matrix by feature channel data from a previous CNN block.

[0022] In another example implementation, the instructions cause the computing device to operate in such a way as to convert versions of the indicator and the sensor data into separate embeddings intended for location encoding, and to perform multivariate word embeddings to combine the separate embeddings into a combined vector, which will be fed into a multi-attention transformer neural network.

[0023] In another example implementation, the multi-attention transformer neural network includes a spatial encoder and a temporal encoder, the spatial encoder receiving a combined vector and the temporal encoder receiving the output from the spatial encoder and the concatenation of the combined vector.

[0024] This disclosure provides the following examples:

[0025] Example 1. A method comprising:

[0026] Sensor data is received from a sensor group, including a real-time positioning sensor for performing object positioning, via at least one processor.

[0027] Using at least one processor, the temporal dependence of sensor data from the same sensor in the sensor group and the spatial dependence between sensor data from different sensors in the sensor group are repeatedly determined for multiple sensors.

[0028] The processor generates an indicator indicating whether sensor data associated with each sensor in the sensor group is faulty sensor data, including inputting the spatial dependency and the temporal dependency into a convolutional neural network (CNN).

[0029] The process of recovering the faulty sensor data into non-faulty sensor data includes combining a version of the indicator and a version of the sensor data associated with the indicator to generate combined fault data; and inputting the combined fault data into a multi-attention transformer neural network; and

[0030] The positioning is performed using the non-faulty sensor data to locate the object.

[0031] Example 2. According to the method of Example 1, wherein the non-faulty sensor data is a distance reading, and wherein the positioning includes: performing trilateration using the positioning of the real-time positioning sensor and the distance reading by applying a least squares function or a density-based spatial clustering with noise (DBSCAN) function.

[0032] Example 3. The method according to Example 1, comprising: formatting the spatial dependencies into a spatial diagonal matrix, wherein the spatial dependency values ​​in the spatial diagonal matrix indicate the dependencies between each available sensor pair in the sensor group; and inputting the spatial diagonal matrix into the CNN.

[0033] Example 4. The method according to Example 1 includes formatting the time dependencies into a time diagonal matrix, wherein the time dependency values ​​in the time diagonal matrix show the dependency between each available pair of different sample time points of a single sensor in the sensor group, and repeating for multiple sensors, and inputting the time diagonal matrix into the CNN.

[0034] Example 5. According to the method of Example 1, wherein the CNN comprises a sequence of multiple CNN blocks to define a plurality of iterations and output intermediate features at each CNN block, the plurality of CNN blocks being repeating neural network blocks having the same neural network structure, wherein the output of each block is an intermediate feature and each output is a three-dimensional vector, the three-dimensional vector comprising a first channel for a plurality of samples acquired over time for the sensor group, a second channel for a plurality of sensors in the sensor group, and a third channel, the third channel being a feature channel having an indicator in the form of a plurality of feature values, the plurality of feature values ​​cooperatively forming a feature map and indicating a feature distribution, wherein the CNN is trained such that different output feature distributions from the CNN indicate fault data from different sensors.

[0035] Example 6. The method according to Example 1 includes training the CNN, which includes inputting the indicator output from the CNN into both a fault sensor data classifier neural network and a domain adaptive neural network (DA-NN), wherein the fault sensor data classifier neural network operates using only labeled training data, and the DA-NN operates using both labeled and unlabeled training data.

[0036] Example 7. According to the method of Example 6, wherein the training includes: the DA-NN generating a dynamic weighting factor for a domain adaptive loss used to modify the weights of the CNN, wherein the dynamic weighting factor depends on a distribution difference amount, the distribution difference amount being the difference between the source domain and the target domain.

[0037] Example 8. The method according to Example 6, wherein the amount of distribution difference is determined by using the maximum mean difference (MMD).

[0038] Example 9. The method according to Example 6, wherein the amount of distribution difference is determined by using a radial basis function (RBF) kernel.

[0039] Example 10. The method according to Example 6, comprising: modifying the weights of the CNN using both the ground value loss from the fault sensor data classifier neural network and the domain adaptive loss from the DA-NN.

[0040] Example 11. A system comprising:

[0041] memory,

[0042] A processor circuit system, forming at least one processor, said at least one processor being communicatively coupled to said memory and arranged to operate in such a way as:

[0043] Receive sensor data from a sensor array that includes real-time positioning sensors for performing object positioning;

[0044] The temporal dependency of sensor data from the same sensor in the sensor group is repeatedly determined for multiple sensors, and the spatial dependency between sensor data from different sensors in the sensor group is determined.

[0045] Generating an indicator that indicates whether sensor data associated with each sensor in the sensor group is faulty sensor data includes inputting the spatial dependency and the temporal dependency into a convolutional neural network (CNN).

[0046] The process of recovering the faulty sensor data into non-faulty sensor data includes combining the version of the indicator and the version of the sensor data associated with the indicator to generate combined fault data, and inputting the combined fault data into a multi-attention transformer neural network.

[0047] Perform a monitoring task, the monitoring task including using the non-faulty sensor data; and

[0048] Pre-training the CNN includes providing the indicator to both a fault sensor data classifier neural network and a domain adaptive neural network (DA-NN), which are arranged to improve the CNN's generalization across domains that have the same set of sensors but different environments between domains that affect the same set of sensors.

[0049] Example 12. The system according to Example 10, wherein the monitoring task involves health monitoring using multiple different types of sensors in the sensor group, the multiple different types of sensors including at least sensors for heart monitoring and human movement to collaboratively generate a health score.

[0050] Example 13. The system according to Example 10, wherein the monitoring task involves at least one of the following: navigation, vehicle performance and maintenance, biological health monitoring, traffic, weather, industrial machinery, residential, commercial and industrial buildings, utility systems and appliances, computers, computing devices and mobile devices, industrial machinery, environmental conditions, safety, occupancy, noise levels, light intensity, water usage, gas detection, smoke detection, vibration, structural integrity, waste management, alarm systems, asset tracking, worker safety, resource allocation, manufacturing processes, energy efficiency, cooling systems, machine condition, and equipment certification.

[0051] Example 14. The system according to Example 10, wherein at least one sensor in the sensor group has multiple dependencies including both time dependence and spatial dependence.

[0052] Example 15. The system according to Example 10, wherein the CNN comprises a plurality of blocks, the plurality of blocks being repeated neural network blocks for a plurality of iterations, wherein each block has, in sequence:

[0053] A spatial feature embedding layer is used to embed the spatial dependencies using: (1) intermediate features from the output of the previous block, or (2) sensor data values ​​used to form the spatial dependencies and to form spatial enrichment features.

[0054] The first subtractor is used to calculate the first difference between (1) the intermediate feature or sensor data from the output of the previous block and (2) the value of the spatial enrichment feature;

[0055] The first or more convolutional layers, using the first difference.

[0056] A first Corrected Linear Unit (ReLU) layer uses the results from the first one or more convolutional layers to form a first ReLU result.

[0057] A temporal feature embedding layer receives the temporal dependencies and embeds them using the first ReLU result to generate temporally enriched features.

[0058] A second subtractor calculates a second difference between the first ReLU result and the temporal enrichment feature.

[0059] The second or more convolutional layers, using the aforementioned temporal enrichment features, and

[0060] The second ReLU layer uses the results of the second or more convolutional layers and outputs the next intermediate feature.

[0061] Example 16. At least one non-transitory computer-readable medium, including instructions thereon, which, when executed by a computing device, cause the computing device to operate in such a way as:

[0062] Receive sensor data from a sensor array that includes real-time positioning sensors for performing object positioning;

[0063] The temporal dependency of sensor data from the same sensor in the sensor group is repeatedly determined for multiple sensors, and the spatial dependency between sensor data from different sensors in the sensor group is determined.

[0064] Generating an indicator that indicates whether sensor data associated with each sensor in the sensor group is faulty sensor data includes inputting the spatial dependency and the temporal dependency into a convolutional neural network (CNN).

[0065] The process of recovering the faulty sensor data into non-faulty sensor data includes combining the version of the indicator and the version of the sensor data associated with the indicator to generate combined fault data, and inputting the combined fault data into a multi-attention transformer neural network.

[0066] Perform a monitoring task, the monitoring task including using the non-faulty sensor data; and

[0067] Pre-training the CNN includes providing the indicator to both a fault sensor data classifier neural network and a domain adaptive neural network (DA-NN), which are arranged to improve the CNN's generalization across domains that have the same set of sensors but different environments between domains that affect the same set of sensors.

[0068] Example 17. According to the medium of Example 16, wherein the instructions cause the computing device to operate in such a way as to generate an adjacency matrix by the following formula. normalization results Where D is the dihedral matrix of the sum of the connection weights of each dependency value, and It is a matrix of spatial dependency values ​​or temporal dependency values, wherein the connection weights are each the sum of the spatial dependency values ​​or the sum of the temporal dependency values ​​having edges connected to the respective sensor values.

[0069] Example 18. According to the medium described in Example 17, wherein the instructions cause the computing device to operate in such a way as to multiply the normalized adjacency matrix by feature channel data from a previous CNN block.

[0070] Example 19. The medium according to Example 16, wherein the instructions cause the computing device to operate in such a way as to: convert the version of the indicator and the version of the sensor data into separate embeddings intended for location encoding; and perform multivariate word embeddings to combine the separate embeddings into a combined vector, which will be input into the multi-attention transformer neural network.

[0071] Example 20. The medium according to Example 19, wherein the multi-attention transformer neural network includes a spatial encoder and a temporal encoder, the spatial encoder receiving the combined vector, and the temporal encoder receiving the output from the spatial encoder and a concatenation of the combined vector. Attached Figure Description

[0072] The present disclosure will be described below in conjunction with the accompanying drawings. The drawings are not to scale and the reference numerals in the drawings denote similar elements, and wherein:

[0073] Figure 1 This is a schematic diagram of an example device having a system for detecting fault sensor data according to at least one embodiment of the embodiments described herein;

[0074] Figure 2 This is a schematic diagram of an example monitoring data analysis system according to at least one embodiment of the embodiments described herein;

[0075] Figure 3 This is a schematic diagram of an example sensor scenario for monitoring an object using a sensor according to at least one of the embodiments described herein;

[0076] Figure 4 It is based on at least one embodiment of the embodiments described herein. Figure 2 A schematic diagram of an example domain adaptive spatiotemporal graph convolutional neural network (DA-ST-GCN) unit in the system;

[0077] Figure 5 This is a schematic diagram of an example sensor data input sequence array according to at least one embodiment of the embodiments described herein;

[0078] Figure 6 This is a graph illustrating example time-dependent correlations of sensor data from a sensor according to at least one embodiment of the embodiments described herein;

[0079] Figure 7 It is a graph illustrating example spatial dependency correlations between sensor data from multiple sensors according to at least one embodiment of the embodiments described herein;

[0080] Figure 8 This is a schematic diagram of an example conceptual spatial adjacency sensor arrangement based on at least one embodiment of the embodiments described herein;

[0081] Figure 9 This is a schematic diagram of an example conceptual temporal adjacency sensor arrangement based on at least one embodiment of the embodiments described herein;

[0082] Figure 10This is a schematic diagram illustrating both spatial and temporal adjacency of an example conceptual sensor arrangement according to at least one embodiment of the embodiments described herein.

[0083] Figure 11 This is a schematic diagram illustrating the time and space dependence between a sensor and sensor readings according to at least one embodiment of the embodiments described herein.

[0084] Figure 12 This is a schematic diagram of an example dependency input matrix based on at least one of the embodiments described herein;

[0085] Figure 13 This is a schematic diagram of an example convolutional neural network (GCN) block according to at least one embodiment of the embodiments described herein;

[0086] Figure 14 This is a schematic diagram of a training arrangement of an example fault sensor identification neural network and an example domain classifier neural network according to at least one embodiment of the embodiments described herein.

[0087] Figure 15A-15G It is a diagram showing the distribution of domains for classifying domains according to at least one embodiment of the embodiments described herein;

[0088] Figures 16A-16B This is a schematic diagram of an example sensor data recovery system according to at least one embodiment of the embodiments described herein;

[0089] Figure 17 This is a schematic diagram of an example flowchart illustrating the detection of faulty sensor data and the denoising of the faulty sensor data to recover sensor values ​​according to at least one embodiment of the embodiments described herein; and

[0090] Figure 18 This is a schematic diagram of an example flowchart for training a fault sensor data detection neural network according to at least one embodiment of the embodiments described herein. Detailed Implementation

[0091] The following detailed description merely illustrates exemplary embodiments and is not intended to limit this disclosure or its application and use. Furthermore, there is no intention to be bound by any theory set forth in the foregoing background or the following detailed description.

[0092] Faulty sensor data may be caused by external sources such as the environment around the sensor, which may obstruct or reflect wireless communication or monitoring radiation or signals emitted from the sensor, and may be considered a temporary cause of the fault. In addition, permanent causes of faults include internal damage to the sensor or accidental movement of the sensor from its intended mounting location on an object (such as a vehicle) to an unintended location on the object, where the sensor is still operating and providing sensor data. Faults may manifest as missing measurements, including random missing values ​​with short-term reading errors, missing values ​​in time blocks (or long-term blockages), and non-line-of-sight (NLOS) measurement problems such as random jumps or persistent secondary path interference.

[0093] For any of these types of failures, the methods and systems disclosed herein for analyzing monitoring data include detecting faulty sensor data and optionally denoising (or restoring or cleaning sensor data) for a group of sensors with a common objective. This common objective can be many different objectives and is not particularly limited. Some common objectives may be a final score, such as a person's health score, or the location of an object within the sensor group. The disclosed methods and systems employ data-oriented machine learning approaches that enhance robustness by effectively identifying inaccurate measurements and denoising or restoring them to make them more accurate. This is achieved by analyzing the spatial and temporal correlations (or dependencies) between sensor data from the sensor group.

[0094] More specifically, spatial and temporal dependencies between sensor data are determined to form a dependency graph, which is then fed into a Domain Adaptive Spatiotemporal Graph Convolutional Neural Network (DA-ST-GCN), a model, or a unit (or simply a GCN unit). The GCN unit is a graph convolutional neural network (GCN) in the form of a sequence of repeating GCN blocks, which performs faulty sensor data (or anomaly) detection on the input sensor data in the dependency graph. This identifies faulty sensor data and, consequently, the source sensors from which the faulty sensor data originates from the group of sensors in use. Then, by combining both the separately embedded anomaly detection data and the embedded sensor data and feeding them into a multi-attention transformer neural network, recovered or cleaned sensor values ​​can be generated using the multi-attention transformer neural network. The transformer may have both a spatial encoder and a temporal encoder to handle spatial and temporal dependencies.

[0095] To train the GCN, two training neural networks were used: one network performed supervised learning using a labeled training dataset, while the other network performed semi-supervised learning using both labeled and unlabeled training data. The supervised neural network is a ground truth or fault sensor identification neural network (FS-ID NN) that monitors the data analysis system or model and provides a ground truth loss during training to modify the GCN's weights. The semi-supervised neural network is a domain adaptive neural network (DA-NN) that provides a domain adaptive loss, which is adjusted using dynamic domain adaptive parameters and then used to modify the GCN's weights. The DA-NN generalizes the GCN to handle a variety of domains applicable to the sensor arrays described in this paper.

[0096] Specifically, the disclosed GCN-enabled systems can be used in a wide variety of monitoring scenarios or domains with the same sensor set. For example, for a set of positioning sensors on a vehicle, the domain could be internal, external, in a garage, in severe weather, or with different terrains (such as mountains or forests). Generalization across various domains is achievable because cooperative sensor data (which may have mapped positioning or other mapping structures or concepts) has been found to possess hidden domain-independent correlations. Therefore, DA-NN can be used to provide good cross-domain performance, where cross-domain refers to changes in the environment surrounding the sensor set (or changes in the "application space"), or changes in the distribution of sensor readings. DA-NN improves generalization by using both labeled and unlabeled training data, reducing the requirement for a fully labeled dataset. It has also been found that GCNs trained with DA-NN are effective for a wide range of applications (which should not be confused with the various domains within a single sensor group described herein), where "various applications" refers to any different common objective of the sensors, such as monitoring for location or object detection, environmental monitoring (e.g., a geographical environment (related to security), weather, traffic or vehicle movement, computers, computing devices, smartphones and devices), software system monitoring, industrial machine monitoring, building monitoring (e.g., residential, commercial or industrial), etc., without any specific limitations. There are no specific limitations regarding the type or application of the sensor group, as long as all sensors in the sensor group directly or indirectly contribute to and influence the common objective.

[0097] Therefore, the implementation of the disclosed methods and systems better ensures robust performance across a wide variety of environments, thereby improving the robustness of end applications regardless of different environmental conditions and with relatively limited training data.

[0098] Now for reference Figure 1Example device or system 100 has a monitoring data analysis system 106, and device or system 100 can be a monitored device, or it can simply be a device having the monitoring data analysis system 106 described herein. The monitored object, environment, or scene may or may not be far from device 100. In one embodiment, device 100 is a vehicle such as a car, but can be any vehicle, whether it is an airplane, ship, spacecraft, etc., as long as it uses sensors to collect sensor data to monitor the condition of the vehicle and its components and systems 122 (such as steering systems, drive system engines, etc.), or to monitor the environment outside the vehicle (such as for navigation or positioning of remote objects).

[0099] Device 100 has a processor circuitry that forms one or more processors 102 to receive data from sensor 104 and operate a monitoring data analysis system 106, which may also be referred to herein as a fault sensor data detection system. Device 100 may also have other applications 108, such as navigation or autonomous driving applications that use sensor data. These applications 108 may be referred to as end applications, but include any application that can use sensor data from sensor 104.

[0100] Device 100 also includes a memory 110, which can store any operating system or application mentioned herein, and can also store data used to operate neural networks or other machine learning algorithms used by the monitoring data analysis system 106. This can include a graph convolutional neural network (GCN) 112, a recovery neural network (RES NN) 114, a fault sensor identification neural network (FS_ID_NN) 116 and / or a domain adaptive neural network (DA-NN) 118, and an NN database 120, which stores any network-related data used by any neural network herein. Network-related data includes, for example, any version of sensor data (which includes raw input data, intermediate feature data, and / or output data), layer weights, biases, training gradients, loss values, activation functions, and calculated values. As mentioned below, the hardware of the neural network can be considered as part of processor 102.

[0101] As an example, sensor 104 may include a sensor array or other sensor arrangement, and may include one or more of any desired type of camera, one or more other detection sensors (e.g., radar, sonar, light detection and ranging (LiDAR), infrared, real-time positioning sensor (RTLS), signal measurement sensor such as Wi-Fi or Bluetooth strength sensor, etc.) and / or other sensors (e.g., vehicle positioning sensor, speed sensor, accelerometer, gyroscope, inertial sensor, brake sensor, steering sensor, inertial measurement unit (IMU), etc.).

[0102] Other devices may have different types of sensors, and they are considered to be included here regardless of whether these sensors are located on the same device or far apart from each other. Therefore, sensor 104 may include any sensor for any monitoring arrangement, such as for monitoring vehicle performance and maintenance, navigation, health, traffic, weather, industrial machinery, robots, residential buildings, commercial buildings, industrial buildings, automation systems, appliances, computers, computing devices, mobile devices, environmental conditions, air quality, temperature, humidity, pressure, motion, energy use, safety, occupancy, noise levels, light intensity, water usage, gas detection, smoke detection, vibration, structural integrity, fault detection, performance optimization, waste management, asset tracking, worker safety, disease monitoring, vital signs data, sleep patterns, heart rate, blood pressure, glucose levels, ECG, brain activity, proximity detection, resource allocation, manufacturing processes, energy efficiency, cooling, heating and ventilation systems, machine condition, equipment certification, and many other uses. This may also include software sensors that monitor other software systems, such as software sensors that monitor autonomous driving systems on vehicles, which may also be considered as the sensor group defined herein. Many variations may be included.

[0103] Regarding the types of sensors within a sensor group sharing a common objective as discussed herein, the sensor group can be a group of sensors of the same type or a group of sensors of different types, as long as these sensors contribute sensor data to the common objective and the same sensor group is maintained for the operation of the monitoring data analysis unit 106. The common objective can be determining a monitoring score, such as a person's health score when monitoring various objects and things (such as heartbeat, blood components, human movement, etc.). Alternatively, the localization of the detected object or the determination of self-localization can be a common objective for all sensors of the same type (such as a group with only RTLS).

[0104] In more detail now, it will be understood that processor(s) 102 may be, or have a control system and / or controller to operate device 100, other units and systems. It will also be understood that device 100 may differ from... Figure 1The implementation described herein. For example, processor 102 may be coupled to or otherwise utilize one or more remote computer systems and / or other remote control systems, for example as part of one or more of the aforementioned units or systems of system 100.

[0105] In various embodiments, processor(s) 102 is part of or part of a computer system and includes memory 110 and a computer bus (not shown). In various embodiments, processor(s) 102 acquires sensor data from sensor 104, and in some embodiments, acquires additional data via one or more communication systems (such as transceivers) when sensor 104 or any other unit or system (or any part of a unit or system) is physically away from device 100 and processor(s) 102.

[0106] In the depicted embodiments, processor(s) 102 may include circuitry or circuitry for operating any unit or system shown on device 100, including circuitry for operating neural networks. Processor(s) 102 forms any type of processor or multiple processors, a single integrated circuit such as a microprocessor, or any suitable number of integrated circuit devices and / or circuit boards that work together to implement the functionality of a processing unit. This may include a system-on-a-chip (SoC) and one or more processor cores, and / or shared hardware circuitry such as with a central processing unit (CPU), digital signal processor (DSP), etc. In addition, dedicated or function-specific processors for operating neural networks (including convolutional neural networks, linear layers, etc.) may be provided, as well as other structures, such as integrated circuits (ASICs), field-programmable gate arrays (FPGAs), neural processing units (NPUs), graphics processing units (GPUs), image signal processors (ISPs), etc., for example, for sensor data analysis. During operation, processor 102 executes one or more programs or applications 106 or 108 that may be stored in memory 110, and thus, when provided for a vehicle, controls the general operation of the computer system of the controller or controller, and generally performs the processes described herein (such as... Figure 2-18 Control is performed when the process and implementation methods described herein are further described below.

[0107] Memory 110 can be any suitable type of memory and can include a single physical memory or multiple memories located on the same device or remotely to each other, or any combination thereof. For example, memory 110 can include various types of dynamic random access memory (DRAM) such as SDRAM, various types of static RAM (SRAM), cache, and various types of non-volatile memory (PROM, EPROM, and flash memory drives). In some examples, memory 110 is located on and / or co-located on the same computer chip as processor 102. In the depicted embodiments, memory 110 stores applications 106 and 108 referenced above, as well as one or more databases 120 (e.g., data about neural networks and / or any other data related to device 100 as described herein) and other stored values.

[0108] The bus mentioned above can also be provided on device 100 to transfer programs, data, status, and other information or signals between various components of a system operated by one or more processors 102. The bus can be any suitable physical or logical means of connecting one or more processors 102 to other computer systems and components. This includes, but is not limited to, direct hardwired connections, fiber optics, infrared, and wireless bus technologies. During operation, monitoring data analysis system 106 or other applications 108 can be stored in memory 110 and executed by one or more processors 102.

[0109] It will be understood that although this example implementation is described in the context of a full-featured computer system, those skilled in the art will recognize that the mechanisms of this disclosure can be distributed as applications in which one or more types of non-transitory computer-readable signal-bearing media serve as part of memory 110 and are used to store the application and its instructions, as well as to execute its distribution, such as non-transitory computer-readable media carrying the application and containing computer instructions stored therein for causing a computer processor (such as processor 102) to execute and run the application—and in particular, to execute and run the monitoring data analysis system 106 and neural networks 112, 114, 116, and 118. Such applications can take many forms, and this disclosure applies equally regardless of the specific type of computer-readable signal-bearing medium used for distribution. Examples of signal-bearing media include recordable media such as floppy disks, hard disks, memory cards, and optical disks, and transmission media such as digital and analog communication links. It will be understood that cloud-based storage and / or other technologies may also be used in some implementations. Similarly, it will be understood that processor(s) 102 can form a computer system or controller, which may differ from... Figure 1The implementation described herein differs, for example, in that (one or more) processors 102 may be coupled to or may otherwise utilize one or more remote computer systems and / or other control systems.

[0110] refer to Figure 2 Example implementation of the Monitoring Data Analysis System (MDAS) 200 (which is a fault sensor data detection system) and System 106 ( Figure 1 The MDAS200 has a sensor / monitoring data unit 202, a DA-ST-GCN backbone unit (or simply GCN unit) 204, an optional recovery (or denoising) unit 206, and an application unit 208 (which may or may not be considered part of the MDAS200). The MDAS200 also has units and inputs for training the neural network of the GCN unit 204. This includes inputs from the training unlabeled cross-domain data input unit 210 and the labeled source domain data input unit 212. The training units used to operate the training neural network include the training fault sensor recognition unit 214, the gradient inversion unit 216, and the domain classifier unit 218 (all shown in dashed lines), and they cooperate to form a training system 1400 (hereinafter referred to as...). Figure 14 (Detailed description).

[0111] In operation, sensor and / or monitoring data are received by GCN unit 204 with a convolutional neural network (CNN), also known as GCN 112. Figure 1 The GCN unit 204 generates intermediate features in the form of a feature distribution or feature map that represents the correlation between sensor data and identifies faulty sensor data. Therefore, different feature distributions will indicate whether sensor data is faulty and which sensor is the source of the faulty sensor data. The source sensor may be referred to herein as the faulty sensor, even if the sensor itself is not damaged. This refers to situations where the environment surrounding the sensor causes the fault (such as walls reflecting communication signals or trees blocking communication signals), and where the fault may be temporary.

[0112] Sensor data, along with intermediate features (also known as anomalous features), can optionally be provided to recovery unit 206 to denoise or replace faulty sensor data. The recovered or cleaned data is then provided to end application 208. Sensor reliability estimates 220 from GCN unit 204 can also be provided to application 208. With this arrangement, MDAS 200 performs data-oriented faulty sensor isolation and optionally data recovery.

[0113] To train the GCN and the fault sensor recognition neural network (FS-ID-NN) 116 of the fault sensor recognition unit 214, both labeled and unlabeled training data are provided from the training labeled source domain data unit 212 and the training unlabeled cross-domain data unit 210, respectively, and from the neural network training dataset. The FS-ID-NN 116 is operated by the fault sensor recognition unit 206 and is trained only on labeled data to establish supervised learning. The fault sensor recognition unit 206 determines the ground truth loss based on the output of the FS-ID-NN 116. Both unlabeled and labeled training data are provided to the GCN unit 204 to train the GCN 112, which is then used to provide intermediate features to the gradient inversion unit 216 (as placeholders) and then to the domain classifier unit 220 operating the DA-NN 118. The domain classifier 220 determines the domain loss (or adaptive domain loss). Both domain loss and ground truth loss are used to generate gradients, which in turn modify the weights of GCN 112 at GCN unit 204. Therefore, it can be said that the first stage of the fault sensor data detection process operates GCN 112, which is pre-trained on labeled and unlabeled data for accurate fault sensor detection, while in a second optional stage, system 200 can perform signal recovery based on both sensor readings and backbone features extracted from the first stage. Further details of the operation of MDAS 200 are described below.

[0114] refer to Figure 3 Sensor arrangement 300 illustrates a vehicle 302 with four real-time location sensors (RTLS) 306, 308, 310, and 312 that monitor objects 304, such as people (by locating keychains or smartphones carried by the person). This RTLS can be an ultra-wideband sensor. A processor 102 that receives sensor data from sensors 306, 308, 310, and 312 can be located at a controller on the vehicle or at another remote location communicating with the sensors or computer system or controller on the vehicle 302.

[0115] Using arrangement 300, the faulty sensor detection method and MDAS 200 disclosed herein can be mounted on vehicle 302, and system 200 will provide model robustness in terms of accurate sensor reading measurements. The disclosed system recovers (or denoises, replaces, or cleans) previously distorted sensor readings based on different conditions or environments, such as the time of flight (TOF) of RTLS sensors 306, 308, 310, and 312 transmitting line-of-sight (LOS) signals in ultra-wideband (UWB), and ranging accuracy which may be affected by multipath signals bounced or reflected from objects. The currently disclosed data-oriented method enhances positioning accuracy by identifying faulty sensor data and source sensors, and recovers inaccurate measurements from non-faulty sensor data.

[0116] The currently disclosed methods and systems are also trained to provide accurate results even when vehicle 302 moves to many different environments. Therefore, this method and system employ a training method that allows the use of a scaled-down training database (or dataset, or simply training data) to generalize the neural networks of the disclosed methods and systems to many different situations and environments (or domains as described herein). The methods and systems can be trained for specific applications (such as positioning as mentioned herein) with the same set or group of sensors, but for multiple domains (to name several examples besides those mentioned above: such as outdoor, garage interior, urban or town environments, and rural environments). This allows for training the neural network using a significantly smaller training dataset. This can be applied to many different sensors, including various positioning sensors such as proximity sensors, automotive safety sensors, indoor navigation sensors, object tracking sensors, and device-to-device communication sensors, etc.

[0117] For example, assuming this vehicle-based positioning example arrangement 300, example RTLS sensors 306, 308, 310, and 312 are non-line-of-sight (NLOS) Received Signal Strength Indication (RSSI) sensors (these sensors provide signal strength measurements using Wi-Fi or Bluetooth sensors), or time-of-flight (TOF) and angle-of-arrival (AOA) measurement sensors (providing tag-anchor bidirectional ranging data using UWB sensors). When large objects obstruct or interfere with communication (such as through trees or mountains), shadows (signal power fluctuations) and other effects may occur, resulting in distorted signals and degraded signals with significant loss. Therefore, such sensors struggle to handle multipath problems at certain sensor locations.

[0118] Sensor data from sensors 306, 308, 310, and 312 are input into the DA-ST-GCN backbone unit 204, and specifically into GCN 112, to perform feature extraction. Subsequently, intermediate features can also be used to perform multi-attention signal recovery. The results can be the identification results of the source sensors from the faulty sensor data, as well as the denoised sensor data, in this case, TOA and AOA.

[0119] In this example, the terminal (or downstream) application 208 can perform localization via trilateration by applying least squares (LS), nonlinear least squares (NLL), or density-based noisy spatial clustering (DBSCAN) functions to locate the target based on sensor placement coordinates and distance readings. Many other variations and examples can be used alternatively.

[0120] As another example, a sensor set can be provided for the common goal of a person's health score (or fitness data fusion). This can include various sensors on mobile (phones) or wearable devices (watches, fitness trackers, or skin-tight devices), clinical data, etc., which can provide sensor data such as acceleration or steps (i.e., movement of a person detected by the peak of the acceleration signal), and heart rate, electrocardiogram (ECG) readings from photoplethysmography (PPG). Failures may occur due to device placement, poor skin contact, temperature changes, etc. The system 200 and method allow the GCN unit 204 (and GCN 112) to receive input data and generate intermediate and final features including identification results of faulty sensor data (and one or more faulty sensors), and the recovery unit 206 can then use the intermediate or final features from GCN 112 to generate cleaned sensor data. The health monitoring application can then generate a more accurate fitness score estimate, which can then be used by other fitness and health applications.

[0121] Another example could include a smart city traffic monitor that generates GPS data, radar data, traffic flow data, and traffic speed data, and is sensitive to obstacles caused by weather conditions and building disturbances. This method and system also improves smart city operations. Many other examples for numerous other applications can be improved using this method and system.

[0122] refer to Figure 4The DA-ST-GCN backbone unit 204 has a dependency unit 400, which has a spatial dependency unit 406 and a temporal dependency unit 408. The dependency unit 400 computes the dependencies and provides them to the GCN 404, which is the same as GCN 112 and is shown here as the first spatiotemporal (ST) GCN block 1 (410) to the last GCN block N 413 in a plurality of GCN blocks 412. GCN blocks 1 to N (412) can also be referred to as CNN blocks because GCN 404 uses convolutional layers. The number of GCN blocks 412 may depend on the expected convergence rate during the training of GCN 404 and / or other factors, and at least two GCN blocks 412 are used by way of an example, and two to four GCN blocks 412 are used here. It has been found that the first GCN block focuses more on interdependencies that are closer or adjacent, while later GCN blocks in the sequence of GCN blocks 412 focus on sensor pairs with a greater spatial or temporal distance between the two sensors.

[0123] Sensor 104 can provide raw sensor data (or formatted sensor data) to dependency unit 400 to generate dependencies. The sensor data is also formatted by sequence data unit 402 to first place the sensor data into a sequence data array and then reformat the sensor data into the input vector structure desired by the first GCN block 1 (410).

[0124] For specific references Figure 5 A sequence data array or a sample format of Table 500 has each sensor reading or sensor data value placed in rows and columns. Each row represents a different sensor i in the sensor group of a total of I sensors being analyzed, and each column represents a different time point or sampling time t within the total sampling duration T used to collect sensor data. The duration and sampling time (or interval) can vary widely depending on the type of sensor used and the purpose of the sensor data. A sequence data array is used in sequence data unit 402 to store incoming sensor data and organize the sensor data for reformatting.

[0125] The sequence data can then be reformatted by sequence data unit 402 or another unit. This reformatting involves arranging the sensor data values ​​into a matrix or vector form desired by the first GCN block 1 (410). Thus, for each sensor value or reading, the sequence data is reformatted or transformed into a three-dimensional vector (m×I×X), where the first dimension m is the number of time intervals or time samples t in the duration T, the second dimension I is the total number of sensors in the sensor group, and the third dimension X is one or more sensor values. Therefore, for example, positioning is performed where each sensor provides a quantity as time of arrival (TOA) or d and a direction as angle of arrival (AOA) or θ. In this case, the transformation can be viewed as:

[0126]

[0127] In this example, assume there are five time intervals and four sensors in the sensor group being analyzed. The three-dimensional vector will be (5×4×2), where 5 and 4 are the simple counts of the intervals and sensors, respectively, and two features (TOA and AOA) are provided as vectors in the feature channels in the third dimension. Therefore, for the initial sensor data input, the feature channels can have as many features as a single sensor provides in a single output time, and each three-dimensional vector is provided to a single sensor. The input can be provided as a single channel or a neural network surface, or it can be a single channel of stitched vectors (whether organized as a single vector or a 2D surface, etc.). Many variations can be used.

[0128] The following text utilizes Figure 13 Let's explain the operation of GCN block 412. GCN block 412 Z1 or Z N The output Zn of each of the GCN blocks is also a three-dimensional vector. This three-dimensional vector has a first element consisting of m elements representing time samples t over a duration T, a second element representing the total number of sensors (I) in the sensor group being analyzed together, and a third element representing the feature channels with multiple eigenvalues ​​(or simply features). The third element forms a feature distribution or feature map. Therefore, the operation of GCN 404 can be referred to as feature extraction, and the feature map or distribution output from each GCN block 412 can be referred to as intermediate features, which are output at each GCN block 412. The features from the last GCN block N (412) can also be designated as Z. L However, it can also be called an intermediate feature (because it is output from a GCN block), or a final feature (or last feature) as the output from the last GCN block N 413.

[0129] During runtime, when it is desired to calculate cleaned or restored sensor data values ​​to replace faulty sensor data, the last intermediate feature Z can be used. L Provided to the recovery or denoising system or unit 206. In addition, during training, the last intermediate feature Z can be... L Both FS-ID-NN 214 and DA-NN 218 (via gradient inversion placeholder layer 216) are provided for training the neural network mentioned above. FS-ID-NN 214 is used to generate the ground truth loss, while DA-NN is used to generate the adaptive domain loss for each training epoch. Both losses are then used to modify the weights of the GCN. The following section utilizes... Figure 14 To explain the training.

[0130] refer to Figure 6-7 Figure 600 shows the time (x-axis) mapped to sensor characteristic values ​​(y-axis), where different lines represent different sensors. Relevant here, Figure 600 shows spatially dependent peak 602 and temporally dependent peak 604, each peak occurring at a single sensor, and each peak indicating faulty sensor data due to outlier levels of the indicated characteristic values. Figure 700 shows a persistent spatially dependent peak 704 for a single sensor, and a peak 702 exhibiting both spatial and temporal dependence due to the presence of identical peaks at two sensors.

[0131] refer to Figure 8 The space sensor data dependency graph illustrates the space input feature matrix 800, which has four defined nodes. The spatial input feature matrix 800 contains four sensor readings or values ​​(or inputs), where each edge 802 is a dependency. The spatial input feature matrix 800 can have all sensor readings at a single time point (or a single time frame) and from all sensors in the sensor group. The spatial dependency edges 802 are each designated as A herein. s The sensor value can be either the raw sensor reading as expected by the dependency unit 400 or a preprocessed sensor reading.

[0132] In one example implementation, the spatial dependence unit 406 calculates the spatial dependence as the correlation between sensor readings. Here is an example equation for the correlation:

[0133]

[0134] Where SA is the spatial dependency matrix, and Cov(X) f ,X g ) is the covariance calculation between two sensor readings, and this is determined between each available sensor pair in the sensor group. The variable δxf ,δx g Each is the standard deviation of each reading. Spatial dependence is then added to the dependence figure 1100 described below. Figure 11 In the GCN 404, the dependency graph 1100 is input as represented by the dependency in matrix form. Therefore, the spatial dependency unit 406 can also collect sensor data to generate the spatial dependency matrix SA.

[0135] refer to Figure 12 In one example implementation, the spatial dependency unit 406 generates the spatial dependency matrix SA1200 as a diagonal matrix, where sensors anc1 to anc4 each have rows and columns in the matrix, such that each matrix element represents an available sensor pair, and the spatial dependency matrix SA is filled with spatial dependency values ​​from equation (2). Grayscale indicates the level of the spatial dependency values.

[0136] refer to Figure 9 The time sensor data dependency plot shows the time input feature matrix 900, which has four defined nodes ( The input feature matrix 900 contains four sensor readings or values ​​(or inputs) for the sensor group. The nodes shown here are for two different sensors i = 1 and 3, but all sensors I analyzed for the sensor group will be part of the time-input feature matrix 900. Each sensor 1 or 3 here has four sensor readings or data values, each at a different time t from t-3 to t. Each edge 902 is a dependency or representation of dependency, and in this document, it is a direct (or adjacent) dependency when the sensor readings (or samples) are consecutive or adjacent for a single sensor. The input feature matrix 900 can have all sensor readings for all sensors I in the sensor group over the sampling duration T. The time-dependent edge 902 is designated as A in this document. t The sensor value can be either the raw sensor reading as expected by the dependency unit 400 or a preprocessed sensor reading.

[0137] In one example implementation, the time dependency matrix designated TA can also be a diagonal matrix, where the rows and columns represent different time points for samples within the same sampling duration T and for the same time interval t. Therefore, the rows cover the same sampling duration T as the columns, and each row represents the same time interval t as the interval t in the column. The time dependency matrix TA for a single sensor is shown below:

[0138]

[0139]

[0140] The sampling duration T ranges from t1 to t4. The time dependence values ​​are determined as follows.

[0141] For an example implementation, time dependency unit 408 can calculate time dependency based on Markov chain theory (or Markov properties), where newer or current data is emphasized more than older data to set the current dependency value. Here, sequence dependency in time series data is formulated by constructing a TA weight matrix using a Gaussian probability distribution, where closer readings have a higher temporal influence on each other. As an example, the following calculation is used:

[0142]

[0143] TA is also the time dependency matrix shown above, where h and j are two different time points (or sampling times), and the variable δ... 2 It is the uncertainty (or variance) in the time transition, and it is a fixed hyperparameter determined during training. Therefore, the time-dependent unit 408 can collect sensor data and generate the time-dependent matrix TA shown above.

[0144] refer to Figure 10 ST dependency diagram 1000 shows that a single sensor (or sensor data value or reading) 1002 can have multiple dependencies, including both spatial and direct time dependencies 1006. When the dependency is applied to different sensors at different sampling times, indirect or non-adjacent spatial dependencies 1008 or A may occur. ts There may also be indirect or non-adjacent time dependencies 1010 or A. tt It is a short time jump that skips one or more sampling times of the same sensor (also known as short-term time dependence).

[0145] refer to Figure 11 The ST dependency graph 1100 (or simply the dependency graph mentioned above) represents both spatial and temporal dependencies and can be considered as the input graph of GCN 404. Graph 1100 includes both nodes (numbered 0 to 39) as sensor data values ​​and edges as dependencies (both temporal and spatial). Therefore, a single sensor value on Graph 1100 can have multiple dependencies. The dependency graph 1100 can be specified as a formalization of GCN, where the graph construction (GCN) G = {X, A}, where X is each instance of the feature value (or sensor data) forming the graph node, and A is the dependency forming the graph edge.

[0146] refer to Figure 13An example implementation of the ST-GCN block 1304 (or simply GCN block 1304) of GCN 1300 is the same as or similar to GCN block 412 of GCN 404 or 112, and is in a sequence of GCN blocks from 1 to N. The intermediate GCN block n 1304 shown here receives intermediate feature Z from the previous GCN block n-11302. n-1 And provide intermediate feature Z to the next GCN block n+11306 n+1 .

[0147] As mentioned above, the example intermediate feature output Z from each GCN block 1304 n It can be a 3D vector, but it can be any other desired structure. Assuming, for the example continuing here, for a sensor group with a time window length of five (five intervals or samples) and four sensors, the intermediate feature output Z of all GCN blocks 1304... n It will be (5 (value 5) × 4 (value 4) × 64), including the last GCN block N (or L), thus providing the final output Z with 64 feature channels. N (or Z) L This forms the feature distribution or feature map described in this paper. The output of any GCN block can be in the form of a vector represented by a 3D matrix.

[0148] An example architecture for a GCN block that repeats for each GCN block including GCN block 1304 includes, in sequence: a spatial (enriched) feature embedding layer 1308, a first subtractor 1310, a first graph convolutional layer 1312, a first ReLU layer 1314, a temporal (enriched) feature embedding layer 1316, a second subtractor 1318, a second graph convolutional layer 1320, and a second ReLU layer 1322.

[0149] In operation, the spatial (enriched) feature embedding layer 1308 uses the following to embed a version of the spatial dependency (or a version of the spatial dependency matrix SA): (1) intermediate features from the output of the previous GCN block, or (2) a version of the sensor data values ​​used to form the spatial dependency (e.g., when the current GCN block is the first GCN block 1 (410)) and to form the first embedding result. More specifically, the spatial feature embedding layer 1308 generates a normalized adjacency matrix as follows:

[0150]

[0151] Where the space-normalized adjacency matrix And among them It is the spatial dependency matrix SA, and It is an angle-number matrix, where the matrix Each entry d ini It is the degree, which is the sum of the connection weights. The sum of the connection weights is the sum of the spatial dependency values ​​on all edges of a single sensor value (or node) in Figure 1100.

[0152] The next example operation of GCN block 1304 can then be represented as the equation:

[0153]

[0154] Equation (5) is a general equation, which makes Z in equation (5) n This is the intermediate feature output from the nth ST-GCN block (or, if the GCN block is the first GCN block 1, the input feature matrix Z1, also known as sensor data). It should be noted that in this specific example of GCN block n 1304, as... Figure 13 As shown, Z here n In reality, it will be Z n-1 Because the previous GCN block is n-1, and the output of GCN block n will be Z. n The spatial (enriched) feature embedding layer 1308 is based on equation (5) in the normalized spatial adjacency matrix. (or ) and intermediate feature output Z n Matrix multiplication is performed between them, and specifically, the intermediate feature output Z is... n It is a feature channel with a feature map or feature distribution, or a feature matrix (or sensor data) of the first or original input node. This results in a first embedding or enrichment result output from the spatial (enriched) feature embedding unit 1308.

[0155] The first subtractor 1310 then calculates (1) Z from the previous GCN block (or sensor data) used to form spatial dependence according to equation (5). n The first difference between the intermediate features and (2) the first embedding result.

[0156] The first or more graph convolutional layers 1312 then convolve the first difference. In this example, a single convolutional layer processes the first difference, but many different convolutional layer arrangements can be used, including a series of multiple convolutional layers. By way of an example, all convolutional layers in GCN block 1304 are linear and receive and output 64 features except for the first convolutional layer of the first GCN block 1, which only receives two features from the original input feature vector (Equation (1) above) but still outputs 64 features. Each convolutional layer also uses a bias. It will be understood that other architectures can be used when there are more than two input features.

[0157] Next, the first Corrected Linear Unit (ReLU) layer 1314 transforms the results from (one or more) first convolutional layers into positive numbers and generates a value specified as... The first ReLU result completes equation (5) above.

[0158] The temporal (enriched) feature embedding layer 1316 utilizes the first ReLU result by first applying the same normalized adjacency matrix equation (4) as described above. The time dependencies of the time dependency matrix TA are embedded to form a second embedding result, except that here... It is the time dependency matrix TA. Therefore, equation (4) now establishes the time-normalized adjacency matrix. (or Subsequently, the temporal (enriched) feature embedding layer 1316 generates the second embedding result by applying the following equation (6) to the normalized adjacency matrix. (or ) and the first ReLU result Perform matrix multiplication between them:

[0159]

[0160] Z n This is an intermediate feature, which is the output of GCN block n 1304. The second subtractor 1318 then calculates the first ReLU result. The second difference between the second embedding result and the second embedding result.

[0161] The second or one graph convolutional layer 1320 then convolves the second difference. The graph convolutional layer 1320 is as described above using graph convolutional layer 1312.

[0162] The second ReLU layer 1322 processes the results of (one or more) second convolutional layers and outputs intermediate features Zn, which fulfills equation (6) and is then fed as input to the next GCN block n+11306. The intermediate feature output Zn of the last GCN block N413 is... N or Z L It has characteristic channels or characteristic distributions indicating fault sensors and source fault sensors, and can be provided to a recovery system or network as described below using example recovery unit 206, which is used in the following description. Figures 16A-16B And described in detail.

[0163] Now for reference Figure 14The training system 1400 includes both a fault sensor data recognition (or classifier) ​​neural network (FS-ID-NN) 1408 (or 116) operated by the training fault sensor recognition unit 214 and a domain adaptive neural network DA-NN 1418 (or 118) operated by the domain classifier unit 218. In this example, input sensor data X 1402 is provided to a series of GCN blocks 1 to N (1404 to 1406) of GCN 1300 (or 404 or 112), wherein, as an example, the last GCN block 1406 provides both the FS-ID-NN 1408 and DA-NN 1418 with the last intermediate feature (or feature distribution or feature map) Z with 64 features. N .

[0164] As mentioned above, FS-ID-NN 1408 operates on the labeled data input into GCN 1300 for supervised learning to generate the final intermediate feature (or feature map or distribution) Z. N The last intermediate feature Z N This data is then fed into the FS-ID-NN 1408 for training the GCN 1300. Depending on the sensor set used, the datasets used for both the FS-ID-NN 1408 and DA-NN 1418 in this paper can be derived from known or publicly available datasets, or they can be custom datasets. In one form, the label ID unit or control 1430 can monitor the GCN output Z. N The transmission includes automatic identification of whether the input data is labeled. If the input training data is unlabeled, the label ID control 1430 guides the GCN output Z. N It is only provided to DA-NN 1418 and not FS-ID-NN 1408. This can also be done manually, so that when semi-supervised input is loaded into GCN 1300 for training, FS-ID-NN can be simply turned off or transmission to FS-ID-NN can be manually blocked.

[0165] In one example implementation, the FS-ID-NN 1408 may have a multilayer perceptron (MLP) 1410 that receives the GCN output Z. N The two probabilistic features are then output to a Softmax layer 1412, which in turn outputs a corrected probability, which is either the anomaly classification y or the probability used to calculate the anomaly classification y. The MLP 1410 can have multiple fully connected layers, receiving the last intermediate feature Z from the last GCN block N 412. NThe MLP 1410 outputs a 3D vector, such that in this example, the input to the MLP 1410 is 64 features from the feature channels. The MLP 1410 still outputs a 3D vector, where the first two dimensions remain the same as the counts of the number of time intervals (or samples) and the counts of the sensors in the sensor group. However, the third feature channel outputs two features, one representing the probability of faulty sensor data (and faulty sensors), and the other representing good sensor data (or good sensors), repeated so that each sensor in the sensor group has its own 3D vector output from the MLP 1410. Continuing with the example above, the MLP 1410 can output a 5×4×2 3D vector, and this 3D vector can be aggregated into multiple vectors to form a 2D matrix, where each row or column has two features for different sensors.

[0166] The Softmax layer 1412 does not change the dimension of the vector from MLP 1410, specifically the "out_feature" dimension. Instead, the Softmax layer 1412 transforms the two output features into probability distributions with defined categories. As an example, this is achieved by making the sum of each row of the vector equal to 1 (e.g., changing the feature values ​​to [0.99, 0.01]), where [1, 0] represents good sensor data (and in the case of RTLS sensors, successful LOS), and [0, 1] represents faulty sensor data (and in the case of RTLS sensors, the resulting NLOS scenario). These two features can be provided as two bits (01 or 10) and, when desired, are transformed into a single category y1414. The output of FS-ID-NN1408 can still be in the form of a three-dimensional vector (m × I × 2).

[0167] At the ground truth (GT) loss unit 1414, both the two-feature input probabilities leading to the Softmax layer 1412 and the binary feature output of the Softmax layer 1412 can be used with known or other loss functions (as an example, such as classification cross-entropy) to generate the backpropagation ground truth loss L. c and gradient gradient It has θ as a weight parameter and is related to the MLP-Softmax output. This gradient can then be transformed into the gradient with respect to the weights of GCN 404 (or 1300). The weights of GCN 404 are modified. As shown, this is used for backpropagation 1426. The output of the Softmax layer 1412 can also be provided as the anomaly classification y.

[0168] For adaptive domain training, DA-NN 1418 is used to train the GCN backbone NN 1300 to produce consistent and similar feature distributions regardless of different scenarios (or domains), and to find hidden “domain-independent” correlations between sensor data in a group of sensors by training the GCN 1300, and by using DA-NN 1418 (or a domain classification network) with dynamically, adaptively increasing λ (lambda).

[0169] Specifically, here, the gradient inversion unit or layer 1416 first receives the last intermediate feature (or feature map or distribution) Z. N And it can be viewed as simply forwarding the input vector, which includes 64 feature channels and a three-dimensional vector (m×I×64), to the DA-NN 1418 without modification. The gradient reversal layer 1416 is shown as a placeholder in the forward propagation of the neural network framework diagram, but it will be applied during the backpropagation 1428 and explained below.

[0170] The DA-NN 1418 can have an MLP 1420 and a Softmax layer 1422 to accept both labeled and unlabeled training data. Unlabeled data includes environmental or domain variations (or cross-domain variations) of the same group of sensors providing sensor data. Several examples beyond any of the domains mentioned above are listed: such domains for RTLS localization of sensors in a car could include outdoor, indoor, garage, and parking lot domains. Although the domains are both unlabeled and labeled for semi-supervised training, the system can still track which domain is processed because the domain used as input for training is likely known. Domains are provided as source and target domains, on which the DA-NN attempts to classify the domain as either the target or source domain, and generates a domain classification d indicating the classification result. The source domain is the initial or original first domain on which the DA-NN was originally trained, while the target or cross-domain domain is the domain to which the DA-NN attempts to generalize.

[0171] Specifically, the MLP 1420 layer receives the intermediate feature Z. N The system has 64 feature channels and outputs two feature vectors for each input domain, where one feature is an indication of the probability of the source domain and the other is an indication of the probability of the target domain. As mentioned above regarding the use of Softmax layer 1412, Softmax layer 1422 transforms the features into probabilities, such that the two features are provided as [1,0] for the source domain and [0,1] for the target domain. The output from Softmax layer 1422 can be a three-dimensional vector such as (m×I×2) (as a matrix when desired).

[0172] DA loss unit 1424 then generates domain adaptive loss L by using classification d. D(For unlabeled or labeled semi-supervised training). Domain adaptive loss L D The fact that GCN 1300 is domain-invariant means that it not only depends on specific features of the source domain but also generalizes well to the target domain. This allows GCN 1300 to better transfer knowledge from one domain (source) to another (target), even if their data distributions are different. The loss L can be calculated using the example binary cross-entropy equation (which takes into account both source and target domain probabilities). D However, other equations can also be used.

[0173] Once the loss L is calculated D The domain-adaptive training parameters (or weight factors) λ (or simply parameters λ) are then applied to the loss L. D The parameter λ is used for backpropagation 1128, thus controlling the strength of the domain loss. Therefore, the parameter λ is a hyperparameter that gradually increases during training. Thus, in the early stages of training, the parameter λ should be small or close to negative, causing the GCN to focus more on fitting the source domain. At this point, the domain adaptive loss is weak. Around the middle of training, and as the parameter λ increases, the domain adaptive loss becomes stronger, and the GCN focuses more on aligning the source and target domains, helping it become more domain-invariant. In the later stages of training, the parameter λ becomes larger and positive, allowing the GCN 1300 to learn most of the source domain knowledge and now focusing on minimizing the domain offset, ensuring that the GCN 1300 generalizes well to the target domain.

[0174] As explained below, the parameter λ can be adjusted during training. Assuming that the source and target domain inputs share the same dependency graph structure and differ only in node values ​​(input features), the adaptive strength should be dynamically adjusted based on the degree of domain shift.

[0175] The parameter λ should be related to both the key domain offset metric and the training progress, where a stronger domain offset requires a higher contribution from the domain loss to ensure effective adaptation. The parameter λ can be calculated as follows:

[0176]

[0177] Where p is the training iteration determined by the following formula:

[0178]

[0179] The total step is the total number of steps (or iterations) in the training progress. The variable 's' is a scaling factor that adjusts the overall strength of the domain adaptive loss and is related to the domain offset (how much difference exists between the source and target domains) and the degree of domain difference. When the domain offset is large, a stronger adaptive loss helps GCN 1300 learn how to better adapt to the target domain. When the domain offset is small, a reduced adaptive strength avoids overfitting to the target domain. Therefore, the scaling factor 's' here is the estimated maximum mean difference (MMD), which is a quantification of the distributional difference between the source and target domains calculated as follows:

[0180]

[0181] Where k is the kernel function (e.g., RBF kernel), It is the expected value (such as a weighted average or other combined calculation), X s It is the source domain sensor value, and X t This corresponds to the sensor value in the target domain. A higher MMD score indicates a larger distribution shift.

[0182] refer to Figure 15A-15G A graph is provided to illustrate the domain shift and feature distribution convergence due to training. The MMD domain shift indicator graph is shown as a histogram 1500 ( Figure 15A Histogram 150 shows the MMD values ​​along the x-axis, which are normalized values ​​of the sensor data feature values ​​mapped to the y-axis of the source domain. MMD Histogram 1502 ( Figure 15B The figure shows the MMD values ​​of the normalized sensor data features mapped to the target domain, indicating significant differences between the two domains.

[0183] Figure 1504 is used for the source domain and the target domain respectively. Figure 15C ) and Figure 1506 ( Figure 15D The diagram illustrates the input distribution, where asterisks indicate good sensor data (and good sensors, also known as LOS data when the domain is used for RTLS localization), while × indicates faulty sensor data (and faulty sensors, also known as NLOS when the domain is used for RTLS localization). The y-axis indicates one characteristic (such as AOA for RTLS localization), while the x-axis provides another characteristic (such as TOA for RTLS localization). Figures 1504 and 1506 illustrate the significant differences between the source domain input distribution and the target domain input distribution.

[0184] Figure 1508 is used for the source domain and the target domain respectively. Figure 15E ) and Figure 1510 ( Figure 15FFigures 1508 and 1510 show the feature distribution generated as an intermediate feature map (from the intermediate GCN block before the last GCN block N 413), where asterisks indicate good sensor data (and good sensors also referred to as LOS data when the domain is used for RTLS localization), while × indicates faulty sensor data (and faulty sensors also referred to as NLOS when the domain is used for RTLS localization). The y-axis indicates the feature values, while the x-axis is a unitless scale used to map the feature distribution. Figures 1508 and 1510 show that, due to the processing at the GCN, the feature distributions are now much closer to each other.

[0185] Figure 1512 illustrates that the feature distributions as the final output from the GCN are now strikingly similar, with the trained (or source) domain exhibiting a feature distribution of good (or LOS) sensor data as a star shape and bad (or NLOS) sensor data as an × shape, and the test (or target) domain exhibiting a feature distribution of good (or LOS) sensor data as a triangle shape and bad (or NLOS) sensor data as a plus sign shape. The feature distributions now largely overlap and are not significantly distinguishable, thus indicating that the feature distributions largely represent domain-independent features for fault sensor identification.

[0186] Refer again Figure 14 And return to the domain loss L D For backpropagation, the DA loss unit 1424 (or other units) can generate gradients for backpropagation 1428 and will be applied to GCN 1300, as with the GT loss unit 1414 described above, but here the gradients are specified as follows:

[0187]

[0188] Subsequently, the gradient is provided to the gradient reversal layer 1416 to modify the parameter λ, specifically by reversing the sign of the gradient (or parameter λ) to make it negative or positive from its original value. The parameter λ can also be viewed as controlling the strength of the gradient reversal. As explained above using the ground-value loss, the resulting negative domain loss gradient is also adjusted based on the eigenvalues ​​or probability values ​​of MLP 1420, and the result is the following gradient:

[0189]

[0190] By applying negative gradients to GCN block layers, the negative gradients force the GCN to compete with each other by reversing the gradient direction. This introduces an adversarial signal, which forces one part of the network to become more invariant or resilient to certain features (e.g., domain-specific properties or biases). Thus, GCN 1300 learns features invariant to domain-specific differences.

[0191] Subsequently, as an example, the domain adaptive loss LD can be followed by the truth loss L. c The total loss L is combined to determine the weights that will be applied to the GCN 1300. In some examples, the total loss L can be applied to the weights of the GCN based on either the loss itself (12a) or the gradient (12b):

[0192] L = L c +λL d (12a)

[0193]

[0194] Alternatively, the loss L c and L D It can be combined with other techniques (such as averaging or other combinations) to apply a single gradient weight factor to the weights of GCN 1300.

[0195] The following is example pseudocode illustrating layer specifications that can be used throughout the entire network during training:

[0196]

[0197]

[0198] The following is sample Python code for PyTorch pseudocode that is limited to the GCN architecture:

[0199]

[0200]

[0201] Now for reference Figures 16A-16B Example fault sensor data denoising or recovery system 1600 (or unit 206) has a multivariate embedding segment 1602 and a position encoding segment or unit 1604. Segments 1602 and 1604 may be referred to together as the input encoding branch. The output of the position encoding segment 1604 is provided to a multi-attention signal recovery neural network (or simply recovery NN or NN branch) 1630, and the multi-attention signal recovery neural network 1630 has a spatial encoder 1632 and a temporal encoder 1634.

[0202] Embedding segment 1602 may include some overlap with GCN operations (including acquiring sensor readings), except that this is shown here as receiving sensor readings from sensor reading database 1606 or alternatively directly from the sensor or sensor analysis unit. ST-GCN anomaly detection unit 1614 operates as described above with GCN or other fault sensor data detection algorithms to identify fault sensor data “a,” which may be in the form of a feature distribution indicating that the corresponding raw sensor data (such as the two input features AOA and TOA used for RTLS localization) has at least one fault sensor value. Separately, raw input processing unit 1608 may preprocess the raw sensor data to a desired format to generate sensor data or value “x.” Quantization unit 1610 then scales the raw sensor value x such that the sensor data input and the anomaly feature (or feature distribution) will have a data bit size that can be placed together in a single word after separate embedding.

[0203] Therefore, the characteristic distribution a and the quantized sensor data x (specified as v) x The input embedding unit 1612 and the anomaly feature embedding unit 1616 are respectively provided to the input embedding unit 1612 and the anomaly feature embedding unit 1616. Both the input embedding unit 1612 and the anomaly feature embedding unit 1616 perform operations on the feature distribution a and the quantized sensor data x (or v). x The individual embedding of ) . The embedding here can be similar to encoding, and a linear transformation (which can be performed on the anomalous feature a via matrix multiplication) is performed to change the dimension of the anomalous feature a. In sensor data v x In the current example, sensor data embedding is based on quantized sensor data v x The value range is used instead of matrix multiplication. The resulting embedding of the anomalous features from the GCN is specified as ψ(x), and the embedding of the sensor data is specified as ψ(a).

[0204] Subsequently, in one implementation, positional encoding can be performed by positional encoding unit 1604 because the transformer NN 1630 itself cannot track the input or lexical order. Positional encoding adds positional information to the data, allowing the transformer NN 1630 to track the word sequence embedded in the data. In this example, adder 1618 combines the anomaly and sensor data embeddings ψ(x) and ψ(a) to form a multivariate embedded word e. p As an example, embeddings are combined by concatenation to form vectors (although other structures can be used alternatively).

[0205] For a word (or lexical unit) at position p in a word sequence, the positional encoding function PE is calculated using sine and cosine functions. p :

[0206]

[0207] Where L is the scaling factor, K is the total dimension of the embedding space, and i is the dimension index (or coordinate) of the embedding vector. As some examples, the transformer embedding space can have a fixed size of 512, 768, or 1024 dimensions.

[0208] Therefore, the final input embedding is constrained to: x p It can be determined as:

[0209] x p =e p +PE p (14)

[0210] In this example, each x from the position encoding p It is a three-dimensional vector that includes the positional information of words in a word sequence. This vector can take the form of: one dimension for spatial location (e.g., sensor ID), one dimension for temporal location (e.g., time step), and an additional dimension for multiple input features (which include raw sensor readings (e.g., ToA, AoA) and all features provided by the GCN feature embedding backbone). PE values ​​describe the absolute position of the word and implicitly encode the relative distance between words.

[0211] Turning now to the transformer NN 1630, the three-dimensional vector is provided to three different places in the transformer NN 1630, including inputs to the following: spatial encoder 1632, splicing unit or layer 1648 (which also receives output from spatial encoder 1632), and ST vector unit or layer 1662 (which also receives output from time encoder 1634), each of which is described in turn below.

[0212] The spatial encoder 1632 has three linear layers 1636, 1638, and 1640, each with a learnable weight matrix. And the superscript 's' represents space. p The input is provided to each linear layer 1636, 1638, and 1640. The output of linear layers 1636, 1638, and 1640 is a query Q that matches the name of the weight matrix. S Key K S Sum V S The query (Q) refers to the current term, the key (K) refers to the other terms being compared, and the value (V) refers to the aggregated content (sensor values ​​and anomaly features). The spatial encoder 1632 also features a scaled dot product layer 1642, which determines the Q from linear layers 1636 and 1638 using dot products. S and K S The similarity is calculated, and the scaled dot product attention score S is output.S .

[0213] Both the spatial encoder 1632 and the temporal encoder 1634 have a hierarchical multi-head attention structure. Therefore, S is obtained in the spatial encoder 1632. S and V S Subsequently, the matrix multiplication (MatMul) and Softmax layer 1644 perform spatial head equations over the spatial domain. S To focus on the spatial relationships between sensors, as follows:

[0214]

[0215] Where * represents matrix multiplication. Head equation S (15) Captures relationships such as spatial dependencies between different parts of the input sequence, and the Softmax operation provides the probability of those relationships.

[0216] head S The output is then fed to MLP layer 1646 to assist in capturing complex patterns by providing nonlinear transformations and feature mixing, including dimensional expansion. The output of MLP layer 1646 is also the output of spatial encoder 1632, and is then fed to stitching unit or layer 1648, which also directly receives x. p enter.

[0217] 1648 splicing units, each x p To form matrix X p' And it continuously stitches together the received data to form a 2D surface, thereby continuously expanding X. p' Dimensions.

[0218] Regarding the time encoder 1634, its structure is basically the same as or similar to that of the space encoder. Here, three linear layers 1650, 1652, and 1654 receive the matrix X. p' In addition, operations involving Q, K, and V (including head) T The equation is the same, except that the name refers to T in terms of time rather than S in terms of space, and does not need to be described again. The output of the time encoder 1634 is provided to the spacetime (ST) vector unit or layer 1662, which also directly receives x. p Input vector to generate x from time encoder 1634 p The k-dimensional feature vector formed by concatenating the input and output.

[0219] Finally, as an example implementation, the MLP layer 1664 can have two feedforward layers that take a κ-dimensional feature vector as input and transform the feature vector into a one-dimensional output. One-dimensional output This represents the recovered or cleaned (or denoised) sensor readings as the final output.

[0220] refer to Figure 17 An example flow 1700 for detecting faulty sensor data and recovering sensor data according to at least one embodiment described herein is shown. Flow 1700 has generally uniformly numbered operations 1702 to 1722. To explain flow 1700, reference can be made to the description in the document. Figure 1-16B Any device, system, and neural network disclosed herein.

[0221] Process 1700 may include "grouping contributing sensors for a common objective" 1702, where the common objective is as defined above, wherein the sensors may contribute directly or indirectly to determining scores, detection objects, etc., and the identified sensors form a sensor group for fault sensor detection as described herein. Once the sensors are identified, process 1700 may include "receiving sensor data" 1704, where sensor data from the sensor group is collected for fault sensor analysis. This also includes formatting the sensor data into a time-relative sequence data array of the sensors, which is then converted into an input feature vector.

[0222] Process 1700 may include “generating sensor data dependencies” 1706, where, as described above, spatial and temporal dependencies are determined as the correlation between sensor readings or data (spatial dependencies are determined using covariance equation (2), and temporal dependencies are determined using Markov chain theory (or Gaussian probability distribution) in equation (3). The dependencies are formed as spatial and temporal matrices.

[0223] Process 1700 may include “generating GCN backbone feature output” 1708, and operation 1708 may include “traversing N GCN blocks” 1710. The GCN blocks have the same structure as described above. Operation 1708 may include “inputting spatial and temporal dependencies into separate layers” 1712, where the spatial matrix is ​​input into a spatial enrichment feature embedding layer, which receives both the spatial dependency matrix and the output or input sensor data from the previous neighboring GCN block. A normalized adjacency matrix is ​​generated for the spatial dependency matrix (Equation (4)), and then multiplied by whichever of the following two things exists for the GCN block: the output or input sensor data from the previous neighboring GCN block (Equation (5)), and then the product is differiated from the output or input sensor data from the previous neighboring GCN block. The result is input into one or more convolutional layers, and ReLU is applied to the convolution result. As explained above using GCN block n ( Figure 13This process is repeated for the temporal enrichment feature embedding layer of the received temporal dependency matrix to multiply the normalized temporal adjacency matrix with the output spatial result.

[0224] Process 1700 may include "Identifying Faulty Sensors" 1714, in which the GCN block operation is repeated for each GCN block, with each GCN block outputting intermediate features, and the output of the last GCN block providing feature channels with a feature map or distribution; for example, this output is 64 feature values. The feature distribution or feature map indicates whether the sensor has faulty sensor data. A classifier system or end application can receive the feature map and determine whether the sensor has faulty sensor data.

[0225] In addition, process 1700 may include “performing sensor data recovery” 1716, which may include “combining individually embedded sensor input data with embedded sensor anomalous feature outputs to form combined data” 1718, wherein linear embedding is applied to anomalous data indicating faulty sensor data, while the sensor data is quantized and then embedded over a range of quantized data values. The embeddings are then combined by concatenation.

[0226] Operation 1716 may include “performing position encoding” 1720, which is then performed to transform the combined embedding into input for a multi-attention transformer, and through an example, where the input is in the form of a three-dimensional vector that includes the spatial location of the sensor, the temporal location in time (or time step), and one or more feature values. Term positions in the vector can be implicitly encoded into the vector to indicate the relative positions between terms.

[0227] Operation 1716 may include “performing spatial and temporal encoding on a multi-attention transformer-based signal denoiser” 1722. Specifically, a vector or set of vectors as 2D input can be provided to a multi-attention transformer neural network having a spatial encoder and a temporal encoder. The input vector is placed at the input of the spatial encoder, at the concatenation unit at the output of the spatial encoder, and as input to the concatenation to the temporal encoder, and finally placed into the vector unit before the MLP generates clean (or recovered or replaced) sensor data for faulty sensor data.

[0228] refer to Figure 18 An example flow 1800 of a neural network in a training flow 1700 according to at least one embodiment described herein is shown. Flow 1800 has operations 1802 to 1810 that are generally evenly numbered. To explain flow 1800, reference can be made to the descriptions herein. Figure 1-16B Any device, system, diagram, graph, chart, and neural network disclosed herein.

[0229] Process 1800 may include “classifying the domain of sensor data using a domain NN” 1802, wherein a domain adaptive neural network (DA-NN) determines whether the domain is a source domain or a target domain. Using both labeled and unlabeled training data, the DA-NN uses MLP layers and Softmax layers to output the domain classification to perform semi-supervised learning. The domain classification is then used to compute the domain loss, and the domain loss gradient is used to compute the domain loss gradient (equation (10) above).

[0230] Procedure 1800 may include “adjusting the domain loss gradient using dynamic training parameters to adjust the weights of the GCN” 1804. This refers to calculating the domain adaptive training parameters (or weight factors) λ according to equation (7) above, which takes into account both training progress and domain offset (or domain difference).

[0231] Adjusting the domain loss gradient may also include “performing gradient reversal”1806, where the parameter λ is made negative (or its sign is changed) by the gradient reversal layer described above to control the strength of the domain loss, and thus control the domain loss gradient.

[0232] Procedure 1800 may include “adjusting the weights of the GCN by using a fault sensor ID neural network to determine the ground truth loss” 1808. Here, the FS-ID-NN performs supervised learning using a labeled training dataset to determine the fault classification and fault-free classification used to form the ground truth loss.

[0233] Procedure 1800 may include “adjusting GCN weights by using both domain loss and ground truth loss” 1810. Here, both the ground truth gradient and the domain loss gradient are used for backpropagation and for applying to the weights of the GCN layer. This can be done first by combining the loss using equation (12a), or similarly by combining the ground truth gradient and the domain loss gradient before applying them to the weights of the GCN layer (equation 12b).

[0234] The results of the disclosed system and method provide significant improvements in fault sensor detection with multiple domains. A test run was performed using an RTLS sensor for positioning on a vehicle (with...). Figure 3 (Similarly), and fault sensor detection is referred to as NLOS detection. In supervised learning for FS-ID-NN, labeled data is used to test the training domain or source domain, and in semi-supervised learning for DA-NN, both labeled and unlabeled data are used to test across the domain or target domain.

[0235] The training domain yielded the following results: accuracy: 94.75%, precision: 94.39%, recall: 99.38%, and F1 score (or the harmonic mean of precision and recall): 96.83%. The cross-domain results were: accuracy: 88.63%, precision: 91.55%, recall: 94.30%, and F1 score: 92.90%.

[0236] In this document, relational terms such as "first" and "second" may be used merely to distinguish one entity or action from another, without necessarily requiring or implying any actual such relationship or order between these entities or actions. Numerical ordinal numbers such as "first," "second," "third," etc., simply represent distinct singularities among a plurality and do not imply any order or sequence (unless specifically defined by the language of the claims). The textual order in any claim does not imply that the process steps must be performed in a chronological or logical order according to such order (unless specifically defined by the language of the claims). Process steps may be interchanged in any order without departing from the scope of the invention (provided that such interchange does not contradict the language of the claims and is not logically meaningless).

[0237] Furthermore, depending on the context, unless otherwise mentioned, the use of terms such as “connected” or “coupled to” when describing the relationship between different elements or components of a nozzle does not imply a direct physical connection between these elements. For example, two elements may be physically, electronically, logically, or in any other way connected to each other by one or more additional elements.

[0238] Although at least one exemplary embodiment has been presented in the foregoing detailed description, it should be understood that numerous variations exist. It should also be understood that the exemplary embodiments are not intended to limit the scope, applicability, or configuration of this disclosure in any way. Rather, the foregoing detailed description will provide those skilled in the art with a convenient roadmap for implementing the exemplary embodiments. It should be understood that various changes can be made to the function and arrangement of the elements without departing from the scope of this disclosure as set forth in the appended claims and their legal equivalents.

Claims

1. A method comprising: Sensor data is received from a sensor group, including a real-time positioning sensor for performing object positioning, via at least one processor. Using at least one processor, the temporal dependence of sensor data from the same sensor in the sensor group and the spatial dependence between sensor data from different sensors in the sensor group are repeatedly determined for multiple sensors. The processor generates an indicator indicating whether sensor data associated with each sensor in the sensor group is faulty sensor data, including inputting the spatial dependency and the temporal dependency into a convolutional neural network (CNN). The process of recovering the faulty sensor data into non-faulty sensor data includes combining the version of the indicator and the version of the sensor data associated with the indicator to generate combined fault data; and inputting the combined fault data into a multi-attention transformer neural network. as well as The positioning is performed using the non-faulty sensor data to locate the object.

2. The method of claim 1, wherein the non-faulty sensor data is a distance reading, and wherein the positioning includes: Trilateration is performed using the localization from the real-time positioning sensor and the distance readings by applying a least-squares function or a density-based spatial clustering (DBSCAN) function with noise.

3. The method according to claim 1, comprising: The spatial dependencies are formatted into a spatial diagonal matrix, wherein the spatial dependency values ​​in the spatial diagonal matrix indicate the dependencies between each available sensor pair in the sensor group; And input the spatial diagonal matrix into the CNN.

4. The method of claim 1, further comprising formatting the time dependencies into a time diagonal matrix, wherein the time dependency values ​​in the time diagonal matrix illustrate the dependency between each available pair of different sample time points of a single sensor in the sensor group, and repeating this process for multiple sensors, and inputting the time diagonal matrix into the CNN.

5. The method of claim 1, wherein the CNN comprises a sequence of multiple CNN blocks to define a plurality of iterations and output intermediate features at each CNN block, the plurality of CNN blocks being repeating neural network blocks having the same neural network structure, wherein the output of each block is an intermediate feature and each output is a three-dimensional vector, the three-dimensional vector comprising a first channel for a plurality of samples acquired over time for the sensor group, a second channel for a plurality of sensors in the sensor group, and a third channel, the third channel being a feature channel having an indicator in the form of a plurality of feature values, the plurality of feature values ​​cooperatively forming a feature map and indicating a feature distribution, wherein the CNN is trained such that different output feature distributions from the CNN indicate fault data from different sensors.

6. The method of claim 1, further comprising training the CNN, which includes inputting the indicator output from the CNN into both a fault sensor data classifier neural network and a domain adaptive neural network (DA-NN), wherein the fault sensor data classifier neural network operates using only labeled training data, and the DA-NN operates using both labeled and unlabeled training data.

7. The method of claim 6, wherein the training comprises: The DA-NN generates dynamic weighting factors for the domain-adaptive loss used to modify the weights of the CNN, wherein the dynamic weighting factors depend on the distribution difference, which is the difference between the source domain and the target domain.

8. The method of claim 6, wherein the amount of distribution difference is determined by using the maximum mean difference (MMD).

9. The method of claim 6, wherein the amount of distribution difference is determined by using a radial basis function (RBF) kernel.

10. The method of claim 6, comprising: The weights of the CNN are modified using both the ground value loss from the fault sensor data classifier neural network and the domain adaptive loss from the DA-NN.

Images