Gesture category prediction and interpretable artificial intelligence

Abstract

The invention relates to gesture category prediction and interpretable artificial intelligence. Techniques are disclosed for inferring gesture category predictions at an edge deployment device (131, 132, 133, 134). Gesture category prediction is based on measurement data from a radar sensor. The user output may be provided based on feature relevance vectors determined using interpretable artificial intelligence. The invention discloses retraining of edge deployments. Joint learning techniques at a server (139) are disclosed.

Description

Technical Field

[0001] Various examples of the present disclosure generally relate to explainable artificial intelligence (XAI) techniques. Various examples of the present disclosure specifically relate to inferring a gesture category prediction based on a feature vector and providing a user output to a user based on a feature relevance vector associated with the feature vector, wherein the feature relevance vector is determined based on the XAI technique. Background Art

[0002] Traditional methods for determining gesture class predictions based on measurement data from sensors such as radar sensors are based on pre-trained machine learning (ML) models such as neural networks.

[0003] Inference can be performed at an edge-deployed processor. “Edge-deployed” means that the processor is not a central server, but is co-located with sensors that acquire measurement data in the field. For example, an edge-deployed processor and an edge-deployed radar sensor can be coupled to a vehicle’s common local computing network for automotive use cases.

[0004] According to the reference implementation, during the development phase, a large measurement campaign is conducted to build a training dataset that densely samples the entire input space expected in the field (i.e., has good coverage under various scenarios and users). During the measurement campaign, the feature vectors of the training dataset are acquired using radar sensors of the same type as those that will be deployed later in the field.

[0005] To enable training, feature vectors require ground-truth labels. There are various options for determining the ground-truth labels for feature vectors in the training dataset. For example, manual annotation by an expert is one option. Alternatively or additionally, alternative sensors such as cameras can be used to collect images based on which ground-truth labels can be reliably determined. This is called supervised learning. Using these labeled training data, a neural network can be trained and then deployed to an edge deployment processor for inference.

[0006] Despite initial efforts for accurate data collection and labeling during central training, performance in the field often degrades due to the mismatch between the scenarios considered in training and actual deployments. Summary of the invention

[0007] Therefore, there is a need for advanced techniques that can use ML models for inferring gesture class predictions based on feature vectors determined based on measurement data from sensors such as radar sensors or other depth sensors. There is a need for techniques that can mitigate at least some of the above limitations and disadvantages. There is a need for techniques that can achieve accurate and reliable gesture class predictions. There is a need for techniques that can achieve edge-deployed inference.

[0008] This need is met by the features of the independent claim. Features of the dependent claims define exemplary embodiments.

[0009] In the following, techniques based on ML inference are disclosed that enable gesture category prediction. Edge-deployed inference is disclosed. This means that an edge-deployed processor collocated with a sensor that acquires measurement data is used for inference. Data indicating the measurement data does not need to be transmitted via a wide area network (e.g., the Internet) to perform the inference. Communication of such data does not require a cellular network. An embedded computing circuit device can be used to implement the edge-deployed processor. In particular, there is no need to use a central server for inference.

[0010] The ML model is used to infer gesture category predictions. A technique is disclosed that enables improving the ML model in edge-deployed retraining. In other words, the edge-deployed processor can perform retraining of the ML model based on a locally acquired training data set. The disclosed technique is able to maintain the privacy of sensitive user data. This is because the measurement data is not provided to a central server.

[0011] According to various examples, a user output is determined based on a feature relevance vector. A feature relevance vector is determined for a corresponding feature vector. The feature relevance vector includes a plurality of dimensions. The number of dimensions of the feature vector is equal to the number of dimensions of the feature vector. For each dimension, the feature relevance vector includes a corresponding feature relevance value. Each feature relevance value indicates the importance of one or more feature values ​​of the corresponding dimension to the prediction of the ML model. For example, there may be a one-to-one correspondence between the feature relevance vector and the feature vector; that is, for each feature vector, there may be a corresponding feature relevance vector. Here, the feature relevance value of the feature relevance vector indicates the dependence of the prediction on a single corresponding feature value of the feature vector. In such a one-to-one relationship, the feature relevance vector is sometimes labeled as "local". However, in some cases, a feature relevance vector may be associated with multiple feature vectors (then, the feature relevance vector may be referred to as a global feature relevance vector). Here, each feature relevance value indicates the (aggregate or overall) dependence of the prediction on a set of feature values ​​of multiple related feature vectors (all feature values ​​have the same dimension, the same dimension as the feature relevance value).

[0012] More generally, a feature relevance vector is determined accordingly using the XAI technique. For example, the feature relevance vector may include Shapley values, such as local or global Shapley values. The feature relevance vector may also be referred to as an "XAI vector".

[0013] Based on the user output associated with the feature relevance vector, a variety of effects can be achieved. First, continuous guidance can be provided to the user to provide feedback to the user in response to the user performing a certain gesture. The feedback can highlight which attributes of the gesture (associated with different dimensions / features of the feature vector) have a significant impact on the gesture category prediction. Thus, the user can learn to perform the gesture appropriately and interact with the system; that is, the guided and interactive user training process can be promoted by the user output. Secondly, the training data set can also be filled. Based on the user output, the user can make an informed decision on whether the feature vector associated with the gesture is to be included in the training data set. Thus, a carefully planned training data set can be obtained, which can help retrain the ML model to provide more accurate results and specifically adapt to the deployment scenario. That is, user output can promote the retraining process of the ML model. This is particularly useful for use cases where sensors are used for gesture category prediction of gestures performed by non-expert users (e.g., hands-free car infotainment or navigation control, video or audio applications, game applications). This makes it possible to fill user-specific training data sets. Such a method can enable disabled users to compensate for any limitations to perform gestures captured in a certain way in a predefined training data set.

[0014] As can be understood from the above, a technology is disclosed that can implement edge deployment retraining of a machine learning model based on a training data set, wherein the training data set is populated based on feature vectors obtained in an edge deployment scenario using edge deployment sensors. According to various examples, when retraining the machine learning model, an uplink message indicating the weights of the machine learning model can be provided to a central server. The central server can collect corresponding uplink messages from several corresponding edge deployment processors (clients). The central server can then perform a federated learning technique to merge the weight sets obtained from several clients.

[0015] Federated learning can be used to merge such weight sets obtained from multiple clients to determine an updated set of weights. After merging, the correspondingly retrained ML model can be redeployed to the client. Corresponding downlink messages indicating updated sets of weights can be provided to multiple edge deployment processors participating in the federated learning. Thus, additional clients can benefit from the retraining of the edge deployment performed at another client while maintaining privacy. In particular, information about the performed gestures is not shared between clients or between clients and servers. Instead, these weight sets are shared, which only include encoded information about the gestures that resulted in such updates to the weights.

[0016] The present invention discloses a method for operating a processor. For example, the processor can be edge deployed. The method includes obtaining at least one feature vector. At least one feature vector encodes measurement data. The measurement data is provided by a sensor. For example, the sensor can be a depth sensor, such as a radar sensor or a light detection and ranging (LIDAR) sensor, a time of flight (TOF) sensor, a radar sensor, or a capacitive sensor. For example, the sensor can be edge deployed. The processor and the sensor can be deployed together. Measurement data is provided for a gesture performed by a user. The method also includes inferring a gesture category prediction based on at least one feature vector. The above inference uses a machine learning model. The gesture category prediction is associated with the gesture performed by the user. The method also includes determining at least one feature correlation vector of at least one feature vector. Each feature correlation vector in at least one feature correlation vector includes a feature correlation value. Each feature correlation value in the feature correlation value indicates the dependency of the gesture category prediction on the corresponding one or more feature values ​​of at least one feature vector. The method also includes determining a user output. The user output is associated with the gesture. The user output is determined based on at least one feature correlation vector. The method also includes controlling a user interface to provide a user output to the user.

[0017] Program code executable by a processor is disclosed. The processor may be edge deployed. Execution of the program code causes the processor to perform a method. The method includes obtaining at least one feature vector. The at least one feature vector encodes measurement data. The measurement data is provided by a sensor. For example, the sensor may be a radar sensor. For example, the sensor may be edge deployed. The processor and the sensor may be deployed together. Measurement data is provided for a gesture performed by a user. The method also includes inferring a gesture category prediction based on the at least one feature vector. The inference utilizes a machine learning model. The gesture category prediction is associated with a gesture performed by the user. The method also includes determining at least one feature relevance vector for the at least one feature vector. Each feature relevance vector in the at least one feature relevance vector includes a feature relevance value. Each feature relevance value in the feature relevance value indicates a dependency of the gesture category prediction on a corresponding one or more feature values ​​of the at least one feature vector. The method also includes determining a user output. The user output is associated with the gesture. The user output is determined based on the at least one feature relevance vector. The method also includes controlling a user interface to provide the user output to the user.

[0018] The present invention discloses a processor. The processor is configured to execute the method described above. The execution of the method may be based on a program code loaded and executed by the processor.

[0019] The present invention discloses a method for operating a server. The server can be a central server, that is, communicating with multiple clients. The method includes obtaining a corresponding uplink message from each of a plurality of processors (e.g., edge-deployed processors). Each of the uplink messages indicates a corresponding weight of a machine learning model. The machine learning model is used at the processor to infer gesture category predictions based on a feature vector representing measurement data obtained from a sensor. For example, the measurement data can be obtained from a radar sensor. For example, the sensor can be deployed together with an edge-deployed processor. The method also includes merging weights to determine updated weights of the machine learning model. The method also includes providing a corresponding downlink message indicating updated weights of the machine learning model to at least one edge-deployed processor among the plurality of edge-deployed processors.

[0020] A program code executable by a processor is disclosed. The processor may be edge deployed. Execution of the program code causes the processor to execute a method for operating a server. The server may be a central server, i.e., communicating with multiple clients. The method includes obtaining a corresponding uplink message from each of a plurality of processors (e.g., edge deployed processors). Each of the uplink messages indicates a corresponding weight of a machine learning model. The machine learning model is used at the processor to infer a gesture category prediction based on a feature vector representing measurement data obtained from a sensor. For example, the measurement data may be obtained from a radar sensor. For example, the sensor may be co-deployed with an edge deployed processor. The method also includes merging weights to determine updated weights of the machine learning model. The method also includes providing a corresponding downlink message indicating updated weights of the machine learning model to at least one of the plurality of edge deployed processors.

[0021] A server is disclosed. The server is configured to execute the method described above. The execution of the method may be based on program code loaded and executed by a processor or multiple processors of the server.

[0022] The present invention discloses a system, which comprises the above server and one or more above processors.

[0023] It is to be understood that the features mentioned above and features not yet explained below can be used not only in the respective combination indicated but also in other combinations or alone, without departing from the scope of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 schematically illustrates a system including a radar sensor and a processing device for edge deployment according to various examples;

[0025] Figure 2 The diagram shows various examples of Figure 1 Additional details of the radar sensor;

[0026] Figure 3 schematically illustrates a plurality of gesture categories for which ML models may be used to infer gesture category predictions according to various examples;

[0027] Figure 4 A system including a central server and a plurality of edge-deployed devices is schematically illustrated;

[0028] Figure 5 is a flow chart of a method that may be performed by an edge deployment processor according to various examples;

[0029] Figure 6 schematically illustrates a time series of local feature correlation values ​​according to various examples;

[0030] Figure 7 The global feature correlation values ​​are schematically illustrated;

[0031] Figure 8 is a flow chart of a method that may be performed by an edge deployment processor according to various examples;

[0032] Fig. 9 is a flow chart of a method that may be performed by a central server according to various examples;

[0033] Fig.10 is the pseudocode of the algorithm according to various examples;

[0034] Fig.11 illustrates feature correlation values ​​(implemented by global Shapley values) for multiple dimensions of a feature vector (feature range, Doppler, azimuth, elevation, and peak) obtained from measurements provided by a radar sensor according to various examples for non-anomalous gestures;

[0035] Fig.12 roughly corresponds to Fig.11 , however, illustrates feature correlation values ​​for abnormal gestures according to various examples; and

[0036] Fig.13 Pictured Fig.11 and Fig.12 The order of the feature correlation values. DETAILED DESCRIPTION

[0037] Some examples of the present disclosure generally provide multiple circuits or other electrical devices. All references to circuits and other electrical devices and the functions provided by each device are not intended to be limited to only cover the contents shown and described herein. Although specific labels can be assigned to the various circuits or other electrical devices disclosed, such labels are not intended to limit the operating range of circuits and other electrical devices. Such circuits and other electrical devices can be combined and / or separated from each other in any way based on the specific type of electrical implementation required. It should be recognized that any circuit or other electrical device disclosed herein may include any number of microcontrollers, graphics processor units (GPUs), integrated circuits, memory devices (e.g., flash memory (FLASH), random access memory (RAM), read-only memory (ROM), electrically programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM) or other suitable variants thereof), and software that cooperates to perform (one or more) operations disclosed herein. In addition, any one or more of the electrical devices may be configured to execute a program code, which is implemented in a non-transient computer-readable medium, which is programmed to perform any number of functions disclosed.

[0038] Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings. It should be understood that the following description of the embodiments should not be considered restrictive. The scope of the present invention is not intended to be limited by the embodiments or drawings described below, which are only for illustration.

[0039] The accompanying drawings should be considered as schematic diagrams, and the elements shown in the accompanying drawings are not necessarily shown to scale. On the contrary, the various elements are represented so that their functions and general purposes become clear to those skilled in the art. Any connection or coupling between the functional blocks, devices, components or other physical or functional units shown in the figures or described herein can also be achieved by indirect connection or coupling. The coupling between components can also be established by wireless connection. The functional blocks can be implemented with hardware, firmware, software or a combination thereof.

[0040]

[0013] Hereinafter, techniques for analyzing predictions of ML models are disclosed. Various kinds and types of ML models can benefit from the techniques disclosed herein. Examples include classification tasks performed by ML models. Examples of ML algorithms include deep neural networks, such as convolutional neural networks, support vector machines, multilayer perceptrons, and the like.

[0041] ML models are used to infer predictions based on feature vectors. Feature vectors encode the measured data.

[0042] The measurement data may be acquired using various measurement modalities and sensors. The measurement data may be acquired using a depth sensor, such as an optical time-of-flight (ToF) sensor, e.g., a time-of-flight camera or a light detection and ranging (LIDAR) sensor. Another type of depth sensor is a radar sensor, e.g., a millimeter wave or ultra-wideband radar sensor.

[0043] Time of Flight (ToF) cameras perform ranging measurements based on the duration of a light pulse traveling to a scene and back. ToF cameras typically use light emitting diodes as light sources. LIDAR sensors use laser diodes or lasers as light sources. LIDAR sensors can use ToF ranging or the measurement of interference of a continuous wave of primary light and a continuous wave of reflected secondary light. The light sources for ToF cameras and LIDARs are typically in the infrared or deep infrared region. The various techniques disclosed herein will focus on use cases where radar sensors are used to obtain measurement data; however, similar techniques can also be applied to other types of sensors.

[0044] In general, all depth sensors can provide features related to the distance or range of an object. Some depth sensors can provide additional information, such as the speed of an object. Depth sensors can have lateral resolution (i.e., perpendicular to the range direction). This lateral resolution can be obtained by separating the corresponding beam paths (e.g., using scanning mirrors or using multiple separated beam paths simultaneously) and / or using interference-based beam forming.

[0045] Figure 1 The system 65 is schematically illustrated. The system 65 can be edge deployed. The system 65 includes a sensor 70 and a processing device 60. In the following, the sensor 70 is exemplified as a radar sensor in a non-limiting manner and will be referred to as such, but the sensor 70 can be any sensor, such as a depth sensor. The processing device 60 can obtain measurement data 64 from the radar sensor 70. The processor 62 (e.g., a general processor (central processing unit CPU), a field programmable gate array (FPGA), an application-specific integrated circuit (ASIC)) can receive the measurement data 64 via an interface 61 and process the measurement data 64. For example, the measurement data 64 can include a data frame that includes samples of an analog-to-digital converter. Additional preprocessing can also be implemented at the radar sensor 70; for example, the radar sensor 70 can output a 2-D spectrogram, such as a range Doppler spectrogram or an azimuth elevation spectrogram or a range time spectrogram or a Doppler time spectrogram or an azimuth time spectrogram or an elevation time spectrogram. Based on such preprocessing, a feature vector encoding the measurement data can be determined.

[0046] For example, a typical feature vector determined based on measurement data from radar sensor 70 includes one or more of the following dimensions: the distance of the gesture object; the velocity of the gesture object (sometimes also referred to as the Doppler shift); the angular orientation of the gesture object; the azimuth of the gesture object; and the elevation of the gesture object.

[0047] Figure 2 Various aspects of radar sensor 70 are illustrated. Radar sensor 70 includes a processor 72 (labeled as a digital signal processor DSP) coupled to a memory 73. Based on program code stored in memory 73, processor 72 can perform various functions related to transmitting radar pulses 86 using a transmit antenna 77 and a digital-to-analog converter (DAC) 75. Once radar pulse 86 has been reflected by scene 80, the corresponding reflected radar pulse 87 can be received by processor 72 using ADC 76 and multiple receive antennas 78-1, 78-2, 78-3 (e.g., arranged in an L-shape with a half-wavelength distance; see FIG. 2 ). Figure 2 The processor 72 may process the raw data samples obtained from the ADC 76 to a greater or lesser extent. For example, a data frame may be determined and output. In addition, a spectrogram may also be determined.

[0048] The radar measurement can be implemented as a basic frequency modulated continuous wave (FMCW) principle. A frequency chirp can be used to implement the radar pulse 86. The frequency of the chirp can be adjusted in the frequency range of 57 GHz to 64 GHz. The transmit signal is backscattered and has a time delay corresponding to the distance of the reflecting object captured by all three receiving antennas. The received signal is then mixed with the transmit signal and then low-pass filtered to obtain an intermediate signal. The frequency of this signal is significantly lower than the transmit signal, and therefore the sampling rate of the ADC 76 can be reduced accordingly. The ADC can operate at a sampling frequency of 2 MHz and 12-bit accuracy.

[0049] As shown, scene 80 includes multiple objects 81-83. For example, objects 81, 82 may correspond to the background, while object 83 may belong to the user's hand. Based on radar measurements, gestures by the hand may be classified. Figure 3 Some gesture categories that can be distinguished are illustrated in FIG.

[0050] Figure 3 Such gestures 501-510 and corresponding labels are schematically illustrated for gesture category 520, but other gestures are possible. According to the techniques described herein, gesture category predictions for such gestures 501-510 can be reliably inferred.

[0051] Such gesture category inference can be implemented on an edge-deployed processor (such as processor 62). Figure 4As explained, a central server may also be part of the overall data processing.

[0052] Figure 4 The system 130 is schematically illustrated. The system 130 comprises a central server 139. For example, the central server 139 may be located in a server farm. The central server 139 may be an Internet server connected to the Internet.

[0053] The central server 139 includes a processor 311 coupled to a memory 312. The processor 311 can load program code from the memory 312 and execute the program code to perform the technology disclosed herein. The processor 311 is also coupled to a communication interface 313. The processor 311 can transmit messages or receive messages via the communication interface. For example, the server can be coupled to the Internet via the interface 313.

[0054] The system 130 also includes a plurality of edge deployment devices 131-134, which may also be referred to as clients. The edge deployment devices 131-134 may be automobile control units (sometimes also referred to as vehicle head units). The edge deployment devices 131-134 may be vending machines, such as for distributing train tickets, etc. The edge deployment devices 131-134 may be game consoles.

[0055] Each of the edge deployment devices 131-134 may include: Figure 1 The system 65 is shown.

[0056] like Figure 4 As shown, edge deployment devices 131-135 are communicating with a central server 139, such as via the Internet. For example, cellular radio access technology may be employed.

[0057] According to an example, the central server 139 may provide a downlink message to the edge deployment devices 131-134 indicating the ML model 121. For example, the downlink message may indicate the architecture (e.g., number of layers, skip connections, layer types, etc.). The ML model 121 may then be used at each of the plurality of edge deployment devices 131-134 to infer a gesture category prediction associated with a gesture performed by a corresponding user based on a feature vector encoding the locally acquired measurement data.

[0058] The ML model 121 can also be retrained at each edge deployment device in the edge deployment device 131 based on the locally populated corresponding training data set. In particular, a user output can be determined, which is based on the feature correlation vector at each edge deployment device in the plurality of edge deployment devices 131-134. Such a feature correlation vector can be determined based on the XAI method. Then, based on the feature correlation vector indicated to the user, the user can decide whether to include the corresponding feature vector in the training data set. Thus, a carefully planned training data set is obtained, which includes customized feature vectors and associated true value gesture categories. Therefore, each of the initial versions of the ML model 121 can be personalized by edge-deployed retraining at the edge deployment devices 131-134. Then, an uplink message can be provided by the edge deployment devices 131-134 to the central server 139, and the uplink message indicates the weight 122 of the ML model 121 when such retraining is completed. This enables the server 139 to collect such updates from multiple edge deployment devices and combine or specifically average the corresponding weights 122 to determine an updated version of the ML model 121. Then, in further iterations, the updated version of the ML model 121 may be provided to the edge deployment devices 131-134. The corresponding further downlink messages may include updates to the weights; it may not be necessary to signal all parameters of the ML model, for example, the architecture may be fixed, the number of layers may be fixed, and the layer types may remain unchanged.

[0059] As can be understood from the above, federated learning techniques can be used. Federated learning helps train ML models shared among multiple decentralized clients while ensuring that the training dataset remains distributed and private. Instead of transmitting training data from the client to a central server for training, federated learning allows the ML model to be trained locally on each edge deployment device, where only the ML model updates (weights) are sent back to the central server. (This will be combined later.) Figure 8 Explain the details about federated learning)

[0060] As a general rule, it can be assumed that Figure 4 Various modifications and additions. For example, in Figure 4In , a scenario has been discussed in which one or more versions of the ML model 121 are provided to the edge deployment devices 131-134 by the central server 139. Optionally, auxiliary information can be provided from the central server 139 to one or more edge deployment devices 131-134. For example, such auxiliary information can include information that enables the edge deployment device 131-134 to determine whether a certain gesture being detected and classified is abnormal or non-abnormal. For example, one or more thresholds can be provided, such as upper thresholds and / or lower thresholds of feature values ​​or feature correlation values ​​or feature vectors or feature correlation vectors. Alternatively or in addition, reference feature values ​​and / or reference feature correlation values ​​can be provided to the edge deployment devices 131-134 by the server 139. (The corresponding technology will be discussed in more detail later in conjunction with Table 2.)

[0061] Figure 5 is a flow chart of a method according to various examples. Figure 5 Generally related to XAI methods. Figure 5 Generally involves enabling user interaction based on information provided by XAI methods.

[0062] Figure 6 is a method used to operate a processor. For example, an edge-deployed processor can be based on Figure 5 method to operate. Figure 5 The method may be executed by the processor 62 of the processing device 60 .

[0063] Figure 5 Generally involves gesture classification combined with XAI methods.

[0064] At block 3005, it is determined whether a gesture class prediction is to be inferred for a new gesture performed by the user. For example, at block 3005, it may be determined whether one or more feature vectors are newly acquired from pre-processing of the radar measurement frame. At block 3005, the gesture detection algorithm may signal that the user is performing a new gesture.

[0065] If a new gesture category prediction is to be inferred, the method begins at box 3010.

[0066] At block 3010, one or more feature vectors are obtained. The one or more feature vectors each include feature values ​​of multiple dimensions. Thus, these feature values ​​describe a certain number of features of the gesture. For example, each feature vector may include one or more dimensions associated with one or more of the following attributes measured by the radar: range, Doppler, azimuth, elevation; peak.

[0067] For example, each feature vector may encode radar measurement data included in one or more data frames acquired from a radar sensor.

[0068] Corresponding techniques for determining feature vectors based on processing radar measurement data are known in the art. The specific implementation of the preprocessing for obtaining feature vectors encoding the measurement data obtained from the radar sensor is independent of the techniques disclosed herein. More specifically, the techniques disclosed herein are able to flexibly process feature vectors having features associated with different physical observables. For example, a time-of-flight sensor that can be used as sensor 70 instead of a radar sensor may not be able to determine the Doppler shift (as a radar sensor) and may therefore lack corresponding features. Nevertheless, the techniques disclosed herein can also be used to process such feature vectors when inferring gesture category predictions.

[0069] For example, a time series of multiple feature vectors may be obtained. Gestures are typically performed over extended durations, such as 200 milliseconds to 2 seconds. Multiple feature vectors may sample different locations in such a gesture time window.

[0070] At box 3015, a gesture class prediction is inferred based on the at least one feature vector using the ML model in its current training state.

[0071] For example, if multiple feature vectors are obtained at block 3010, these multiple feature vectors may be concatenated and provided as a combined input to the ML model.

[0072] The output of box 3015 is an indicator indicating the category of the gesture. For example, according to Table 1, an integer value in the range of 1-10 can be output:

[0073]

[0074]

[0075] Table 1: Examples of various gesture categories that can be inferred. See also Figure 3 .

[0076] At block 3020, a feature correlation vector is determined. The feature correlation vector is associated with the feature vector obtained at block 3010 and the gesture class prediction inferred at block 3015. The feature correlation vector indicates the dependency of the gesture class prediction of block 3015 on the feature values ​​of the feature vector.

[0077] The feature relevance vector includes multiple feature relevance values. Each feature relevance value is associated with an associated feature value of the feature vector. The feature relevance values ​​explain the relative contribution of each corresponding feature value to the prediction of the ML model.

[0078] For example, considering that a given feature value has a greater impact on gesture category prediction than another feature value, the corresponding feature relevance value associated with the given feature value has a higher value than the feature relevance value associated with the other feature value.

[0079] As a general rule, there are multiple options for determining feature relevance values. There are many XAI methods available in the prior art and can be relied upon. For example, see Lundberg, Scott M. and Su-In Lee, "A unified approach to interpreting model predictions" (Advances in neural information processing systems 30 (2017)) - the so-called SHAP method. Shapley values ​​can be calculated. The Shapley value is the average marginal contribution of the feature value to the prediction of the ML model. The Shapley value provides the additive impact of each feature value on a specific local prediction of the ML model. See also Ribeiro, Marco Tulio, Sameer Singh and Carlos Guestrin, "Why should I trust you?" Explaining the predictions of any classification" (Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining. 2016) - the so-called LIME method.

[0080] The Shapley XAI method fairly distributes values ​​among a set of individuals or features based on their contribution to the overall outcome. The algorithm calculates the average marginal contribution of each feature value in all possible alliances, which are combinations of the presence or absence of the feature. See Shapley, Lloyd S. "A value for n-person games" ((1953): 307-317). To achieve this, it first generates predictions for different alliances with and without the analyzed feature, and then takes the difference between these predictions to calculate the marginal contribution of the feature. This process is repeated for all features, and the resulting value is the Shapley value, which represents the importance of each feature to the model prediction. This is defined as:

[0081]

[0082] where g is the ML model used for prediction, z′∈{0,1} is the coalition vector, is the Shapley value of feature j, and M is the maximum coalition size.

[0083] The SHAP method provides interpretation at both the local level (local Shapley value) and the global level (global Shapley value). The global Shapley value is obtained by averaging all the absolute local Shapley values ​​used for a specific inference task:

[0084]

[0085] Among them I j is the global explanation of feature j, and n is the number of samples in the dataset. As a general rule, feature relevance (sometimes also called feature importance) can be determined at a local level over time, for example by calculating the local Shapley value for each gesture frame. In addition, by calculating the global Shapley value, features that have little or no overall influence can be determined.

[0086] As a general rule, a single or multiple feature correlation vectors may be determined at block 3020 .

[0087] For example, one or more first feature correlation vectors may be determined that include local Shapley values; and further, a second feature correlation vector may be determined that includes global Shapley values.

[0088] For example, if a plurality of feature vectors are obtained at box 3010 and processed at box 3015, a plurality of first feature correlation vectors may be determined, each of the plurality of first feature correlation vectors being associated with a corresponding feature vector in the plurality of feature vectors and including a corresponding local Shapley value. In addition, a second feature correlation vector may be determined, the second feature correlation vector including a global Shapley value and being determined based on the plurality of first feature correlation vectors.

[0089] To illustrate, Figure 6 The time series of the amplitudes of the local Shapley values ​​601, 602 of the features "distance" and "Doppler" included in the time series of the corresponding feature values ​​are illustrated. A time window 605 associated with the execution of the gesture is also illustrated. Feature values ​​and feature correlation values ​​outside the time window 605 can be discarded. The time window 605 can be determined based on prior knowledge and / or based on a threshold technique (threshold 606 is shown). Although Figure 6 Local Shapley values ​​601, 602 are illustrated, but alternatively or additionally, a global Shapley value may be calculated. To this end, a time series of feature correlation values ​​may be aggregated within a time window 605. This is Figure 7 Medium picture. Figure 7The sum of all absolute local Shapley values ​​601 during a time window 605 and the sum of all absolute Shapley values ​​602 during a time window 605 are illustrated; these are the global Shapley values ​​611 , 612 .

[0090] Now refer to it again Figure 5 : At optional block 3025, the observed gesture may be classified as abnormal or non-abnormal. That is, in addition to the classification of gesture type (block 3015), a binary classification of abnormal and non-abnormal may also be performed.

[0091] As a general rule, in box 3020, various options are available for classifying gestures as abnormal or non-abnormal.

[0092] The classification of block 3020 may be based on one or more of the decision criteria listed in Table 2 below:

[0093] (1) One or more feature correlation vectors of block 3020 (2) One or more feature vectors of block 3010 (3) Uncertainty measures for gesture category prediction

[0094] Table 2: Multiple criteria that can be used alone or in combination with each other to determine whether a gesture is abnormal.

[0095] See example (1) of Table 2 - Classification based on one or more feature correlation vectors. One or more attributes of the feature correlation vectors may be considered to determine whether a gesture is abnormal.

[0096] For example, a relationship between each feature correlation value in the feature correlation values ​​of one or more feature correlation vectors and an associated threshold value may be considered. Such a threshold value may be predefined. The threshold value may be defined based on previously determined feature correlation values ​​(as will be explained later in conjunction with block 3032). The threshold value may be obtained from a user training process in which the user is asked to perform a certain gesture a certain number of times. The threshold value may be determined based on the corresponding feature value. The threshold value may be static or dynamic. A static threshold value is fixedly predefined, for example, once at a central server such as the central server 139 (see Figure 4 ) and provided to each client (see Figure 4 , downlink messages to edge deployment devices 131-134). The dynamic threshold may be determined locally at each edge deployment device in edge deployment devices, such as edge deployment devices 131-134. Different edge deployment devices then use different thresholds. Such thresholds may be determined based on previous feature correlation values, for example, by considering a normal range of feature correlation values. A sliding window may be used to select previous feature correlation values. That is, historical feature correlation values ​​may be considered.

[0097] Alternatively or in lieu of using a threshold, a reference feature relevance value may also be used. In the case of using a reference feature relevance value, the relationship between each feature relevance value in the feature relevance value and the reference feature relevance value may be considered. The distance between a feature relevance vector and one or more predefined feature relevance vectors may be considered. The similarity between the feature relevance value and the reference feature relevance value may be compared using distance and / or similarity metrics, such as Euclidean distance, Manhattan distance or Mahalanobis distance and cosine similarity. The reference feature relevance value may be a historical average, for example, an average feature relevance value of n previously recorded for each gesture category. The reference feature relevance value may be obtained from a user training process, in which the user is asked to perform a specific gesture for a specific number of times. For example, basic facts about a gesture category may be obtained from user input. Then, a reference feature relevance value for the gesture category may be determined. Then, the dissimilarity or distance between each reference feature relevance value and the associated feature relevance value serves as a basis for determining whether the gesture is abnormal or non-abnormal.

[0098] As an alternative or in addition to the thresholds and / or references, the order of the feature correlation values ​​relative to each other can be considered. For example, it can be determined whether the order of the feature correlation vectors changes between the nominal order (e.g., determined based on previous feature correlation vectors associated with the same gesture category). In such an analysis, the slope or distance between consecutive feature correlation values ​​can be considered.

[0099] Above, the scenario has been discussed in conjunction with Example (1) of Table 2. Similar aspects can also be applied to Example (2) of Table 2. That is, instead of or in addition to using a threshold, reference, or feature correlation value order, a threshold, reference, or feature value order can be used. The aspects of dynamic determination and / or static determination explained above in conjunction with Example (1) are also applicable to Example (2).

[0100] Next, example (3) of Table 2 will be discussed. In such a scenario, the above classification of gestures as abnormal or non-abnormal is based on the uncertainty measure associated with the above inference at box 3015. Various prior art solutions can be used to determine the uncertainty measure, such as Monte Carlo Dropout, Evidential Deep Learning, etc. Evidential deep learning is described in "Evidential deep learning to quantify classification uncertainty" (Advances inneural information processing systems 31 (2018)) by Sensoy, Murat, Lance Kaplan, and Melih Kandemir. Evidential deep learning has the advantage of predicting uncertainty in a single pass of the neural network. A specific measure of classification uncertainty is the probability distribution between multiple candidate gesture categories predicted by the gesture category of the ML model. The output layer of the ML model may include multiple neurons, each neuron indicating the probability of a corresponding gesture category. An example is a Softmax layer. The distribution between such output neurons can be considered. For example, if the distribution is uniform so that many output neurons have similar values ​​and the most likely output class has approximately the same probability as the other output classes, then the prediction can be considered uncertain and the gesture can be considered an anomalous gesture. On the other hand, if the probability of the most likely class is significantly higher than the other classes, then the prediction is relatively certain and the gesture can be considered a non-anomalous gesture. Such an approach can be based on a threshold comparison of the relative deviation between the classification probabilities. The specific technique chosen to determine the uncertainty measure is independent of the function.

[0101] In summary, the above techniques have been discussed in conjunction with Table 2, which is able to classify gestures as abnormal or non-abnormal.

[0102] In box 3030, it is evaluated whether the gesture is classified as abnormal in box 3025. If it is determined at box 3030 that the gesture is not classified as abnormal at box 3025, the method proceeds to box 3031. At box 3031, a use case is implemented based on the gesture category prediction. For example, a device or machine can be controlled. A user interface can be controlled based on the detected gesture category. For example, a vehicle infotainment system can be controlled based on the gesture category prediction. For example, a vending machine can be controlled based on the gesture category prediction. The door of a car can be opened or closed based on the detected gesture category. Various use cases are conceivable.

[0103] Once box 3031 is completed, the method starts at box 3032. Box 3032 is an optional box. Optionally, based on the record of the feature vector and / or feature correlation vector associated with the current gesture, one or more thresholds and / or references for classifying the gesture as abnormal or non-abnormal can be adapted in subsequent iterations 3025. The corresponding techniques have been discussed above in conjunction with Table 2. For example, at box 3032, it can be determined whether the current feature value and / or feature correlation value is lower than or exceeds the corresponding minimum and maximum feature values ​​and / or feature correlation values ​​stored from the previous iteration; if this is the case, the corresponding current feature value and / or feature reference value can be stored as the new minimum or maximum threshold. In another example, the threshold can be determined based on the 3-Sigma rule: the mean μ and standard deviation σ of each feature and gesture category are calculated on n global feature correlation values ​​(e.g., global Shapley values) and / or feature values ​​from good records. The lower threshold is calculated as μ-i*σ and the upper threshold is calculated as μ+i*σ, where i∈{1, 2, 3}. In another example, weighted thresholds can be considered: With this threshold mechanism, the feature values ​​are taken into account to determine the thresholds for the feature relevance values. For each gesture category, the feature value ranges for each feature over the n previous records are extracted. In addition, for each feature, the average global Shapley value over the n good records is calculated. For each feature, the feature value ranges can then be weighted with the average global Shapley value and used as the lower and upper thresholds.

[0104] Finally, a further iteration of block 3005 begins, namely, determining whether a new gesture is to be classified.

[0105] If it is determined at block 3030 that the gesture was classified as abnormal at block 3025, the method starts at block 3035. At block 3035, a user output associated with the gesture is determined. The user output is determined based on the feature correlation vector determined at block 3020.

[0106] The user output may indicate a feature relevance vector. The user output may indicate one or more of the feature relevance values. For example, the user output may indicate an order of the feature relevance values. The user output may indicate an order of the feature relevance values, for example, indicating an order of global Shapley values ​​(see Figure 7). The user output may indicate features that led to the classification of the gesture as abnormal at box 3025. For example, such feature correlation values ​​that exceed predefined norms may be indicated, for example, below or above a certain threshold or significantly deviating from a reference. The same threshold or reference used in box 3025 may also be used here. Such a reference or threshold may be determined as discussed in conjunction with box 3032, i.e., based on historical feature correlation vectors and / or a user training process. Alternatively, the reference or threshold may be fixedly predefined, for example, configured by a central server. Since different gestures may be associated with different reference feature correlation values ​​and / or different thresholds, an appropriate reference and / or threshold may be selected based on the true value gesture category of the gesture (e.g., entered by the user). Thus, the user interface may be controlled to obtain a user input associated with the user output, the user input indicating the true value gesture category of the gesture. To give a specific example, the user may indicate a left and right swipe gesture 501 (see Figure 3 ) is performed. Then, relevant upper and lower thresholds can be selected and loaded for the feature relevance values ​​for multiple dimensions of the feature relevance vector; and each feature relevance value is compared with these upper and lower thresholds. Then, such dimensions of the feature relevance vector that are below the corresponding minimum threshold or exceed the corresponding upper threshold can be output to the user. Alternatively or additionally, it can also be indicated whether certain dimensions of the feature relevance vector are out of order. In other words, the left-to-right swipe gesture 501 can be associated with a specific order of feature relevance values, such as from large to small. Whenever this order is broken, a corresponding indication can be provided to the system user.

[0107] At block 3040, the user interface is controlled to provide the user output determined at block 3035. For example, the output may be Figure 7 The graph shown.

[0108] The user output may be provided to the user as part of a retraining process for populating a training data set to retrain an ML model. In such a scenario, the user may be prompted with certain attributes of the currently executed gesture; this enables the user to reflect whether the gesture has been properly executed and whether the gesture is to be included in the training data set. This enables the acquisition of a carefully curated training data set in which irrelevant or unimportant gestures are excluded. Therefore, at box 3050, it may be determined whether the gesture has been properly executed. Only in the affirmative case, the feature values ​​associated with the gesture and the associated true value gesture category of the gesture will be added to the training data set. To this end, the user interface may be controlled to obtain user input. The user input may be associated with the user output provided at box 3045. The user input may indicate the true value gesture category of the gesture. In other words, the user may indicate which type of gesture was previously executed. The user may also indicate whether the gesture has been accurately executed based on the user output provided at box 3045. The user input may confirm the gesture positively or negatively. Only in the affirmative case, the feature vector will be included in the training data set.

[0109] Figure 5 is an example flow chart, and various modifications may be made. For example, at block 3030, the criteria for determining whether a gesture is abnormal may not depend on the feature correlation value (see Example 2 or 3 of Table 2). In such a scenario, block 3020 may be executed only after the gesture is determined to be abnormal at block 3030. That is, block 3030 may be before block 3020.

[0110] To further illustrate, the decision criteria for classifying a gesture as abnormal at block 3025 may depend on user input. For example, the user may input a reference or threshold. In such a scenario, block 3025 may be preceded by controlling the user interface to obtain the corresponding user input.

[0111] For further explanation, a scenario is disclosed above in which user output is provided to the user as part of a retraining process for retraining an ML model. Alternatively or additionally, the user output may be provided as part of a guided and interactive user training process. For example, in such a guided and interactive user training process, the user may be prompted to perform a certain category of gestures. In response to such a prompt, the user performs the corresponding gesture, and then as part of box 3040, corresponding information about one or more dimensions of the feature vector is obtained, which has a relevant feature correlation value that deviates from the relevant reference, such as in the case of an abnormal gesture. In such a scenario, the true value gesture category is inherently available because the user is prompted to perform a certain gesture. The guided and interactive training process helps the user to re-execute the gesture in a more appropriate manner by adapting the specific characteristics of the gesture associated with the dimension or dimensions that deviate from the reference. For example, it can be indicated to the user that, in particular, the speed / Doppler dimension deviates from the speed / Doppler reference of the "swipe left and right" gesture. The user can then re-execute the gesture, but everything is the same as the previous attempt except for the movement speed; the user can move the hand faster or slower, depending on whether the predefined reference has been executed. This helps train the user to perform better gestures. For example, unclean gesture trajectories (i.e., jittery motions) can be marked and corrected by the user. Incorrect angles and distances to the radar can be marked and corrected by the user. Inappropriate gesture execution speeds can be identified. Inadequate radial gesture execution can be identified.

[0112] Above, aspects of the ML model that support XAI retraining and / or user training to correctly perform gestures have been disclosed. The initial version of the ML model can be provided by the central server. In addition, updates to the ML model can be provided by the central server. Next, we will combine Figure 8 Discuss this.

[0113] Figure 8 is a flow chart of a method according to various examples. Figure 8 The method may be performed by an edge deployment device, such as Figure 4 One of the edge deployment devices 131-134 is shown. Figure 4 The method may be specifically performed by an edge deployment processor, for example by the processor 62 when loading and executing program code stored in the memory 63.

[0114] Figure 8 The method generally involves bidirectionally exchanging updates of the weights of the ML model between edge-deployed devices and a central server.

[0115] At block 3105, it is determined whether a new version of the ML model is available. For example, an over-the-air-update process for the edge deployment device may be determined. An update process for the software or firmware of the edge deployment device may be triggered. For example, execution Figure 8 The processor of the method can be part of a car control unit (sometimes also referred to as a vehicle head unit). Here, it can be determined that an update process (e.g., over-the-air update) of the vehicle control processor has begun.

[0116] If it is determined at box 3105 that an update is available, the method starts at box 3110.

[0117] At block 3110, a downlink message is obtained from the central server. The downlink message indicates the weights of the ML model in the current training state / version. Thus, an update of the ML model is obtained. Techniques for providing such downlink messages (DL messages) of the current version of the ML model have been incorporated into Figure 4 The ML model 121 is discussed.

[0118] DL messages can be encrypted. For example, asymmetric encryption can be used, which uses a private key public key pair. And end-to-end encryption can be used.

[0119] Acquiring the ML model in its current training state at block 3110 may include registering the corresponding edge deployment device at the central server, and transmitting a download request from the edge deployment device to the central server. Acquiring the downlink message at block 3110 may include receiving the downlink message via a wireless carrier of a cellular network. Acquiring the downlink message at block 3110 may include acquiring the downlink message via the Internet.

[0120] Then, at block 3111, the updated ML model may be used for inference, i.e., for inferring gesture class predictions. This may involve performing a combination of Figure 5 As part of block 3111, the training dataset can be populated based on the feature vectors observed during inference. Figure 5 Box 3055 in discusses the corresponding technology.

[0121] At box 3115, it is determined whether retraining of the ML model is required and / or authorized and / or possible. For example, at box 3115, it may be checked whether the central server that provided the update at box 3110 has enabled or disabled retraining of the edge deployment of the ML model. Alternatively or additionally, at box 3115, it may be determined whether the training data set populated in box 3111 has reached a certain size. Retraining may only be performed if sufficient training data is available. If retraining is to be performed, the method starts at box 3120.

[0122] At block 3120, the training data set may optionally be augmented. Augmenting the training data set may include one or more of: Gaussian noise injection, time warping, drifting.

[0123] Data augmentation involves increasing the size of the training dataset without taking additional measurements. New training samples can be generated by slightly modifying the original data. Augmented data provides a kind of regularization, reducing the chances of overfitting.

[0124] Gaussian noise injection involves adding random noise that follows a Gaussian distribution to the data. In the contextual feature correlation vector of radar measurements, this includes random variations in the value of each feature, such as range, Doppler, angular dimensions, etc. By adding this noise, the ML model is trained to be more robust to small distortions or perturbations, and thus generalize better to real-world noise scenarios. Time warping involves stretching or compressing the time axis of the feature correlation vector sequence (see Figure 6 ). Drift refers to the introduction of small changes in characteristic values ​​over time to simulate scenarios where systematic changes in sensor readings are observed.

[0125] At block 3125, retraining of the ML model is performed. Existing techniques for such retraining may be employed.

[0126] Various losses may be considered in block 3125. One example loss is a supervised loss (for labeled data samples) which may be expressed as follows:

[0127]

[0128] Here, N is the number of labeled samples (i.e., feature vectors) in the training dataset. C is the number of categories in the classification problem, i.e., the number of gesture categories. i，c is the true value label of sample i and category c. It indicates whether sample i belongs to category c (1 if it is true, 0 otherwise). This can be obtained from the user. is the predicted probability that sample i belongs to class c. It is the output of the model's softmax function, which represents the model's confidence in the predicted class probability. However, the specific training technique and in particular the specific loss adopted are not relevant.

[0129] After the retraining is complete, at block 3130, a UL message may be provided to the central server indicating the weights of the ML model at the time the retraining is complete. For example, an incremental update may be signaled that is limited to changes in weights compared to an earlier training state, e.g., as provided in the DL message at block 3110. This may include a gradual update. As explained above in conjunction with block 3110, block 3130 may be performed as part of an over-the-air update process.

[0130] UL messages may be encrypted. For example, asymmetric encryption may be used, which uses a private key public key pair. And end-to-end encryption may be used.

[0131] In short, combined Figure 8 , have disclosed edge retraining techniques that can reduce the impact of background motion (by retraining the model to ignore it), adapt ML models to the user's potential disabilities, and adapt ML models to different hand sizes and handedness.

[0132] The uplink and downlink updates of the ML model (see blocks 3110 and 3130) can enable joint learning of the ML model. Fig. 9 This is explained.

[0133] Fig. 9 is a flow chart of a method according to various examples. Fig. 9 Broadly involves federated learning. Fig. 9 and Figure 8 Interrelated.

[0134] Fig. 9 The method may be performed by a central server, for example, Figure 4 More specifically, when loading the program code from the memory 312 and when executing the program code, the processor 311 of the server 139 may execute Fig. 9 method.

[0135] At block 3205, an initial version of the ML model is trained. This may be based on a relatively large training data set acquired during a coordinated measurement campaign.

[0136] Then, at block 3210, the initial version of the ML model is distributed to multiple edge deployment devices. The central server may provide corresponding DL messages. Block 3210 is associated with block 3110.

[0137] The edge deployment devices may then use the initial version of the ML model for inference. They may also retrain the ML model locally and provide a corresponding UL message indicating the updated weights at box 3215. This may include gradient updates. The server may collect these updates and merge the weight sets obtained from multiple edge deployment devices at box 3220. This is called federated learning. Retraining may be performed. Data augmentation may be included, as discussed previously for the edge side at box 3120.

[0138] At block 3225, the server may provide a DL message to the edge deployment device indicating the new version of the ML model thus acquired. The updated weights may be signaled. Additional iterations 3226 may then begin. Such federated learning based on the merging of ML models retrained at multiple clients enables the ML model to improve over time, incorporating knowledge from different edge deployment devices.

[0139] There are several advantages to federated learning. First, it prioritizes privacy by ensuring that training data remains on edge devices (rather than being shared directly with central servers or other edge devices). This is particularly valuable in cases involving sensitive or personal data. Second, federated learning allows data to remain on edge devices, reducing the need to transfer large amounts of data to a central location. This is beneficial in cases where network bandwidth is limited or the cost of transferring data is high. In addition, federated learning reduces communication overhead by exchanging only model updates instead of raw data, making it suitable for devices with limited bandwidth or unstable connections. In addition, federated learning leverages the computing resources of multiple devices, enabling faster training and scalability. It helps to train on a large number of devices at the same time, potentially resulting in more diverse and more representative data sets. Finally, federated learning improves the generalization performance of the trained model by leveraging data from multiple devices, capturing a wider range of patterns, and reducing the risk of overfitting to specific device data. Federated learning is particularly valuable in applications where data privacy and security are critical, such as healthcare, finance, or Internet of Things (IoT) scenarios involving sensitive data. By keeping data local and distributed, federated learning addresses privacy issues while enabling collaborative model training across decentralized devices or servers.

[0140] Fig.10 is the pseudo code for the algorithm that implements the above technique. Figure 5 , Figure 8 and Fig. 9 method.

[0141] The pseudocode provided outlines an algorithm called “Radar SHAP Explanation” for the federated learning setting. The algorithm assumes that there are clients represented by a dataset L, where each client li Including weight w i and dataset D i There is also a central server l S With weights w and a dataset D. The goal is to perform federated learning to train an ML model using data from clients while preserving privacy.

[0142] The algorithm starts with the “Main()” function: it initializes the “SelectedFeatures” variable by evaluating the feature relevance vector using the “EvaluateFeatureImportance()” function on the server. It pretrains the ML model on the server using the “PretrainModel()” function, see box 3205.

[0143] The algorithm then enters a loop to perform the following steps after each time interval: It runs the server using the "RunServer()" function, which updates the server weight based on the average of the client weights, see box 3220. In addition, it also runs the "RunSensorinCars()" function.

[0144] The "RunSensorinCars()" function performs the following steps for each client: Collect data using the "CollectData()" function (see box 3215), and additionally enhance the data using, for example, Gaussian noise injection, time warping, drift, etc. Update the client agent weights using the "UpdateAgent()" function. i and data D i Update Client i .

[0145] If sharing is enabled, the algorithm calculates the average weight of all clients and updates the server with the average weight (see box 3220). Otherwise, it keeps the original client weight.

[0146] "RunSensorinCars()" function: Initialize a copy of the client data set to L0 (see block 3110). For each client l in L i , it performs the following steps: Collect data using the "CollectData()" function. Determine the feature correlation vector. Update the client agent weights using the "UpdateAgent()" function. Use the new weights w i and data D i Update Client i .

[0147] If sharing is enabled, it calculates the average weight of all clients and updates the server with the average weight. Otherwise, it returns the original client dataset and the server weight.

[0148] "EvaluateFeatureImportance()" function: Use the "UpdateAgent()" function to update the server weight. S The SHAP method (or other XAI methods) is used to calculate the feature relevance value ranking. Feature selection is performed based on the feature importance ranking, and the selected features are returned.

[0149] PretrainModel() function: Updates the server weights using UpdateAgent() function. Returns the server l with the updated weights S .

[0150] "RunServer()" function: set the weight Initialized to server weight w S For each round i (maximum R rounds), perform the following steps: S Use the "UpdateAgent()" function to update the server weight Randomly select a group of A clients from L as L r For each client in L It updates the client weight using the "UpdateAgent()" function The average of the client weights is calculated as Returns the updated server l S And its final weight

[0151] “UpdateAgent()” function: It performs local training iterations for the agent with weights w. In each local iteration, it updates the weights w by subtracting the learning rate eta times the gradient of the loss L.

[0152] "CollectData()" function: It collects data D for the client, which includes a set of data samples (x i ,y i ).

[0153] Fig.10 The algorithm operates by iteratively running server and client updates. The server aggregates the client weights and updates its own weight based on the average weight (see Fig. 9 ). The client collects data, updates its weights, and potentially shares its weights with the server (see Figure 8 ).

[0154] Fig.10 The algorithm also includes functions for evaluating feature importance, pre-training the ML model on the server, and updating the agent weights using local training (see box 3125).

[0155] The pseudocode assumes the presence of other functionality, such as the SHAP method for feature relevance ranking and feature selection algorithms, which are not explicitly defined in the pseudocode provided but are known in the art.

[0156] on the whole, Fig.10 The algorithm aims to perform federated learning, where clients contribute their local data while preserving privacy, and the server coordinates the aggregation of client updates to improve the performance of the model.

[0157] Fig.11 and Fig.12 An abnormal gesture using the techniques disclosed herein is shown Fig.10 As explained in conjunction with Tab 2, there are multiple ways to characterize abnormal gestures. Fig.11 and Fig.12 In , a minimum and maximum threshold mechanism based on the Shapley values ​​of n (n=20) good gestures is selected. That is, a dynamic threshold using a sliding window is adopted, where the threshold is adaptive (see box 3032). In order to make sufficient statistical representation during deployment, n can be increased, for example, to n=100.

[0158] Fig.11 and Fig.12 Shown at normal speed ( Fig.11 ) and at a fast speed ( Fig.12 ) gesture 501 (slide from left to right; see Figure 3 ) of the global Shapley value (see Figure 7 ). Fig.11 The gestures in are detected correctly, but in Fig.12 , the gesture is still not detected. Fig.12 In such scenarios, it is helpful to use the feature correlation vector of global Shapley values ​​including the dimensions "Doppler", "range", "azimuth", "elevation", and "peak" to understand the root cause of the wrong prediction. Fig.11Of the nominal gestures shown, the feature "range" has the greatest impact, followed by "Doppler"; then, depending on whether a left / right swipe (gesture 501) or an up / down swipe (gesture 503) was performed, it is followed by "azimuth" (horizontal angle) or "elevation" (vertical angle), in that order. The feature "peak" (signal amplitude) has almost no impact, which indicates that it does not contribute to the ML model output (and can even be omitted in future iterations). In more detail: "peak" represents the signal amplitude of the largest signal in the Doppler profile. After obtaining the range distribution, a fast Fourier transform is performed across all receive channels. After integrating the resulting Doppler profile across the channels, the maximum signal is searched. The position indicates the radial velocity, while the amplitude is the signal amplitude.

[0159] Now when comparing non-exceptional ( Fig.11 ) and exceptions ( Fig.12 ) When ranking the two global features of the gesture, it can be seen that the “Doppler” feature has a higher impact than the nominal feature. Since the user is asked to provide feedback on the actual gesture category (user input indicating the true value label), the correct Shapley reference value (based on the upper and lower thresholds of the gesture category 501) can be selected. Using Fig.11 The threshold mechanism shown and the interpolated median map based on the nominal gesture are obtained from Fig.13 (solid line) It can be seen that the slope between the range and Doppler characteristics should be nominally negative, because the range characteristic has the greatest impact on the model output. Fig.12 This is not the case for the abnormal gesture (dashed line). Therefore, a user output can be provided (see box 3040) indicating that speed is a factor that prevents gesture detection (it abnormally and significantly affects the ML model output) and needs to be adjusted. This user output can also be further subdivided, for example, by including typical behavior for fast gestures and very slow gestures. By comparing the typical Shapley value behavior of fast gestures and slow gestures, direct feedback can be given to the user that not only speed is the cause of the incorrect prediction, but also that the hand is moving too fast.

[0160] In summary, techniques have been disclosed that can improve the accuracy of ML models for gesture category prediction based on (i) XAI and / or (ii) federated learning. A decision process for identifying and characterizing abnormal gesture predictions is disclosed. For example, reference feature correlation values, such as those collected from an average ML model, predefined at a central server, or dynamically determined, and user-specific records compared to nominal gestures can be used to characterize abnormal gestures. There are several options for how to describe the difference between well-performing and abnormal gestures. These options include defining static or dynamic thresholds: defining upper and lower thresholds calculated based on the feature correlation vector of the nominal gesture to determine an acceptable range of values ​​for each feature correlation value. Based on these thresholds, possible root causes of abnormal gestures can be characterized. Alternatively or additionally, a similarity metric, such as Euclidean distance or cosine similarity, can be used. Techniques have been disclosed that can provide instructions to the user on how to repeat the gesture for a more informed recalibration process.

[0161] In summary, at least the following examples have been disclosed.

[0162] Example 1. A method of operating an edge deployment processor (62), the method comprising:

[0163] - acquiring (3010) at least one feature vector encoding measurement data (64) provided by a depth sensor (70) for a gesture (501-510) performed by a user,

[0164] - using a machine learning model (121) to infer (3015) a prediction of a gesture class associated with the gesture based on the at least one feature vector,

[0165] - determining (3020) at least one feature correlation vector of the at least one feature vector, each of the at least one feature correlation vector comprising feature correlation values, each of the feature correlation values ​​indicating a dependency of the gesture class prediction on corresponding one or more feature values ​​of the at least one feature vector,

[0166] - determining (3035) a user output associated with the gesture based on the at least one feature correlation vector, and

[0167] - controlling (3040) a user interface to provide said user output to said user.

[0168] Example 2. The method according to Example 1 further includes:

[0169] - classifying said gesture as anomalous or non-anomalous,

[0170] Wherein the user output is selectively provided to the user in response to classifying the gesture as abnormal.

[0171] Example 3. The method according to Example 2,

[0172] Wherein classifying the gesture as abnormal or non-abnormal is based on the at least one feature correlation vector.

[0173] Example 4. The method according to Example 3,

[0174] Wherein classifying the gesture as abnormal or non-abnormal is based on a threshold comparison between each of the feature correlation values ​​of the at least one feature correlation vector and a corresponding threshold.

[0175] Example 5. The method according to Example 4,

[0176] The threshold is determined based on a previously determined feature correlation value, or is obtained from a central server, or is fixedly predefined.

[0177] Example 6. The method according to any one of Examples 2 to 5,

[0178] Wherein classifying the gesture as abnormal or non-abnormal is based on a comparison between each of the feature correlation values ​​of the at least one feature correlation vector and a corresponding reference feature correlation value.

[0179] Example 7. The method according to Example 6,

[0180] The reference feature correlation value is determined based on a previously determined feature correlation value, or is obtained from a central server, or is fixedly predefined.

[0181] Example 8. The method according to any one of Examples 2 to 7,

[0182] Wherein classifying the gesture as anomalous or non-anomalous is based on an order of the feature correlation values.

[0183] Example 9. The method according to any one of Examples 2 to 8,

[0184] Wherein classifying the gesture as abnormal or non-abnormal is based on an uncertainty metric associated with inferring the gesture category prediction using the machine learning model.

[0185] Example 10. The method according to Example 9,

[0186] The uncertainty measure is determined based on a probability distribution among a plurality of candidate gesture categories predicted by the gesture category.

[0187] Example 11. A method according to any one of Examples 2 to 10,

[0188] Wherein classifying the gesture as abnormal or non-abnormal is based on a distance of the at least one feature vector from one or more predefined reference feature vectors.

[0189] Example 12. The method according to Example 11,

[0190] The one or more predefined reference feature vectors are determined based on previously acquired feature vectors, or are acquired from a central server, or are fixedly predefined.

[0191] Example 13. A method according to any one of Examples 2 to 12,

[0192] Wherein classifying the gesture as abnormal or non-abnormal is based on a threshold comparison between each of the feature values ​​of the at least one feature vector and a corresponding threshold.

[0193] Example 14. The method according to Example 13,

[0194] The threshold is determined based on previously acquired feature values, or obtained from a central server, or is fixedly predefined.

[0195] Example 15. A method according to any one of the preceding examples,

[0196] Wherein the user output is provided to the user as part of a retraining process for populating a training dataset to retrain the machine learning model.

[0197] Example 16. A method according to any one of the preceding examples,

[0198] Wherein the user output is provided to the user as part of a guided and interactive user training process.

[0199] Example 17. The method according to any of the preceding examples further includes:

[0200] - controlling the user interface to obtain a user input associated with the user output,

[0201] - selectively including the at least one feature vector in a training data set based on the user input, and

[0202] -Retrain the machine learning model based on the training data set to obtain the machine learning model.

[0203] Example 18. The method according to Example 17,

[0204] The user input indicates a true value gesture category of the gesture.

[0205] Example 19. The method according to Example 17 or 18,

[0206] Wherein the user input confirms the gesture either affirmatively or negatively.

[0207] Example 20. The method according to any one of Examples 17 to 19, further comprising:

[0208] -Providing (3130) an uplink message to a central server (139), the uplink message indicating the weights (122) of the machine learning model upon completion of the retraining.

[0209] Example 21. The method according to Example 20,

[0210] wherein the uplink message indicates an incremental update of the weights of the machine learning model compared to an earlier trained state of the machine learning model.

[0211] Example 22. The method according to Example 20 or 21,

[0212] The processor is part of the car's control unit.

[0213] The uplink message is provided during an over-the-air update process of the vehicle control unit.

[0214] Example 23. The method according to any one of Examples 17 to 22, further comprising:

[0215] -Before performing the retraining of the machine learning model, augmenting the training dataset based on one or more of noise injection; time warping; drift.

[0216] Example 24. A method according to any one of the preceding examples,

[0217] Wherein the user output is indicative of one or more feature values ​​of the at least one feature vector, the one or more feature values ​​being associated with feature correlation values ​​that deviate from a predefined reference and / or exceed or fall below a predefined threshold.

[0218] Example 25. The method according to Example 24,

[0219] Wherein the predefined reference and / or the predefined threshold are determined based on a previously determined feature correlation vector.

[0220] Example 26. The method according to Example 25,

[0221] The predefined reference and / or the predefined threshold are / is fixedly predefined.

[0222] Example 27. The method according to any one of Example 25,

[0223] The predefined reference and / or the predefined threshold are obtained from a central server (139).

[0224] Example 28. The method according to any one of Examples 24 to 27, further comprising:

[0225] - selecting the predefined reference and / or the predefined threshold value according to the true value gesture class of the gesture.

[0226] Example 29. The method according to Example 28, further comprising:

[0227] - controlling the user interface to obtain a user input associated with the user output, the user input being indicative of the truth-valued gesture category of the gesture.

[0228] Example 30. A method according to any one of the preceding examples,

[0229] The at least one feature vector includes feature values ​​in one or more of the following dimensions: distance of the gesture object; speed of the gesture object; angular orientation of the gesture object; azimuth of the gesture object; and elevation of the gesture object.

[0230] Example 31. A method according to any one of the preceding examples,

[0231] The at least one feature vector comprises a time series of a plurality of feature vectors.

[0232] Example 32. The method according to any of the preceding examples, further comprising:

[0233] - Prior to said inference, obtaining (3110) a downlink message from a central server (139) indicating the weights of said machine learning model.

[0234] Example 33. The method according to Example 32,

[0235] The processor is part of the car's control unit.

[0236] The downlink message is obtained during an over-the-air update process of the vehicle control unit.

[0237] Example 34. A method according to any one of the preceding examples,

[0238] wherein the at least one feature vector comprises a time series of a plurality of feature vectors,

[0239] wherein the at least one feature correlation vector comprises a global feature correlation vector,

[0240] The feature correlation values ​​of the global feature correlation vector indicate the dependency of the gesture category prediction on corresponding feature values ​​of the plurality of feature vectors.

[0241] Example 35. A method according to any one of the preceding examples,

[0242] wherein the at least one feature vector comprises a time series of a plurality of feature vectors,

[0243] The at least one feature correlation vector comprises a plurality of local feature correlation vectors, each local feature correlation vector of the plurality of local feature correlation vectors being associated with a corresponding feature vector of the plurality of feature vectors.

[0244] Example 36. A method of operating a central server (139), the method comprising:

[0245] - obtaining, from each of a plurality of edge-deployed processors, a respective uplink message, each of the uplink messages indicating a respective weight of a machine learning model used at the edge-deployed processor to infer a gesture class prediction from a feature vector representing measurement data obtained from a depth sensor,

[0246] - combining the weights to determine updated weights for the machine learning model, and

[0247] -Providing a corresponding downlink message indicating the updated weights of the machine learning model to at least one edge deployment processor of the plurality of edge deployment processors.

[0248] Example 37. The method according to Example 36,

[0249] Each of the plurality of edge deployment processors operates according to a method described in any one of Examples 1 to 35.

[0250] Example 38. A processor configured to perform a method according to any one of Examples 1 to 35.

[0251] Example 39. A central server configured to perform the method according to Example 36.

[0252] Example 40. A system comprising the processor of Example 38 and the central server of Example 39.

[0253] Although the invention has been shown and described with respect to certain preferred embodiments, equivalents and modifications will occur to others skilled in the art upon the reading and understanding of the specification. The present invention includes all such equivalents and modifications, and is limited only by the scope of the appended claims.

[0254] For illustration, various examples have been disclosed above, according to which the feature relevance value and the feature relevance vector are determined using the Shapley method. In other examples, other XAI methods, such as LIME, may be employed.

[0255] To further illustrate, various examples are disclosed above in which radar sensors are used to make measurements that enable inference of gesture class predictions. Similarly, other types of depth sensors may be used. For example, examples include optical ToF sensors (such as ToF cameras, ToF LIDAR sensors, or direct or indirect ToF sensors). Optical ToF sensors (which are similar to radar sensors that use radio waves) can provide information about the distance to an object, i.e., a feature associated with "distance". Other types of depth sensors include stereo cameras and ultrasonic sensors.

[0256] To further illustrate, even other sensor types than depth sensors may be relied upon. In particular, sensors capable of determining human-understandable features may be used, whereby feature relevance values ​​associated with corresponding feature values ​​may be indicated to a user, e.g. as part of a retraining process for retraining an ML model and / or as part of a guided and interactive user training process. One example is a capacitive sensor, which is, e.g. integrated into a display, and may provide features such as “touch pressure” and “touch position”. Furthermore, such capacitive sensors are often used to detect gestures performed on top of the display, e.g. in continuous interaction with a graphical user interface.

Claims

1. A method of operating an edge deployment processor (62), the method comprising: acquiring (3010) at least one feature vector encoding measurement data (64) provided by a depth sensor (70) for a gesture (501-510) performed by a user, inferring (3015) a gesture class prediction associated with the gesture based on the at least one feature vector using a machine learning model (121), determining (3020) at least one feature relevance vector for the at least one feature vector, each of the at least one feature relevance vector comprising feature relevance values, each of the feature relevance values ​​indicating a dependency of the gesture class prediction on a corresponding one or more feature values ​​of the at least one feature vector, determining (3035) a user output associated with the gesture based on the at least one feature correlation vector, and A user interface is controlled (3040) to provide the user output to the user.

2. The method according to claim 1, further comprising: classifying the gesture as anomalous or non-anomalous, Wherein the user output is selectively provided to the user in response to classifying the gesture as abnormal.

3. The method according to claim 2, Wherein classifying the gesture as abnormal or non-abnormal is based on the at least one feature correlation vector.

4. The method according to claim 2 or 3, Wherein classifying the gesture as abnormal or non-abnormal is based on an uncertainty metric associated with inferring the gesture category prediction using the machine learning model.

5. The method according to any one of claims 2 to 4, Wherein classifying the gesture as abnormal or non-abnormal is based on a distance of the at least one feature vector from one or more predefined reference feature vectors.

6. The method according to any one of claims 2 to 5, Wherein classifying the gesture as abnormal or non-abnormal is based on a threshold comparison between each of the feature values ​​of the at least one feature vector and a corresponding threshold.

7. The method according to any one of the preceding claims, Wherein the user output is provided to the user as part of a retraining process for populating a training dataset to retrain the machine learning model.

8. The method according to any one of the preceding claims, further comprising: controlling the user interface to obtain a user input associated with the user output, selectively including the at least one feature vector in a training data set based on the user input, and The machine learning model is retrained based on the training data set to obtain the machine learning model.

9. The method according to claim 8, further comprising: An uplink message is provided (3130) to a central server (139), the uplink message indicating the weights (122) of the machine learning model upon completion of the retraining.

10. The method according to any one of the preceding claims, Wherein the user output is indicative of one or more feature values ​​of the at least one feature vector, the one or more feature values ​​being associated with feature correlation values ​​that deviate from a predefined reference and / or exceed or fall below a predefined threshold.

11. The method according to claim 10, Wherein the predefined reference and / or the predefined threshold are determined based on a previously determined feature correlation vector.

12. The method according to any one of the preceding claims, The at least one feature vector includes feature values ​​in one or more of the following dimensions: distance of the gesture object; speed of the gesture object; angular orientation of the gesture object; azimuth of the gesture object; and elevation of the gesture object.

13. The method according to any one of the preceding claims, further comprising: Prior to the inference, a downlink message indicating the weights of the machine learning model is obtained (3110) from a central server (139).

14. A method of operating a central server (139), the method comprising: obtaining, from each of a plurality of edge-deployed processors, a respective uplink message, each of the uplink messages indicating a respective weight of a machine learning model used at the edge-deployed processor to infer a gesture class prediction based on a feature vector representing measurement data obtained from a depth sensor, combining the weights to determine updated weights for the machine learning model, and A corresponding downlink message indicating the updated weights of the machine learning model is provided to at least one edge deployment processor of the plurality of edge deployment processors.

15. The method according to claim 14, Each edge deployment processor of the plurality of edge deployment processors operates according to the method of any one of claims 1 to 13.

Images