Radar data collection for gesture recognition training

Abstract

The invention relates to radar data collection for gesture recognition training. Various embodiments of the present disclosure relate to collecting training data for a neural network, in particular, to collecting radar data for training a neural network to perform gesture detection via a radar. In one example embodiment, a technique (200) is provided for acquiring radar data for training a neural network to perform gesture recognition via a radar. The technique (200) first includes identifying radar data collected during a time period between a first cue (901) and a second cue (902). Next, the technique (200) includes identifying a subset of the radar data associated with a gesture based at least on Doppler processing (903). Finally, the technique (200) includes marking the subset (903) of the radar data as the gesture.

Description

[0001] Cross Reference to Related Applications

[0002] This application is related to Indian Provisional Patent Application No. 202441029307, filed on April 10, 2024, entitled “Automated labeling for mmWave radar gesture training,” which claims priority benefit of the Indian Provisional Patent Application, which is hereby incorporated by reference in its entirety. TECHNICAL FIELD

[0003] The present disclosure relates generally to computing hardware and software, and in particular, to gathering training data for neural networks. BACKGROUND

[0004] Gesture recognition is a field of research in computer vision applications that primarily studies interpreting body language using sensors (i.e., cameras, radars, etc.). For example, in the context of machine learning applications, a neural network can be trained to recognize a gesture performed by a user and, in response, perform a task associated with the gesture.

[0005] Generally, a neural network is trained to perform a task using a large amount of training data. For example, to train a network to perform gesture recognition, the network is fed training data associated with one or more gestures so that the network can learn how to accurately identify a particular gesture based on the training data fed to it. Thus, it is crucial to train the network based on high-quality training data because the accuracy of a neural network depends on the data it is trained on.

[0006] Currently, there are various techniques for obtaining training data related to gesture recognition. In one application (a camera-based gesture recognition system), a camera is utilized to capture video data of a user performing a gesture. Once captured, the user can then label the video data with the particular gesture performed and supply the labeled data as a training data set. The training data set represents data that can be used to train a neural network. In another application (a radar-based gesture recognition system), a radar device is utilized to capture radar data while a camera captures video data. After the required data is captured, the user can then synchronize the radar data and the video data, label the synchronized data, and supply the labeled data as a training data set. It should be noted that in the radar-based gesture recognition system, the video data collected in parallel with the radar data is intended to assist the user in accurately labeling the radar data.

[0007] The problem is that current methods for collecting training data for gesture recognition rely on users to manually label, and possibly synchronize, the collected data. Thus, current methods for collecting training data for gesture recognition are time consuming and prone to user error. Furthermore, neural networks trained to perform gesture recognition with user-labeled training data can be inaccurate. SUMMARY

[0008] Disclosed herein is a technology including systems, methods, and apparatuses for collecting radar data for training a neural network to perform gesture recognition.

[0009] In various embodiments, a technology for collecting radar data for training a neural network to perform gesture recognition via radar is provided. In one example embodiment, the technology first includes identifying radar data collected during a time period between a first prompt and a second prompt. Next, the technology includes identifying a subset of the radar data associated with a gesture based on at least Doppler processing. Finally, the technology includes labeling the subset of the radar data as the gesture.

[0010] This summary is provided to introduce a selection of concepts, in a simplified form, that are further described below in the detailed description. It is to be understood that this summary is not intended to identify key or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter. BRIEF DESCRIPTION OF DRAWINGS

[0011] Many aspects of the disclosure can be better understood with reference to the following drawings. The components in the drawings are not necessarily to scale, emphasis instead being placed on clearly illustrating the principles of the present disclosure. Furthermore, in the drawings, like reference numerals designate corresponding parts throughout the several views. While several embodiments are described in connection with these drawings, the disclosure is not limited to the embodiments disclosed herein. On the contrary, it is intended to cover all alternatives, modifications, and equivalents.

[0012] Figure 1 An operating environment in an embodiment is illustrated.

[0013] Figure 2 A labeling process in an embodiment is illustrated.

[0014] Figure 3 A system in an embodiment is illustrated.

[0015] Figure 4 An operating sequence in an embodiment is illustrated.

[0016] Figure 5A An operating environment in an embodiment is illustrated.

[0017] Figure 5B Sequences of operations in embodiments are illustrated.

[0018] Figure 6 User environments in embodiments are illustrated.

[0019] Figure 7 Collection processes in embodiments are illustrated.

[0020] Figure 8 Another tagging process in embodiments is illustrated.

[0021] Figure 9 Operational scenarios in embodiments are illustrated.

[0022] Figure 10 Operational scenarios in embodiments are illustrated.

[0023] Figure 11 Operational scenarios in embodiments are illustrated.

[0024] Figure 12 Computing systems suitable for implementing various operational environments, architectures, processes, scenarios, and sequences discussed below with respect to other figures are illustrated. DETAILED DESCRIPTION

[0025] Systems, methods, and apparatuses for collecting training data for a neural network to be trained to perform gesture recognition via radar are disclosed herein. Training data represents data used to train a neural network to perform a specified task. Generally, a network needs a large amount of training data to learn how to accurately perform a task. Thus, the accuracy of a neural network depends on the quality of its training data.

[0026] Existing techniques for collecting training data related to gesture recognition rely on a user manually labeling data and supplying the labeled data to a training data set of a network. For example, a camera-based gesture recognition system can require a user to label video data of one or more gestures and supply the labeled data to a training data set of a network. Alternatively, other systems such as a radar-based gesture recognition system require a user to synchronize radar data and video data of one or more gestures, label the synchronized data, and supply the labeled data to a training data set of a network. The problem is that these systems require a user to invest a large amount of manual work. Furthermore, these systems are prone to user errors, which can result in inaccurate gesture recognition when deploying the network. In contrast, a new technique for collecting training data related to gesture recognition is disclosed herein that relies only on radar data and no longer requires a user to manually label collected data.

[0027] In one example embodiment, a computer-readable medium having stored thereon executable instructions related to collecting training data for a neural network is provided. The instructions are configured to be executed by processing circuitry, such that when executed the instructions cause the processing circuitry to collect and label radar data for training a neural network to perform gesture recognition via radar.

[0028] In an embodiment, the program instructions first cause the processing circuitry to identify radar data associated with a gesture based on issuance of a first cue and a second cue. The first cue and the second cue can represent an audio cue, a visual cue, or another similar sensory cue. In an embodiment, the first cue represents an instruction to initiate a gesture collection period, and the second cue represents an instruction to terminate the gesture collection period. The gesture collection period represents a period of time that allows a user to perform a gesture and allows the processing circuitry to collect radar data of the user performing the gesture.

[0029] In an embodiment, the user is expected to perform and complete a gesture at any time between the first cue and the second cue. For example, the user can be expected to wave a hand from left to right within a period of time between the first cue and the second cue. Any non-gesture related movement that involves the hand retracting to prepare the hand for a subsequent gesture should be performed outside the interval between the first cue and the second cue. Thus, the radar data collected between the first cue and the second cue represents radar data associated with a gesture.

[0030] In an embodiment, prior to identifying the radar data collected between the two cues, the instructions first cause the processing circuitry to issue the first cue and the second cue during a data collection period. The data collection period represents a period of time that allows the processing circuitry to collect radar data. In an embodiment, the user provides a time delay for issuance of the two cues, and during the data collection period, the instructions cause the processing circuitry to output the first cue and the second cue after the user specified period of time. After the data collection period terminates, the instructions can then cause the processing circuitry to identify radar data associated with a gesture based on issuance of the first cue and the second cue.

[0031] In another embodiment, the instructions cause the processing circuitry to identify radar data associated with a gesture based on a signal generated by a user input device. For example, the user input device can represent a cell phone, a tablet, a computer, or another similar device, including one or more sensors configured to collect audio signals, video signals, haptic signals, or another similar signal (i.e., microphones, cameras, touchscreens, etc.) of user input. During the data collection period, the user can provide the first cue and the second cue via the user input device, and after the data collection period terminates, the processing circuitry can identify radar data collected between the first cue and the second cue based on the signal received from the user input device.

[0032] Next, the instructions cause the processing circuitry to perform Doppler processing on the collected radar data to identify a subset of the radar data consisting of data directly associated with the user performing the gesture. Doppler processing describes a method for capturing the relative velocity between a radar device and a moving target. In the context of the present disclosure, the radar device is stationary. Thus, the processing circuitry can perform Doppler processing to identify radar data associated with the moving target. For example, the processing circuitry can perform Doppler processing on the radar data collected between the first cue and the second cue to identify a subset of the radar data directly associated with the user performing the gesture (i.e., gesture data).

[0033] In embodiments, the instructions further cause the processing circuitry to identify radar data not associated with the gesture (i.e., non-gesture data). Non-gesture data represents any radar data in which the user did not perform the gesture. For example, the instructions can cause the processing circuitry to identify radar data collected within the data collection period but outside the range of the first cue and the second cue. This means that the processing circuitry can identify a second set of radar data collected between the start of the data collection period and the first cue, and a third set of radar data collected between the second cue and the end of the data collection period.

[0034] Finally, the instructions cause the processing circuitry to label the subset of the radar data as gesture. The instructions further cause the processing circuitry to label the second set of radar data and the third set of radar data as non-gesture. In embodiments, the processing circuitry is coupled to a memory configured to store the labeled data for training the neural network. For example, the processing circuitry can store the labeled gesture data in a first section of the memory and the labeled non-gesture data in a second section of the memory. In embodiments, the instructions cause the processing circuitry to collect multiple iterations of the labeled gesture data and the labeled non-gesture data.

[0035] Advantageously, the proposed technology generates a large amount of training data related to gesture movements and no longer requires the user to manually label the collected radar data. Thus, the proposed solution is cheaper than applications that require a camera to label the training data of the user performing the gesture. Furthermore, the proposed solution generates a more accurate neural network than other applications that require the user to manually label the training data set.

[0036] Turning now to the figures, Figure 1 An operating environment 100 in an embodiment is illustrated. The operating environment 100 represents an example environment that can be configured to gather data for training a neural network to perform gesture recognition via radar. The operating environment 100 includes, but is not limited to, a collection engine 103, a training engine 107, and an inference engine 109.

[0037] The collection engine 103 represents software, hardware, firmware, or a combination thereof configured to collect and label data for training a neural network to recognize gestures. For example, the collection engine 103 can represent a laptop computer or a computer or the like configured to collect and label radar data for training a neural network to perform gesture recognition via radar. The input to the collection engine 103 includes the raw data 101, and the output of the collection engine 103 includes the labeled data 105.

[0038] The raw data 101 represents unlabeled radar data collected by a radar device for training a neural network. For example, the raw data 101 can represent ADC samples collected by the radar device of the collection engine 103. The raw data 101 includes unlabeled radar data associated with gesture movements and unlabeled radar data associated with non-gesture movements. In implementations, the raw data 101 represents user-generated data. For example, during a data collection period, the radar device can collect raw data 101 of a user performing various gesture movements and non-gesture movements and provide the raw data 101 as input to the collection engine 103.

[0039] In implementations, the collection engine 103 is configured to determine which subset of the raw data 101 is associated with gesture movements and which subset of the raw data 101 is associated with non-gesture movements. For example, the collection engine 103 can utilize Doppler processing techniques to identify a subset of the raw data 101 associated with gesture movements, in turn identifying a subset of the raw data 101 associated with non-gesture movements, discussed in greater detail below with reference to Figure 2 In implementations, the collection engine 103 is configured to determine which subset of the raw data 101 is associated with gesture movements and which subset of the raw data 101 is associated with non-gesture movements. For example, the collection engine 103 can utilize Doppler processing techniques to identify a subset of the raw data 101 associated with gesture movements, in turn identifying a subset of the raw data 101 associated with non-gesture movements, discussed in greater detail below with reference to

[0040] The labeled data 105 represents labeled radar data for training a neural network. The labeled data 105 includes labeled radar data associated with gesture movements (i.e., gesture data) and labeled radar data associated with non-gesture movements (i.e., non-gesture data). In implementations, the gesture data of the labeled data 105 includes multiple iterations of labeled radar data representing multiple different gestures. For example, the gesture data can include multiple iterations of radar data associated with a person waving their hand from left to right, a person pinching their thumb and index finger together, and other similar user-generated gestures. In implementations, after acquiring, analyzing, and labeling the required data, the collection engine 103 outputs the labeled data 105 to the training engine 107.

[0041] The training engine 107 represents software, hardware, firmware, or a combination thereof configured to train a neural network to perform a specified task. For example, the training engine 107 can train a network or machine learning algorithm, such as a convolutional neural network (CNN), an artificial neural network (ANN), a recurrent neural network (RNN), or another type of deep neural network (DNN), to perform gesture recognition based on radar data. The input to the training engine 107 includes the labeled data 105, and the output of the training engine 107 includes a trained neural network configured to perform gesture recognition via radar.

[0042] In embodiments, the training engine 107 utilizes the labeled data 105 to train the network to perform a task in response to recognizing a gesture. For example, in the context of an electric vehicle (EV), the training engine 107 can utilize gesture data of the labeled data 105 to train the network to open the trunk of the car when the network recognizes a user kicking the foot. In embodiments, the training engine 107 also utilizes the labeled data 105 to train the network to remain in an off state when no gesture is recognized. For example, the training engine 107 can utilize non-gesture data of the labeled data 105 to train the network to continue monitoring for gestures when no gesture is recognized. After training the network, the training engine 107 outputs the trained neural network to the inference engine 109 for deployment.

[0043] The inference engine 109 represents software, hardware, firmware, or a combination thereof configured to employ a trained neural network. For example, the inference engine 109 can represent a processor in an EV configured to perform gesture recognition via radar. Additional example details related to inference engines can be found in the following commonly-assigned patents: U.S. Patent No. 9,817,109, filed February 27, 2015, entitled “Gesture Recognition Using Frequency Modulated Continuous Wave (FMCW) Radar With Low Angle Resolution,” U.S. Patent No. 11,204,647, filed April 13, 2018, entitled “System and Method for Radar Gesture Recognition,” U.S. Patent No. 11,456,713, filed April 17, 2020, entitled “Low Power Node of Operation for mmWave Radar,” and U.S. Patent Application Publication No. 2023 / 0408120, filed July 29, 2022, entitled “Room Boundary Detection,” all of which are incorporated by reference in their entirety. The input to the inference engine 109 includes the output of the training engine 107 and the sensor data 111, and the output of the inference engine 109 includes the gesture classification 113.

[0044] The sensor data 111 represents input data for the trained neural network. Thus, the sensor data 111 represents radar data associated with an environment. In implementations, the inference engine 109 receives the sensor data 111, and in response, executes the trained neural network to determine whether to perform a gesture. If a gesture is identified, the inference engine 109 performs an action associated with the identified gesture. Alternatively, if a gesture is not identified, the inference engine 109 continues to monitor for gesture movement. In both cases, the inference engine 109 outputs the gesture classification 113. The gesture classification 113 represents an identification of a performed gesture. For example, the gesture classification 113 can indicate that a user waved their hand from left to right. Alternatively, the gesture classification 113 can indicate that no gesture was performed.

[0045] In a brief operational example, during a data collection period, the collection engine 103 collects radar data (i.e., raw data 101) of a user (or multiple users) performing both gesture movements and non-gesture movements. After the data collection period terminates, the collection engine 103 analyzes the raw data 101 to determine which subset of the radar data is associated with gesture movements and which subset of the radar data is associated with non-gesture movements. For example, the collection engine 103 can utilize Doppler processing techniques on the raw data 101 to identify radar data associated with a user performing gesture movements, and in turn, identify radar data associated with a user performing non-gesture movements.

[0046] Next, the collection engine 103 labels the subset of the raw data 101 as gesture data or non-gesture data and outputs the labeled data 105 to the training engine 107. The training engine 107 trains a neural network with the labeled data 105 to perform gesture recognition based on radar data. For example, the training engine 107 can train the network to perform a task based on an identified gesture. The training engine 107 can further train the network to continue monitoring for gesture movements when no gesture is identified. Once the network is trained, the training engine 107 outputs the trained network to the inference engine 109. The inference engine 109 deploys the trained network and begins collecting sensor data 111. The trained network analyzes the received sensor data and, in response, outputs a gesture classification 113. The gesture classification 113 can indicate that a gesture has been performed or that no gesture has been performed.

[0047] Figure 2 A labeling process 200 in an implementation is illustrated. The labeling process 200 represents a process for generating labeled data for training a neural network to perform gesture recognition via radar. The labeling process 200 can be implemented in the context of program instructions that, when executed by a suitable computing system, direct processing circuitry of the computing system to operate as follows, with incidental reference to the steps in Figure 2 For explanatory purposes, the labeling process 200 will be explained with reference to elements of Figure 1 This is not meant to limit the application of the labeling process 200, but to provide an example.

[0048] First, the collection engine 103 identifies (e.g., selects) radar data from the raw data 101 collected during a time period between a first prompt and a second prompt (step 201). In embodiments, the raw data 101 is collected during a data collection period. The data collection period describes a time period that allows the collection engine 103 to collect radar data for training a neural network to perform gesture recognition via radar. In embodiments, when radar data is collected during the data collection period, the collection engine 103 issues a first prompt instructing a user to perform a gesture, and after a period of time, issues a second prompt that terminates the period allowing the user to perform the gesture. The first and second prompts can represent audio prompts, visual prompts, or another similar sensory prompt that provides instructions to the user. Radar data collected between the first and second prompts represents unlabeled gesture data.

[0049] In another embodiment, the user issues the first and second prompts via a user input device. The user input device can represent a cell phone, tablet, or computer, including one or more sensors configured to collect input from a user. The one or more sensors of the user input device can include a microphone, camera, or touch device, such as a touchscreen, touchpad, keyboard, keypad, button, remote control, or another similar touch device. During the data collection period, the user can supply the first and second prompts via the sensors of the user input device. After the data collection period terminates or during the data collection period, the collection engine 103 can identify radar data collected between the first and second prompts based on signals provided by the user input device.

[0050] Next, the collection engine 103 identifies (e.g., selects) a subset of radar data from the radar data collected between the first and second prompts based at least on Doppler processing techniques (step 203). The subset of radar data represents radar data that is directly associated with gesture movement. For example, if the duration of time between the first and second prompts equals five seconds, and the amount of time that a gesture is performed equals two seconds, then the radar data collected between the first and second prompts represents radar data collected during the five-second duration between the two prompts, and the subset of radar data represents radar data collected during the two seconds when the gesture was performed.

[0051] In an embodiment, to identify the subset of radar data directly associated with the gesture movement, the collection engine 103 performs Doppler processing on the radar data collected between the first cue and the second cue and selects the subset of radar data based on the Doppler processing. Doppler processing describes a technique for determining the relative velocity between a radar device and a moving target. In the context of the present disclosure, Doppler processing can be performed on the radar data collected between the first cue and the second cue to identify the subset of radar data associated with the user performing the gesture. In an embodiment, the collection engine 103 performs Doppler processing on the radar data collected between the first cue and the second cue to determine a Doppler metric. The Doppler metric represents a measure that captures the motion content of a moving target over time.

[0052] After identifying the subset of radar data directly associated with the user performing the gesture, the collection engine 103 labels the subset of radar data as indicative of the performed gesture (step 205). For example, if the performed gesture comprises the user performing a “zoom-in” gesture (by bringing the thumb and index finger together), the collection engine 103 can label the subset of radar data as “zoom-in”. In an embodiment, the collection engine 103 is further configured to identify and label radar data associated with non-gesture movement. For example, the collection engine 103 can identify radar data collected between the start of the data collection period and the issuance of the first cue and label the radar data as non-gesture data. The collection engine 103 can further identify radar data collected between the issuance of the second cue and the termination of the data collection period and label the radar data as non-gesture data.

[0053] In an embodiment, the collection engine 103 performs the labeling process 200 multiple times to collect multiple iterations of gesture data and non-gesture data. Advantageously, collecting multiple iterations of data improves the network’s ability to distinguish between user-performed gesture movement and non-gesture movement.

[0054] Figure 3 A system 300 in an embodiment is illustrated. The system 300 represents a data collection system configured to collect radar data for training a neural network to perform gesture recognition via radar. For example, the system 300 can represent the collection engine 103 of Figure 1 The system 300 includes, but is not limited to, a user 301, a radar device 303, and a host device 311.

[0055] User 301 represents a person that performs both gesture movements and non-gesture movements. For example, a gesture movement can represent user 301 moving a hand from a first position to a second position. Alternatively, a non-gesture movement can represent user 301 remaining stationary, user 301 retracting a hand after performing a gesture, or user 301 preparing a hand to perform a subsequent gesture. In embodiments, user 301 performs gesture movements and non-gesture movements during a data collection period. The data collection period describes a period of time that allows radar device 303 to collect radar data.

[0056] Radar device 303 represents a device configured to collect radar data related to gesture movements and non-gesture movements. In embodiments, radar device 303 is also configured to process the collected data to identify radar data that captures a time frame when a gesture is performed. Radar device 303 includes radar processing circuitry 305, transceiver antenna 307, and receiver antenna 309.

[0057] Radar processing circuitry 305 represents circuitry configured to collect and process radar data. For example, radar processing circuitry 305 can represent a microcontroller unit (MCU), a central processing unit (CPU), an application specific integrated circuit (ASIC), or another similar processing device configured to collect and process radar data related to gesture movements and non-gesture movements. In embodiments, radar processing circuitry 305 includes an analog front end. For example, radar processing circuitry 305 can include power amplifiers, low noise amplifiers, analog-to-digital converters (ADCs), filters, and other similar processing elements.

[0058] In embodiments, radar processing circuitry 305 instructs transceiver antenna 307 and receiver antenna 309 to collect radar data during a data collection period. Transceiver antenna 307 and receiver antenna 309 represent antennas configured to gather radar data of an environment. For example, during the data collection period, radar processing circuitry 305 can instruct transceiver antenna 307 to transmit radar signals (i.e., TX signals) to user 301 and instruct receiver antenna 309 to collect radar signals (i.e., RX signals) reflected back to radar device 303. The radar data collected by transceiver antenna 307 and receiver antenna 309 represents unprocessed radar data associated with user 301 performing gesture movements and non-gesture movements.

[0059] In embodiments, the receiver antenna 309 is configured to output collected radar data to an analog front end of the radar processing circuitry 305. In response, the analog front end of the radar processing circuitry 305 is configured to generate ADC samples based on the collected radar data. The ADC samples generated by the analog front end of the radar processing circuitry 305 represent processed radar data associated with gesture and non-gesture movements performed by the user 301. In embodiments, the radar processing circuitry 305 is configured to process the ADC samples to generate unlabeled radar data for the host device 311. For example, the radar processing circuitry 305 can perform various fast Fourier transforms (FFTs) on the collected ADC samples to generate heat maps (e.g., range-angle heat maps) that can be supplied as unlabeled radar data to the host device 311. The radar processing circuitry 305 can also extract various temporal metrics from the generated heat maps and supply the extracted data as unlabeled radar data to the host device 311.

[0060] In embodiments, the radar processing circuitry 305 is also configured to perform Doppler processing on the collected ADC samples to determine Doppler metrics associated with gestures performed by the user 301. The Doppler metrics represent a measure of the motion content (i.e., gesture movements) of a moving target (i.e., the user 301) over time. For example, the Doppler metrics can represent heat maps, time series data, or another similar metric. In embodiments, the radar processing circuitry 305 is configured to output the unlabeled radar data and associated Doppler metrics to the host device 311. In response, the host device 311 is configured to label the radar data as indicative of a gesture movement or a non-gesture movement based on the associated Doppler metrics.

[0061] The host device 311 represents a device configured to manage the collection of radar data by the radar device 303. For example, the host device 311 can represent a CPU, MCU, ASIC, or another similar device. In embodiments, the host device 311 represents a device configured to initiate a data collection period. For example, the host device 311 can output an instruction such that the instruction instructs the radar device 303 to begin collecting radar data. In embodiments, during the data collection period, the host device 311 is configured to output an instruction to the user 301. For example, the host device 311 can output a first prompt instructing the user 301 to perform a gesture and, after a period of time, output a second prompt indicating to the user 301 that the time period for performing a gesture has terminated. The first and second prompts can represent audio prompts, visual prompts, or another similar sensory prompt providing instructions to the user 301.

[0062] In implementations, the host device 311 includes a user interface configured to collect configuration information related to the data collection process. For example, the host device 311 can represent a laptop computer that includes a user interface configured to collect various timing parameters from the user 301, including a duration of the data collection period, a time delay for issuance of the first prompt (after the start of the data collection period), a time delay for issuance of the second prompt (after issuance of the first prompt), and a time delay for issuance of subsequent iterations of the first prompt (after issuance of the second prompt).

[0063] In implementations, the host device 311 further represents a device configured to label radar data for training a neural network to perform gesture recognition. For example, the host device 311 can receive unlabeled radar data and associated Doppler metrics from the radar device 303, and in response, label the radar data based on the associated Doppler metrics. In implementations, the host device 311 executes the labeling application 313 to label the collected radar data.

[0064] The labeling application 313 represents software (i.e., the labeling process 200) that, when executed, causes the host device 311 to label radar data associated with gesture movements as gesture data and radar data associated with non-gesture movements as non-gesture data. In implementations, the user 301 can provide configuration information to a user interface of the host device 311 for configuring the labeling application 313. For example, the user 301 can specify, via the user interface of the host device 311, a type of gesture to be performed, a number of times the gesture is to be performed, and a location in associated storage for storing the labeled gesture data and non-gesture data.

[0065] Figure 4 An operational sequence 400 in an implementation is illustrated. The operational sequence 400 represents a sequence for acquiring data for training a neural network to perform gesture recognition with respect to the elements of Figure 3 Thus, the operational sequence 400 includes the radar device 303 and the host device 311.

[0066] First, the user 301 provides configuration information to a user interface of the host device 311. For example, the user 301 can provide a type of gesture to be performed, a number of times the gesture is to be performed, a time delay for issuance of a prompt, a duration of a data collection period, and a location for storing labeled radar data. In some instances, some or all of the configuration information is preprogrammed on the host device 311.

[0067] Next, the host device 311 instructs the radar device 303 to initiate a data collection period. In an implementation, the host device 311 instructs the radar device 303 to initiate the data collection period after receiving the configuration information from the user 301. In another implementation, the user 301 instructs the host device 311 to initiate the data collection period via a user interface of the host device 311. In either case, after initiation of the data collection period, the radar device 303 can begin collecting radar data of the user 301. It should be noted that data collected prior to issuance of the first cue represents radar data associated with non-gesture movements.

[0068] Next, the host device 311 issues a first cue. In an implementation, the host device 311 issues the first cue based on the configuration information received from the user 301. For example, the user 301 can provide a time delay for issuance of the first cue, such that the time delay represents a duration of time that the host device 311 must wait after initiation of the data collection period to issue the first cue. In another implementation, the host device 311 issues the first cue based on a signal received from the user 301. For example, the host device 311 can include a microphone, camera, or touch device, such as a touch screen, keyboard, or remote control, configured to receive a signal, such as the first cue, from the user 301. As just one example, the host device 311 can be configured to present a graphical user interface on a touch screen that instructs the user to touch the screen (or a button on the screen) before starting the gesture, and then touch the screen (or button) again after ending the gesture.

[0069] The first cue can include an audio cue, a video cue, a visual cue, or any combination of these cues. The audio cue can include a chime or a voice that instructs the user 301 to perform the gesture. Additionally or alternatively, the video cue can be displayed on a screen and include text and / or a video of a person performing the gesture that instructs the user 301 to perform the gesture. The visual cue can include a light, for example, accompanied by an audio cue that instructs the user 301 to perform the gesture.

[0070] The first cue can convey information to the user 301. The information conveyed by the first cue can include when, where, and / or how to perform the gesture. For example, the first cue can instruct the user 301 to start the gesture, and the second cue can instruct the user 301 to stop the gesture. The first cue can instruct the user 301 to perform the gesture within a field of view of the radar device 303. Additionally or alternatively, the first cue can instruct the user 301 of a type of gesture to perform.

[0071] After the host device 311 issues the first cue, the user 301 can perform the indicated gesture. For example, if the user 301 configures the tagging application 313 to tag gesture data for a hand wave from left to right, after the issuance of the first cue, the user 301 can wave a hand from right to left.

[0072] Next, the host device 311 issues a second cue. In an implementation, the host device 311 issues the second cue based on configuration information received from the user 301. For example, the user 301 can provide a time delay for issuance of the second cue, such that the time delay represents a duration of time that the host device 311 must wait after issuance of the first cue to issue the second cue. In another implementation, the host device 311 issues the second cue based on a signal received from the user 301. It is noted that data collected between the first cue and the second cue represents radar data associated with gesture movement.

[0073] After the host device 311 issues the second cue, the radar device 303 can terminate the data collection period. In an implementation, the radar device 303 terminates the data collection period based on configuration information provided by the user 301. For example, the user 301 can provide a duration of time for the data collection period, such that the duration describes a period of time that the transceiver antenna 307 and the receiver antenna 309 are permitted to collect radar data. In another implementation, the radar device 303 terminates the data collection period based on instructions from the user 301. For example, the user 301 can instruct the host device 311 to terminate collection of radar data by the radar device 303. It is noted that data collected after issuance of the second cue but before termination of the data collection period represents radar data associated with non-gesture movement.

[0074] Next, the radar processing circuitry 305 processes the collected radar data to compute a Doppler metric associated with the user 301 performing the gesture. The Doppler metric represents a metric that captures the motion content of a moving target over time (i.e., heat map, time series data, etc.). For example, the Doppler metric can represent a metric that can be used to identify a time frame when the user 301 waved a hand from right to left.

[0075] In an implementation, to determine Doppler metrics associated with the user 301 waving a hand from right to left, the radar processing circuitry 305 performs various Fast Fourier Transforms (FFTs) on the collected ADC samples between the first prompt and the second prompt. For example, the radar processing circuitry 305 can compute a range FFT and a Doppler FFT, and thus generate a range-Doppler heatmap. In another implementation, to determine Doppler metrics associated with the user 301 waving a hand from right to left, the radar processing circuitry 305 further processes the computed heatmap to extract time series metrics. For example, the radar processing circuitry 305 can extract Doppler mean values from the computed heatmap.

[0076] In an implementation, the radar processing circuitry 305 is also configured to process the collected radar data to generate unlabeled radar data associated with the user 301 performing gesture and non-gesture movements. For example, the radar processing circuitry 305 can perform range-FFT, Doppler-FFT, and angle-FFT on the collected ADC samples to generate unlabeled radar data (e.g., range-Doppler heatmaps and range-angle heatmaps) of the user 301 performing gesture and non-gesture movements. In another example, the radar processing circuitry 305 can extract metrics (e.g., Doppler mean values, azimuth angle weighted mean values, elevation angle weighted mean values, and Doppler azimuth angle correlations) from the heatmaps to generate unlabeled radar data of the user 301 performing gesture and non-gesture movements.

[0077] It should be noted that in some implementations, the radar processing circuitry 305 can instead be configured to output the ADC samples to the host device 311, and in response, the host device 311 can perform the required processing (e.g., range-FFT, Doppler-FFT, angle-FFT, or metric extraction) on the ADC samples to generate the unlabeled radar data. Further, it should be noted that the host device 311 can instead be configured to compute the Doppler metrics. For example, the radar processing circuitry 305 can output the collected ADC samples to the host device 311, and in response, the host device 311 can process the collected ADC samples (e.g., range-FFT, Doppler-FFT, or metric extraction) to compute the Doppler metrics.

[0078] Next, after computing the Doppler metrics, the radar device 303 outputs the computed Doppler metrics and the unlabeled radar data to the host device 311. In response, the host device 311 executes the labeling application 313 to label the radar data. In an implementation, the labeling application 313 first causes the host device 311 to identify the radar data associated with the Doppler metrics and label the radar data as gesture data. Next, the labeling application 313 causes the host device 311 to identify the radar data collected between the start of the data collection period and the first cue, and the radar data collected between the second cue and the end of the data collection period, and label the radar data as non-gesture data. Finally, after labeling the radar data as gesture data and non-gesture data, the host device 311 outputs the labeled data to a memory associated with the system 300. For example, the host device 311 can store the labeled gesture data in a location of the memory dedicated to a particular gesture type, and store the labeled non-gesture data in a location of the memory dedicated to non-gesture movements.

[0079] While the foregoing embodiments generally relate to a context in which collection of radar data is performed in a local environment, it can be appreciated that these concepts are also applicable to a global environment in which radar data is collected across multiple client devices for training a neural network to perform gesture recognition. Figure 5A One such example operational environment 500 in an implementation is illustrated. The operational environment 500 represents an environment for gathering and labeling radar data for training a neural network to perform gesture recognition via radar. In an implementation, the operational environment 500 further represents an environment for training and deploying a neural network. Thus, the operational environment 500 can represent Figure 1 the operational environment 100 of FIG. 1. The operational environment 500 includes, but is not limited to, a host device 501, a service 503, and client devices 505, 507, and 509.

[0080] The host device 501 represents a device configured to manage collection of radar data across multiple different devices. For example, the host device 501 can represent a phone, a computer, or another similar device configured to manage collection of radar data across the client devices 505, 507, and 509. In an implementation, the host device 501 includes a user interface configured to collect configuration information for configuring collection of data across the client devices 505, 507, and 509. For example, a user can provide to the user interface of the host device 501 a type of gesture to be performed, a number of times the gesture is to be performed, and other similar inputs. The host device 501 can then supply the configuration information to the service 503, which in turn supplies the configuration information to the client devices 505, 507, and 509.

[0081] The services 503 represent one or more application services configured to provide various functions. For example, the services 503 can include application services related to data collection, data labeling, neural network training, and other similar functions. The application services related to data collection represent applications configured to provide configuration information to the client devices 505, 507, and 509. For example, after receiving configuration information from the host device 501, the application services of the services 503 can then configure the client devices 505, 507, and 509 to collect radar data of a particular gesture and output the data to the application services of the services 503 related to data labeling.

[0082] The application services related to data labeling represent applications configured to label the collected radar data as indicative of a gesture movement or a non-gesture movement. For example, the application services of the services 503 can represent the labeling process 200. In embodiments, the application services related to data labeling further represent a storage server configured to store the labeled radar data. For example, the storage server can include a first location configured to store gesture data collected by various client devices, and a second location to store non-gesture data.

[0083] In embodiments, after labeling the radar data as indicative of a gesture or a non-gesture, the services 503 supply the labeled radar data to the application services related to neural network training. The application services related to neural network training represent applications configured to train a neural network to perform gesture recognition via radar. For example, the application services of the services 503 can represent the training engine 107 of the Figure 1 In embodiments, after training the network to perform gesture recognition, the services 503 can deploy the trained network to the client devices 505, 507, and 509.

[0084] The client devices 505, 507, and 509 represent various user input devices configured to collect data for training a neural network to perform gesture recognition via radar. For example, the client devices 505, 507, and 509 can represent a vehicle, a cell phone, and a laptop computer, respectively, configured to collect radar data of a user performing gesture movements and non-gesture movements. In embodiments, the client devices 505, 507, and 509 are configured to collect data during a data collection period. The data collection period represents a time period specified by a user of the host device 501 during which the client devices 505, 507, and 509 are allowed to collect data associated with the user performing gesture movements and non-gesture movements. For example, during the data collection period, the client devices 505, 507, and 509 can collect ADC samples of their respective users performing gesture movements and non-gesture movements.

[0085] In an implementation, the client devices 505, 507, and 509 further represent devices configured to deploy a trained neural network. For example, the client devices 505, 507, and 509 can each represent an inference engine 109 of Figure 1 It should be noted that while illustrated as different devices (i.e., a vehicle, a cell phone, and a laptop), the client devices 505, 507, and 509 can represent the same type of device. Further, it should be noted that the client devices 505, 507, and 509 are not limited to the illustrated devices, but can represent a computer, a tablet, or another similar device configured to collect radar data and deploy a trained neural network to perform gesture recognition via radar.

[0086] Figure 5B An operational sequence 510 in an implementation is illustrated. The operational sequence 510 represents a sequence for collecting radar data and training a neural network to perform gesture recognition with respect to the elements of Figure 5A Accordingly, the operational sequence 510 includes the host device 501, the service 503, and the client device 505. It should be noted that the client device 505 represents an exemplary device, and thus further represents the client device 507, the client device 509, or another similar client device.

[0087] First, a user provides configuration information to a user interface of the host device 501. For example, the user can provide a number of gestures to be performed, a type of gesture to be performed, and a number of times each gesture should be performed. In an implementation, the user also provides various timing parameters, such as a duration of a data collection period.

[0088] After receiving the required configuration information, the host device 501 outputs the configuration information to the service 503. The service 503 receives the configuration information and routes the configuration information to the client device 505. In response, the client device 505 processes the configuration information to prepare for a data collection process.

[0089] Once prepared, the client device 505 can initiate a data collection period. In an implementation, during the data collection period, the client device 505 is configured to emit a first prompt and a second prompt. The first prompt and the second prompt can represent an audio prompt, a visual prompt, or another similar prompt that provides instructions to a user of the client device 505. More specifically, the first prompt represents an instruction to indicate to the user to perform a gesture, and the second prompt represents an instruction to indicate to the user that a duration for performing the gesture has terminated. Accordingly, data collected between the first prompt and the second prompt represents radar data associated with the user of the client device 505 performing a gesture. Further, data collected within the data collection period but outside the range of the first prompt and the second prompt represents radar data associated with the user of the client device 505 performing a non-gesture movement.

[0090] In embodiments, the client device 505 issues the first prompt and the second prompt based on configuration information provided by a user of the host device 501. For example, the user of the host device 501 can provide a time delay for issuing the first prompt and the second prompt. The time delay for issuing the first prompt represents a duration that the client device 505 must wait after the start of the data collection period to issue the first prompt. Alternatively, the time delay for issuing the second prompt represents a duration that the client device 505 must wait after issuing the first prompt to issue the second prompt.

[0091] After the termination of the data collection period, the client device 505 outputs the collected data to the service 503. In response, the application service of the service 503 begins processing the collected data to determine which subset of the collected data is associated with a gesture and which subset is associated with a non-gesture. In embodiments, to determine which subset of the radar data is associated with a gesture, the service 503 performs Doppler processing on the radar data collected between the first prompt and the second prompt to identify a Doppler metric associated with the user of the client device 505 performing a gesture. The Doppler metric represents a metric of a time frame of radar data that captures when motion (i.e., gesture movement) occurs.

[0092] Alternatively, to determine which subset of the radar data is associated with a non-gesture, the service 503 identifies radar data collected outside of the range of the first prompt and the second prompt. For example, the service 503 can identify radar data collected between the start of the data collection period and the first prompt, and radar data collected between the second prompt and the termination of the data collection period. Once identified, the service 503 can label the radar data associated with the Doppler metric as gesture data and label the radar data collected outside of the range of the two prompts as non-gesture data.

[0093] In embodiments, after labeling the radar data as gesture data or non-gesture data, the service 503 stores the labeled data in a storage server of the service 503. For example, the service 503 can store data of a first gesture in a data file dedicated to the gesture and store data of a different gesture in a data file dedicated to the different gesture. The service 503 can also store non-gesture data in a data file dedicated to non-gesture data. In embodiments, the service 503 stores the labeled radar data based on configuration information provided by a user of the host device 501. For example, the user of the host device 501 can provide a location for storing gesture data and a location for storing non-gesture data in the service 503.

[0094] Next, the service 503 trains a neural network to perform gesture recognition based on the labeled data. For example, the service 503 can train the network to perform a task in response to identifying a gesture. The service 503 can further train the network to continue monitoring for gesture movement in response to identifying a non-gesture. Once trained, the service 503 outputs the trained neural network to the client device 505. In response, the client device 505 deploys the trained neural network and begins performing gesture recognition via radar.

[0095] Figure 6 A user environment 600 in an implementation is illustrated. The user environment 600 represents a user interface configured to collect input data related to gesture recognition. For example, the user environment 600 can represent an interface configured to collect configuration information from a user of the host device 501. In another example, the user environment 600 represents an interface configured to collect user input for configuring the system 300. For purposes of explanation, the user environment 600 will be explained with respect to the elements of Figure 5A This is not meant to limit the application of the user environment 600, but to provide an example.

[0096] Prior to operation, a user of the host device 501 provides configuration information to the interface of the host device 501. In an implementation, the user first provides a location for storing labeled gesture data and labeled non-gesture data. For example, the user can specify a data path within the service 503 for storing the labeled gesture data and non-gesture data. Additionally, the user can also provide a file name for storing the labeled gesture data and non-gesture data.

[0097] Next, the user provides a gesture type to the interface of the host device 501. For example, the user can specify that the gesture type will represent a client lowering their hand from an upper position to a lower position. Once specified, the user provides a number of times the gesture will be performed. In an implementation, during the data collection period, the client devices 505, 507, and 509 can issue multiple iterations of the first prompt and the second prompt to collect multiple iterations of radar data for the same gesture.

[0098] Finally, the user provides various timing parameters to the interface of the host device 501. For example, the user can indicate a duration of the data collection period. The user can also indicate a time delay for issuing the various prompts. For example, the user can indicate that the first prompt should be issued three seconds after starting the data collection period and that the second prompt should be issued two seconds after the first prompt. In an implementation, the user can also specify a time delay for subsequent iterations of the first prompt. The time delay for subsequent iterations describes an amount of time that the client devices 505, 507, and 509 must wait after issuing the second prompt to issue the next iteration of the first prompt.

[0099] Reference is now made to the next figure, Figure 7 The collection process 700 in an implementation is illustrated. The collection process 700 represents a process for collecting data for training a neural network to perform gesture recognition via radar. The collection process 700 can be implemented in the context of program instructions that, when executed by a suitable computing system, direct processing circuitry of the computing system to operate as follows, with incidental reference to Figure 7 The collection process 700 will be explained with reference to elements of Figure 5A More specifically, the collection process 700 will be explained with respect to the client device 505. This is not meant to limit the application of the collection process 700, but to provide an example.

[0100] First, the client device 505 initiates a data collection period (step 701). In an implementation, the client device 505 initiates the data collection period after receiving configuration information provided by a user of the host device 501. In another implementation, the client device 505 initiates the data collection period based on instructions provided by a user of the client device 505. For example, after receiving the configuration information from the service 503, a user of the client device 505 can instruct the client device 505 to initiate the data collection period.

[0101] Next, after initiating the data collection period, the client device 505 issues a first prompt (step 703). The first prompt represents an instruction instructing a user of the client device 505 to perform a particular gesture. For example, the first prompt can represent an audio prompt, a visual prompt, or another similar prompt instructing the user to wave a hand from left to right.

[0102] In an implementation, the client device 505 issues the first prompt based on configuration information provided by a user of the host device 501. For example, prior to the data collection period, the user of the host device 501 can provide a time delay for issuing the first prompt. The time delay for issuing the first prompt describes a duration of time that the client device 505 must wait after initiation of the data collection period to issue the first prompt. In another implementation, the client device 505 issues the first prompt based on instructions provided by a user of the client device 505. For example, the client device 505 can include a microphone, a camera, or a touch device, such as a touchscreen, a touchpad, a keyboard, a keypad, a button, or a remote control, configured to collect user input, such as the first prompt.

[0103] After a period of time after the first prompt, the client device 505 issues a second prompt (step 705). The second prompt can represent an audio prompt, a visual prompt, or another similar sensory prompt that indicates to the user of the client device 505 that the time period for performing the gesture has terminated. In embodiments, the client device 505 issues the second prompt based on configuration information provided by the user of the host device 501. For example, prior to the data collection period, the user can provide a time delay for issuing the second prompt. The time delay for issuing the second prompt describes a duration of time that the client device 505 must wait after issuing the first prompt to issue the second prompt. In another embodiment, the client device 505 issues the second prompt based on instructions provided by the user of the client device 505. For example, the user of the client device 505 can provide input to the sensors of the client device 505 such that the input represents the second prompt.

[0104] Next, the client device 505 determines whether the number of data collection iterations for the specified gesture has been reached (step 707). In embodiments, prior to the data collection period, the user of the host device 501 instructs the client device 505 to collect multiple iterations of the same gesture. For example, the user of the host device 501 can instruct the client device 505 to collect five separate iterations of the user of the client device 505 waving a hand from left to right. In operation, the client device 505 can utilize the configuration information to determine whether the number of data collection iterations for a particular gesture has been reached.

[0105] If the number of data collection iterations has not been reached, the client device 505 returns to step 703 to perform the next data collection iteration. In embodiments, to perform the next data collection iteration, the client device 505 issues subsequent instances of the first prompt and the second prompt based on configuration information provided by the user of the host device 501. For example, the user of the host device 501 can provide a time delay for issuing subsequent iterations of the first prompt such that the time delay for the subsequent iterations describes a duration of time that the client device 505 must wait after issuing the second prompt to issue the subsequent iteration of the first prompt. In another embodiment, the client device 505 issues subsequent instances of the first prompt and the second prompt based on input provided by the user of the client device 505. For example, prior to the termination of the data collection period, the user of the client device 505 can instruct the client device 505 to issue subsequent iterations of the first prompt and the second prompt via the sensors of the client device 505 (i.e., microphone, camera, touch screen, etc.).

[0106] In embodiments, the client device 505 continues to issue the first prompt and the second prompt until a number of data collection iterations has been reached. For example, if the client device 505 is configured to collect five separate iterations of gesture data, after the first iteration of issuing the first prompt and the second prompt, the client device 505 will issue a second iteration, a third iteration, a fourth iteration, and a fifth iteration of the first prompt and the second prompt to capture five separate instances of the user of the client device 505 waving their hand from left to right.

[0107] Next, the client device 505 terminates the data collection period (step 709). In embodiments, the client device 505 terminates the data collection period based on configuration information provided by the user of the host device 501. For example, the user of the host device 501 can provide a duration of time for the data collection period. In another embodiment, the client device 505 terminates the data collection period based on instructions provided by the user of the client device 505. For example, after collecting a desired amount of radar data iterations, the user of the client device 505 can instruct the client device 505 to terminate the data collection period.

[0108] Finally, after termination of the data collection period, the client device 505 outputs the collected radar data to the service 503 (step 711). It should be noted that the collected radar data represents ADC samples associated with unlabeled gesture data and non-gesture data. The unlabeled gesture data represents radar data collected between iterations of the first prompt and the second prompt, while the unlabeled non-gesture data represents radar data collected within the data collection period but outside the range of the first prompt and the second prompt.

[0109] Figure 8 A labeling process 800 in embodiments is illustrated. The labeling process 800 represents a process for labeling data used to train a neural network to perform gesture recognition via radar. For example, the labeling process 800 can represent the labeling application 313 of Figure 3 The labeling process 800 can be implemented in the context of program instructions that, when executed by a suitable computing system, direct processing circuitry of the computing system to operate as follows, with incidental reference to steps in Figure 8 For purposes of explanation, the labeling process 800 will be explained as a process for labeling radar data collected via the collection process 700 (with reference to elements of Figure 5A This is not meant to limit the application of the labeling process 800, but to provide an example.

[0110] First, the service 503 receives unlabeled radar data from the client device 505 and, in response, identifies radar data collected between a first prompt and a second prompt (step 801). For example, the service 503 can receive ADC samples from the client device 505 and, in response, identify ADC samples collected between multiple iterations of a first prompt and a second prompt. The ADC samples collected between the one or more iterations of the first prompt and the second prompt represent unlabeled radar data associated with a gesture movement performed by a user of the client device 505 (i.e., unlabeled gesture data).

[0111] Next, the service 503 computes a Doppler metric associated with the gesture movement performed by the user of the client device 505 (step 803). The Doppler metric represents a metric that captures a time axis of radar data when motion occurs. For example, the Doppler metric can represent a metric that captures a time frame of radar data when the gesture movement is performed by the user of the client device 505.

[0112] In embodiments, to compute the Doppler metric associated with the gesture movement performed by the user of the client device 505, the service 503 performs Doppler processing on the ADC samples collected between the first prompt and the second prompt. For example, the service 503 can generate a range-Doppler heatmap for the ADC samples collected between the first prompt and the second prompt. The range-Doppler heatmap represents a mapping that describes how far away a target (i.e., the user of the client device 505) is and how fast the target is moving (i.e., performing a gesture). In embodiments, to compute the range-Doppler heatmap, the service 503 performs a range Fast Fourier Transform (FFT) and a Doppler FFT on each frame of radar data collected between the first prompt and the second prompt. Next, after generating the range-Doppler heatmap, the service 503 computes the Doppler metric for the radar data collected between the first prompt and the second prompt based on the generated heatmap. For example, the service 503 can employ the following equation:

[0113]

[0114] such that M represents a weighted average of the range-Doppler heatmap (i.e., the Doppler metric), Z i,j represents a heatmap value at cell (i,j) of the range-Doppler heatmap, D i represents a Doppler value at column (i) of the range-Doppler heatmap, and Z i represents a heatmap value at column (i) of the range-Doppler heatmap.

[0115] After computing the Doppler metrics, the service 503 utilizes the computed Doppler metrics to extract a piece of radar data associated directly with a gesture movement performed by a user of the client device 505 (step 805). Next, the service 503 labels the extracted piece of radar data associated with the gesture movement as indicative of a gesture (step 807). For example, if the gesture movement represents the user of the client device 505 waving their hand from left to right, the service 503 can label the extracted piece of radar data as indicative of: “Handwave L:R”.

[0116] Next, the service 503 identifies pieces of radar data associated with non-gesture movements (step 809). In embodiments, to identify pieces of radar data associated with non-gesture movements, the service 503 identifies ADC samples collected within the data collection period but outside of the range of the first and second cues. For example, the service 503 can identify ADC samples collected after the start of the data collection period but before the issuance of the first cue, ADC samples collected between the second cue and a subsequent iteration of the first cue, and ADC samples collected between the final second cue and the termination of the data collection period.

[0117] Next, the service 503 extracts pieces of radar data associated with non-gesture movements (step 811) and labels the pieces as indicative of non-gesture (step 813). Finally, after labeling the gesture data and non-gesture data, the service 503 can provide the labeled radar data to a machine learning algorithm training dataset, such as a neural network training dataset (step 815). For example, the service 503 can provide the labeled data to an application service of the service 503 that is configured to train a neural network to recognize both gesture movements and non-gesture movements. In embodiments, the service 503 can train a neural network to perform gesture recognition and output the trained network to the client device 505 for deployment.

[0118] Turning now to the next figure, Figure 9 Operation context 900 is illustrated in an embodiment. Operation context 900 represents a context for gathering data for training a neural network to perform gesture recognition via radar. More specifically, operation context 900 represents a context for collecting radar data associated with gesture movements and non-gesture movements. In embodiments, operation context 900 depicts a plot of Doppler metrics over time (i.e., over radar frames). For purposes of explanation, operation context 900 will be explained utilizing the elements of Figure 5A This is not meant to limit the application of operation context 900, but to provide an example.

[0119] First, client device 505 initiates a data collection period and, in response, begins collecting radar data of the user of client device 505 performing non-gesture movements. Next, after a period of time (i.e., T1) after the start of the data collection period, client device 505 issues prompt 901. Prompt 901 represents an instruction instructing the user to perform a gesture. For example, prompt 901 may represent an audio prompt, a visual prompt, or another similar prompt instructing the user to wave their hand from left to right.

[0120] After client device 505 issues prompt 901, client device 505 may collect radar data associated with the user performing the gesture movement. Next, after a period of time (i.e., T2, T3, and T4) after issuing prompt 901, client device 505 issues prompt 902. Prompt 902 indicates to the user that the time period for performing the gesture has expired. For example, prompt 902 may be an audio prompt, a visual prompt, or another similar prompt instructing the user to stop waving from left to right.

[0121] Next, after a period of time following prompt 902 (i.e., T5), client device 505 terminates the data collection period and outputs the unlabeled gesture data and non-gesture data to service 503. Service 503 receives the unlabeled radar data and, in response, performs Doppler processing on the radar data to identify a Doppler metric. In one embodiment, the Doppler metric represents radar data collected between the time frames of T1 and T5. In another embodiment, the Doppler metric represents radar data collected between the first prompt and the second prompt (i.e., T2 to T4). For purposes of explanation, the Doppler metric represents data collected during the data collection period (i.e., T1 to T5).

[0122] In an embodiment, service 503 analyzes the Doppler measurements to identify segment 903. Segment 903 represents a subset of the Doppler measurements that captures the time frame when the gesture was performed. For example, segment 903 may represent a segment that captures the time frame (i.e., T3) in which the user of client device 505 is waving from left to right. It should be noted that segment 903 may further represent radar data collected outside the illustrated time frame (i.e., T3).

[0123] In an embodiment, to identify segment 903, service 503 first identifies a peak of the Doppler metric collected between cues 901 and 902. The peak of the Doppler metric is a point at which the Doppler metric has a highest value between cues 901 and 902. Next, service 503 identifies a time frame of radar data before the identified peak (i.e., W L) and a time frame of radar data after the identified peak (i.e., W R) and labels the identified time frames as segment 903 (i.e., W LEN). In an embodiment, the time frame of radar data identified before the identified peak is equal to the time frame of radar data identified after the identified peak (i.e., W L = W R). In another embodiment, the time frame of radar data identified before the identified peak is not equal to the time frame of radar data identified after the identified peak (i.e., W L ≠ W R). In either case, the resulting time frames, referred to herein as segment 903, represent a fixed length input suitable for training a neural network (i.e., ANN) to perform gesture recognition based on radar data.

[0124] In another embodiment, to identify segment 903, service 503 first identifies a plurality of consecutive frames between cues 901 and 902 such that a sum of the Doppler metric across the identified frames is maximized. Next, service 503 labels the identified frames as segment 903. Finally, service 503 outputs segment 903 to a training dataset such that segment 903 represents a fixed length (W LEN) input suitable for training a neural network to perform gesture recognition via radar.

[0125] It should be noted that while fixed length inputs can be suitable for training some neural networks, other networks (i.e., RNNs) can also accept variable length inputs. For example, to identify segment 903, service 503 can first identify a peak of the Doppler metric collected between cues 901 and 902. Next, service 503 can identify a time frame of radar data before the identified peak (i.e., W L) and a time frame of radar data after the identified peak (i.e., W R) such that the identified time frames include Doppler metric values that are always less than the identified peak value by a certain percentage. For example, service 503 can identify a time frame of radar data in which the Doppler metric values are within 10% of the identified peak value. Thus, service 503 can identify radar data associated with a gesture and output segment 903.

[0126] In another example, to identify the segment 903, the service 503 can identify a window within the Doppler metric that captures a percentage of the Doppler metric energy. For example, the service 503 can be configured to identify a segment of radar data that captures 90% of the total energy of the Doppler metric between the cues 901 and 902. In an implementation, to identify a window that meets the energy criteria, the service 503 scans the radar data collected between T2 and T4 to identify a minimum length window that captures a specified amount of energy. Thus, the service 503 can identify a window of radar data associated with the gesture and output the segment 903. In an implementation, the energy of the Doppler metric in a given window is calculated as the sum of the squares of the Doppler metric across the frames within the window.

[0127] Figure 10 An operational scenario 1000 in an implementation is illustrated. The operational scenario 1000 represents another scenario for gathering data for training a neural network to perform gesture recognition via radar. More specifically, the operational scenario 1000 represents a scenario for collecting multiple iterations of radar data associated with gesture movements and non-gesture movements. In an implementation, the operational scenario 1000 depicts a plot of Doppler metric over time (i.e., over radar frames). For purposes of explanation, the operational scenario 1000 will be explained with reference to the elements of Figure 5A This is not meant to limit the application of the operational scenario 1000, but to provide an example.

[0128] First, the client device 505 initiates a data collection period and, in response, begins collecting radar data of the user of the client device 505 performing a non-gesture movement. Next, after a period of time (i.e., Tl) after initiating the data collection period, the client device 505 emits a cue 1001. The cue 1001 represents an instruction instructing the user to perform a gesture. For example, the cue 1001 can represent an audio cue, a visual cue, or another similar cue instructing the user to wave a hand from left to right.

[0129] After the client device 505 emits the cue 1001, the client device 505 can collect radar data associated with the user performing a gesture movement. Next, after a period of time (i.e., T2, T3, and T4) after emitting the cue 1001, the client device 505 emits a cue 1002. The cue 1002 represents an instruction instructing the user to cease the gesture. For example, the cue 1002 can represent an audio cue, a visual cue, or another similar cue instructing the user that the allowable timeframe for waving a hand from left to right has terminated.

[0130] Next, after a period of time (i.e., T5) after the cue 1002, the client device 505 emits a cue 1004. The cue 1004 represents an instruction instructing the user to perform a gesture. For example, the cue 1004 can instruct the user to wave a hand from left to right again.

[0131] After the client device 505 issues the prompt 1004, the client device 505 can collect a second iteration of radar data associated with the user performing the gesture movement. Next, after a time period (i.e., T6, T7, and T8) after issuing the prompt 1004, the client device 505 issues the prompt 1005. The prompt 1005 represents another instruction instructing the user to stop the gesture.

[0132] Next, after a time period (i.e., T9) after the prompt 1005, the client device 505 issues the prompt 1007. The prompt 1007 represents another instruction instructing the user to perform the same gesture again.

[0133] After the client device 505 issues the prompt 1007, the client device 505 can collect a third iteration of radar data associated with the user performing the gesture movement. Next, after a time period (i.e., T10, T11, and T12) after issuing the prompt 1007, the client device 505 issues the prompt 1008. The prompt 1008 represents another instruction instructing the user to stop the gesture.

[0134] Next, after a time period (i.e., T13) after the prompt 1008, the client device 505 terminates the data collection period and outputs the unlabeled gesture data and the non-gesture data to the service 503. The service 503 receives the unlabeled gesture data and the non-gesture data, and in response, performs Doppler processing on the radar data to determine a Doppler metric. The Doppler metric represents radar data collected between the time frames of T1 and T13.

[0135] In embodiments, the service 503 analyzes the Doppler metric to identify the segments 1003, 1006, and 1009. The segments 1003, 1006, and 1009 represent subsets of the Doppler metric that capture time frames when the gesture was performed. For example, the segment 1003 represents a subset that captures the time frame (i.e., T3) when the user of the client device 505 waved their hand from left to right between the prompts 1001 and 1002. The segment 1006 represents a subset that captures the time frame (i.e., T7) when the user of the client device 505 waved their hand from left to right between the prompts 1004 and 1005. The segment 1009 represents a subset that captures the time frame (i.e., T11) when the user of the client device 505 waved their hand from left to right between the prompts 1007 and 1008. It should be noted that the segments 1003, 1006, and 1009 can further represent radar data collected outside of the illustrated time frames (i.e., T3, T7, and T11).

[0136] In an embodiment, to identify the segments 1003, 1006, and 1009, the service 503 first identifies a peak of the Doppler metric collected between the cues 1001 and 1002, the cues 1004 and 1005, and the cues 1007 and 1008. Next, the service 503 identifies a time frame of the radar data before and after the identified peak, and labels the identified time frames as the segments 1003, 1006, and 1009 (i.e., W_LEN). In an embodiment, the time frame of the radar data identified before the identified peak is equal to the time frame of the radar data identified after the identified peak. In another embodiment, the time frame of the radar data identified before the identified peak is not equal to the time frame of the radar data identified after the identified peak. It should be noted that whether the time frames are equal or not depends on the type of gesture being performed.

[0137] Figure 11 An operational scenario 1100 in an embodiment is illustrated. The operational scenario 1100 represents another scenario for collecting data for training a neural network to perform gesture recognition via radar. More specifically, the operational scenario 1100 represents a scenario for extracting radar data associated with a gesture movement. For explanatory purposes, the operational scenario 1100 will be explained with reference to the elements of Figure 5A This is not meant to limit the application of the operational scenario 1100, but to provide an example.

[0138] First, the service 503 analyzes the collected radar data to determine an average Doppler metric of the radar data. The average Doppler metric represents a metric of a time frame of the radar data that captures when the gesture occurs. In an embodiment, the service 503 can perform Doppler processing on the collected radar data to output a plot 1101. The plot 1101 represents a plot that depicts the average Doppler metric of the collected radar data. In other words, the plot 1101 represents a plot that depicts a time at which the user of the client device 505 performs the gesture.

[0139] Next, the service 503 analyzes the collected radar data to determine an azimuth angle weighted mean of the radar data. The azimuth angle weighted mean represents a weighted average distance between the radar device and the user of the client device 505. In an embodiment, the service 503 can analyze the collected radar data to output a plot 1103. The plot 1103 represents a plot that depicts the azimuth angle weighted mean when the user performs the gesture.

[0140] The service 503 can further analyze the collected radar data to determine an elevation angle weighted mean of the radar data. The elevation angle weighted mean represents a weighted average angle between the radar device and the user of the client device 505. In an embodiment, the service 503 can analyze the collected radar data to output a plot 1105. The plot 1105 represents a plot that depicts the elevation angle weighted mean when the user performs the gesture.

[0141] Finally, the service 503 can analyze the collected radar data to determine a correlation between the Doppler metric and the azimuthally weighted mean. In an implementation, the service 503 can analyze the collected radar data to output the plot 1107. The plot 1107 represents a plot depicting a correlation between the Doppler metric and the azimuthally weighted mean.

[0142] In an implementation, the plots 1101, 1103, 1105, and 1107 represent unlabeled radar data associated with a gesture. For example, the plots 1101, 1103, 1105, and 1107 can represent time series data generated from appropriately processing ADC samples. In an implementation, the service 503 labels a WIN_LEN segment of the plots 1101, 1103, 1105, and 1107 as indicative of a gesture and provides the labeled plots to a neural network training dataset. For example, the service 503 can provide the labeled plots to an application service of the service 503 configured to train a neural network to recognize both gesture movements and non-gesture movements via radar.

[0143] Figure 12 An example computer system that can be used in implementations is illustrated. For example, the computing system 1201 represents a computing device capable of collecting and labeling radar data for training a neural network to perform gesture recognition as described herein. The computing system 1201 represents any system or collection of systems that can employ the various operational architectures, processes, contexts, and sequences disclosed herein for collecting and labeling radar data associated with gesture movements and non-gesture movements. Examples of the computing system 1201 include, but are not limited to, a microcontroller unit (MCU), an embedded computing device, a server computer, a cloud computer, a personal computer, a cell phone, and the like.

[0144] The computing system 1201 can be implemented as a single device, system, or apparatus, or can be implemented in a distributed manner as multiple devices, systems, or apparatuses. The computing system 1201 includes, but is not limited to, a processing system 1202, a storage system 1203, software 1205, a communication interface system 1207, and a user interface system 1209 (optional). The processing system 1202 is operatively coupled with the storage system 1203, the communication interface system 1207, and the user interface system 1209. The computing system 1201 can represent a cloud computing device, a distributed computing device, or the like.

[0145] The processing system 1202 loads and executes software 1205 from the storage system 1203 or alternatively, runs software 1205 that has been downloaded from the storage system 1203. The software 1205 includes program instructions 1206, which include the tagging process 1208 (i.e., the tagging process 200, the tagging application 313, the collection process 700, or the tagging process 800). When executed by the processing system 1202, the software 1205 instructs the processing system 1202 to operate at least as described herein for the various processes, operational scenarios, and sequences discussed in the foregoing embodiments. The computing system 1201 can optionally include additional devices, features, or functionality not discussed in the interest of brevity.

[0146] Still referring to Figure 12 The processing system 1202 can include a microprocessor and other circuitry that retrieves and executes software 1205 from the storage system 1203. The processing system 1202 can be implemented within a single processing device, but can also be distributed across multiple processing devices or sub-systems that cooperate in executing program instructions. Examples of the processing system 1202 include general purpose central processing units, graphical processing units, digital signal processing units, data processing units, special purpose processors, and

[0147] The storage system 1203 can include any computer readable storage media readable by the processing system 1202 and capable of storing software 1205. The storage system 1203 can include, in any combination, volatile and non-volatile media, removable and non-removable media, and volatile and non-volatile media. Examples of storage media include random access memory, read only memory, magnetic disks, optical disks, flash memory, virtual memory and non-virtual memory, magnetic cassettes, magnetic tapes, magnetic disk storage or other magnetic storage devices, or any other suitable storage media. In no case is the computer readable storage media a propagated signal.

[0148] In addition to the computer readable storage media, in some embodiments, the storage system 1203 can also include computer readable communication media that can be used to transfer at least some of the software 1205 between the storage system 1203 and another computing device, or between the storage system 1203 and any other entity. The storage system 1203 can be implemented with a single storage device, but can also be implemented using as a plurality of storage devices or sub-systems which can or can not be located relative to one another. The storage system 1203 can include additional elements, such as a controller, capable of communicating with the processing system 1202 or possibly other systems.

[0149] Software 1205 can be implemented in program instructions 1206 and, among other functions, can direct the processing system 1202, when executing the program instructions, to operate as described with respect to the various operational scenarios, sequences, and processes described herein. Specifically, the program instructions can include various components or modules that cooperate or otherwise interact to carry out the various processes and operational scenarios described herein. The various components or modules can be embodied in compiled or interpreted instructions, or in some other variant or combination of instructions. The various components or modules can execute in synchronous or asynchronous fashion, serially or in parallel, in a single threaded environment or multi-threaded, or according to any other suitable execution paradigm, variant, or combination thereof. Software 1205 can include additional processes, programs, or components, such as operating system software, virtualization software, or other application software. Software 1205 can also comprise firmware or some other form of machine-readable processing instructions executable by the processing system 1202.

[0150] In general, software 1205, when loaded into processing system 1202 and executed, can transform a suitable apparatus, system, or device, represented by computing system 1201, from a general purpose computing system into a special purpose computing system customized to support binary convolution operations. Indeed, encoding software 1205 (and marking process 1208) on storage system 1203 can transform the physical system 1203 by

[0151] For example, if the computer-readable storage media are implemented with semiconductor-based memory, software 1205 can transform the physical state of the semiconductor memory when the program instructions are encoded therein, such as by transforming the state of transistors, capacitors or other discrete circuit elements constituting the semiconductor memory. A similar transformation can occur with respect to magnetic or optical media, for example. Other transformations of physical media are possible without departing from the scope of the present description, with the foregoing examples provided only to facilitate this discussion.

[0152] Communication interface system 1207 can include communication connections and devices that allow for communication with other computing systems over a communication network (not shown). Examples of connections and devices that together allow for inter-system communication can include a network interface card, an antenna, power amplifiers, radio frequency circuitry, transceivers, and other communication circuitry. The connections and devices can communicate over a communication medium to exchange communications with other computing systems or systems networks, such as metal, glass, air, or any other suitable communication medium. The aforementioned media, connections, and devices are well known and need not be discussed at length here.

[0153] Communication between computing system 1201 and other computing systems (not shown) can occur through communication networks and according to various communication protocols, combinations of protocols, or variations thereof. Examples include intranets, internets, the Internet, local area networks, wide area networks, wireless networks, wired networks, virtual networks, software-defined networks, data center buses and backplanes, or any other type of network, combination of networks, or variation thereof. The aforementioned communication networks and protocols are well known and need not be discussed at length here.

[0154] As those skilled in the art will appreciate, aspects of the present application can be embodied as a system, a method, or a computer program product. Accordingly, aspects of the present application can be in the form of an entirely hardware embodiment, an entirely software embodiment (including firmware, resident software, microcode, etc.) or an embodiment combining software and hardware aspects that can all generally be referred to herein as a "circuit," "module" or "system." Furthermore, aspects of the present application can be in the form of a computer program product embodied in one or more computer readable medium(s) having computer readable program code embodied thereon.

[0155] Indeed, the descriptions and illustrations herein describe and demonstrate specific embodiments of the best mode contemplated by the inventors. The descriptions and illustrations provided are not intended to limit the scope of the application, but merely to describe and illustrate specific embodiments thereof. Those skilled in the art will appreciate that various modifications can be made to the specific embodiments described herein without departing from the scope of the present application. Accordingly, the descriptions and illustrations herein are intended to be illustrative only and are not intended to limit the scope of the application. Those skilled in the art will appreciate that the features described herein can be combined in various ways with each other. Thus, the application is not limited to the specific embodiments described herein, but only by the claims and their equivalents, and any portion of the disclosure of which can be combined in various ways.

[0156] The above description and associated drawings teach, by way of example, the best mode contemplated by the inventors for carrying out aspects of the application. The detailed description and drawings are intended to be illustrative only of the best mode and are not intended to limit the scope of the application. Rather, the scope of the application is to be determined by the claims and their equivalents, and any portion of the disclosure of which can be combined in various ways with each other. Accordingly, the application is not to be limited by the specific illustrative descriptions and examples made herein, but rather is to be understood broadly inclusive, for example, as set forth in the preceding claims.

Claims

1. A non-transitory computer-readable medium having executable instructions stored thereon, the executable instructions configured to be executable by a processing circuit system to cause the processing circuit system to: identifying radar data collected during a time period between the first prompt and the second prompt; identifying a subset of the radar data based at least on Doppler processing; as well as The subset of the radar data is labeled as a gesture.

2. The non-transitory computer-readable medium of claim 1 , wherein the instructions further cause the processing circuitry to: outputting the first prompt, wherein the first prompt instructs a user to initiate the gesture; and After the time period has elapsed after the first prompt has been output, the second prompt is output, wherein the second prompt indicates to the user that a duration for performing the gesture has expired.

3. The non-transitory computer-readable medium of claim 2, wherein the radar data is collected during a data collection period, and wherein the instructions further instruct the processing circuitry to: outputting the first prompt after initiation of the data collection period; and The second prompt is output prior to expiration of the data collection period.

4. The non-transitory computer-readable medium of claim 3, wherein the instructions further instruct the processing circuitry to: identifying a second set of the radar data collected between the start of the data collection period and the first prompt; identifying a third set of the radar data collected between the second prompt and the termination of the data collection period; as well as The second set of radar data and the third set of radar data are labeled as negative gesture samples.

5. The non-transitory computer-readable medium of claim 1, wherein the first prompt and the second prompt are audio prompts. 6 . The non-transitory computer-readable medium of claim 1 , wherein the instructions further direct the processing circuitry to identify the first prompt and the second prompt based on a signal generated by a user input device.

7. The non-transitory computer-readable medium of claim 6, wherein the user input device comprises a microphone configured to generate the signal based on audio received by the microphone.

8. The non-transitory computer-readable medium of claim 6, wherein the user input device comprises a touch device.

9. The non-transitory computer-readable medium of claim 1 , wherein the instructions further instruct the processing circuitry to: collecting the radar data after the user initiates the first prompt; calculating a Doppler metric after the user initiates the first prompt; and A second set of the radar data is collected after the user initiates the second prompt.

10. A method comprising: collecting radar data during a data collection period; identifying a set of said radar data collected during a time period between a first prompt and a second prompt; identifying a subset of the radar data from the set of radar data based at least on Doppler processing; as well as The subset of the radar data is labeled as a gesture.

11. The method of claim 10, further comprising during the data collection period: outputting the first prompt, wherein the first prompt instructs a user to initiate the gesture; and After the time period has elapsed after the first prompt has been output, the second prompt is output, wherein the second prompt indicates to the user that a duration for performing the gesture has expired.

12. The method according to claim 11, further comprising: identifying a second set of the radar data collected between the start of the data collection period and the first prompt; identifying a third set of the radar data collected between the second prompt and termination of the data collection period; as well as The second set of radar data and the third set of radar data are labeled as negative gesture samples.

13. The method of claim 10, wherein the first prompt and the second prompt are audio prompts.

14. The method of claim 10, further comprising identifying the first prompt and the second prompt based on a signal generated by a user input device.

15. The method of claim 14, wherein the user input device comprises a microphone configured to generate the signal based on audio received by the microphone.

16. The method of claim 14, wherein the user input device comprises a touch device.

17. A method comprising: outputting a first prompt to instruct a user to perform a first instance of a gesture; collecting sensor data after the first prompt; outputting a second prompt to instruct the user to cease the first instance of the gesture after outputting the first prompt; training a machine learning algorithm using the sensor data collected between the first prompt and the second prompt; as well as A second instance of the gesture is detected using the machine learning algorithm.

18. The method of claim 17, further comprising performing range-Doppler processing on the collected sensor data, wherein training the machine learning algorithm uses the range-Doppler processed sensor data.

19. The method according to claim 17, wherein outputting the first prompt comprises outputting a first audio prompt via a speaker, and Wherein outputting the second prompt includes outputting a second audio prompt via the speaker.

20. The method according to claim 17, Wherein outputting the first prompt comprises outputting a first visual prompt via a display, and wherein outputting the second prompt comprises outputting a second visual prompt via the display.

Images