Method and system for character recognition based on radar network

Abstract

Embodiments of the present disclosure generally relate to methods and systems for airwriting character recognition based on radar networks. In an embodiment, a method for airwriting character recognition includes: determining the position of an object in a monitoring space using trilateration by using multiple millimeter wave radars, wherein each of the multiple millimeter wave radars has a field of view, and wherein the intersection of the fields of view of the multiple millimeter wave radars forms the monitoring space; using the multiple millimeter wave radars to track the position of the object that changes over time in the monitoring space; using a neural network (NN) to determine a character symbol depicted by the tracked position of the object that changes over time; and providing a signal based on the determined character symbol.

Description

Technical Field

[0001] The present invention relates generally to electronic systems and methods, and in a particular embodiment to character recognition in airwriting based on a radar network for human-machine interface. Background Art

[0002] Humans interact with machines using human-machine interfaces. Conventional human-machine interfaces have been available for many years. Examples of conventional human-machine interfaces include input hardware such as keyboards, mice, game controllers, microphones, output hardware such as computer monitors, printers, and speakers, and input / output hardware such as touch screens.

[0003] Human-computer interfaces can include several categories, such as command line interfaces (e.g., receiving input via a keyboard and providing output in text form via a computer display), graphical user interfaces (e.g., receiving input via a keyboard and mouse and providing output using graphics via a computer display), voice user interfaces (e.g., using a person's voice to control a device), and gesture interfaces (e.g., using a person's gestures captured via a camera to control a device).

[0004] A human-machine interface can be used to facilitate interaction between people. For example, a first person can interact with a first computer using a first human-machine interface including a microphone, a speaker, a camera, and a computer display in order to interact with a second person. The first computer transmits data associated with such interaction to a second computer. The second person interacts with the second computer using a second human-machine interface including a microphone, a speaker, a camera, and a computer display. Summary of the invention

[0005] According to one embodiment, a method for air writing character recognition includes: determining a position of an object in a monitoring space using trilateration by using multiple millimeter wave radars, wherein each of the multiple millimeter wave radars has a field of view, and wherein the intersection of the fields of view of the multiple millimeter wave radars forms the monitoring space; tracking a position of an object that changes over time in the detection space using multiple millimeter wave radars; determining a character symbol depicted by the tracked position of the object that changes over time using a neural network (NN); and providing a signal based on the determined character symbol.

[0006] According to one embodiment, an air writing character recognition system includes: multiple millimeter wave radars, wherein each of the multiple millimeter wave radars is configured to have a field of view, and wherein the intersection of the fields of view of the multiple millimeter wave radars forms a monitoring space; and a controller, which is configured to: determine the position of an object in the monitoring space based on the outputs of the multiple millimeter wave radars by using trilateration, track the position of the object that changes over time in the monitoring space based on the position determined using the multiple millimeter wave radars, determine a character symbol depicted by the tracked position of the object that changes over time using a neural network (NN), and provide a signal based on the determined character symbol.

[0007] According to one embodiment, a millimeter-wave radar system includes: three millimeter-wave radars, each of the three millimeter-wave radars is configured to have a field of view, wherein the intersection of the fields of view of each of the three millimeter-wave radars forms a monitoring space; and a controller, configured to: determine the position of an object in the monitoring space based on the outputs of the three millimeter-wave radars by using trilateral measurement, determine a trajectory of the object that changes over time based on the determined position of the object, apply a filter to the determined trajectory to generate a filtered trajectory, determine a character symbol described by the filtered trajectory using a long short-term memory (LSTM), and provide a signal based on the determined character symbol. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] For a more complete understanding of the present invention and its advantages, reference is now made to the following description taken in conjunction with the accompanying drawings, in which:

[0009] Figure 1 A schematic diagram of a millimeter wave (mmWave) radar system according to an embodiment of the present invention is shown;

[0010] Figure 2 The embodiment according to the present invention is shown Figure 1 Schematic diagram of millimeter wave radar;

[0011] Figure 3 A flowchart of an embodiment method of fine distance estimation according to an embodiment of the present invention is shown;

[0012] Figure 4 A millimeter wave radar with a field of view (FoV) according to an embodiment of the present invention is shown;

[0013] Figure 5 A millimeter wave radar with FoV according to an embodiment of the present invention is shown;

[0014] Figure 6 It shows a network of millimeter wave radars and a corresponding monitoring space according to an embodiment of the present invention;

[0015] Figure 7 The embodiment according to the present invention is shown Figure 6 Schematic diagram of the millimeter wave radar network;

[0016] Figure 8 A flow chart showing an implementation method of using a Kalman filter to track the trajectory of an object in a monitoring space according to an embodiment of the present invention;

[0017] Fig. 9 A schematic diagram of an LSTM network according to an embodiment of the present invention is shown;

[0018] Fig.10 The trajectories of metal markers in the monitoring space and the corresponding labels generated by the LSTM network according to an embodiment of the present invention are illustrated;

[0019] Fig.11 A flow chart of an exemplary method for character recognition in air writing according to an embodiment of the present invention is shown;

[0020] Fig.12 A flow chart illustrating an embodiment method for character recognition in air writing using frame-by-frame classification according to an embodiment of the present invention is shown;

[0021] Fig.13 A flow chart illustrating an embodiment method for character recognition in air writing using frame-by-frame classification according to an embodiment of the present invention is shown;

[0022] Fig.14 A flowchart of an exemplary method for character recognition in air writing using CTC classification according to an exemplary embodiment of the present invention is shown;

[0023] Fig.15 A method for character recognition in air writing according to an embodiment of the present invention is shown. Figure 6 smartphones that use millimeter-wave radar networks; and

[0024] Fig.16 A method for character recognition in air writing according to an embodiment of the present invention is shown. Figure 6 Computer monitor with network of millimeter wave radars.

[0025] Corresponding numerals and symbols in the different figures generally refer to corresponding parts unless otherwise noted.The figures are drawn to clearly illustrate the relevant aspects of the preferred embodiments and are not necessarily drawn to scale. DETAILED DESCRIPTION

[0026] The making and using of the disclosed embodiments are discussed in detail below. However, it should be understood that the present invention provides many applicable inventive concepts embodied in a variety of specific contexts. The specific embodiments discussed are merely illustrative of specific methods of making and using the present invention, and the specific embodiments discussed do not limit the scope of the present invention.

[0027] The following description describes various specific details to provide a deep understanding of several exemplary embodiments according to the description. The embodiments can be obtained without one or more specific details, or with other methods, components, materials, etc. In other cases, known structures, materials or operations are not shown or described in detail to avoid confusing different aspects of the embodiments. Reference to "an embodiment" in this description means that a specific configuration, structure or feature described according to the embodiment is included in at least one embodiment. Therefore, phrases such as "in one embodiment" that appear at different points in this specification do not necessarily refer to the same embodiment. In addition, specific configurations, structures or features can be combined in any appropriate manner in one or more embodiments.

[0028] Embodiments of the present invention will describe in a specific context a system and method for character recognition in air writing based on a millimeter wave radar network for a human-machine interface. Embodiments of the present invention can be implemented using other types of radars (e.g., radars other than millimeter wave radars), and embodiments of the present invention can be used for other applications (e.g., for machine-machine interfaces). Although embodiments of Latin character symbols and numeric symbols are illustrated, it is understood that other types of symbols, such as non-Latin characters (e.g., Chinese characters, Hebrew characters, Urdu characters, and sign language gestures) may also be used.

[0029] In one embodiment of the present invention, an air writing system including a network of millimeter wave radars uses a two-stage approach to extract and recognize handwriting gestures. The extraction processing stage uses sophisticated distance estimation combined with triangulation techniques to detect and locate the user's hand, and then creates a smooth trajectory of the gesture movement through a Kalman filter or any smoothing filter. The recognition stage classifies the characters drawn by the user by using a long short-term memory (LSTM), such as a bidirectional LSTM (BiLSTM), which is fed with a continuous Kalman filter state along the gesture trajectory.

[0030] Conventional gesture recognition systems can be based on camera modules using computer vision techniques and optical sensors. Camera-based recognition systems typically use a lot of computing power to process graphic images that are illuminated under appropriate lighting and include at least part of a human being. Gesture recognition systems can also be implemented using light and time-of-flight (ToF) technology.

[0031] Advantages of some embodiments include accurately detecting handwritten characters using gestures without processing graphic images, and without being affected by changes in lighting of the hand performing the gesture. Therefore, some embodiments are less computationally expensive than conventional camera-based systems and avoid privacy issues associated with graphic images of humans. Some embodiments advantageously determine character symbols from gestures without using wearable devices that facilitate gesture recognition. Using millimeter wave radar instead of conventional ToF technology advantageously allows the use of estimates of Doppler shift to accurately determine the trajectory of the hand.

[0032] In one embodiment, motion gesture sensing is performed wirelessly by a network of millimeter wave radars. Trilateration is used to accurately locate the hand in a three-dimensional coordinate system. A Kalman filter or any smoothing filter is used to track and smooth the motion trajectory of the hand. A bidirectional LSTM generates a textual representation of the motion sensor data from the hand received from the Kalman filter.

[0033] Figure 1 A schematic diagram of a millimeter wave radar system 100 according to an embodiment of the present invention is shown. During normal operation, the millimeter wave radar 102 transmits a plurality of radiation pulses 106 (such as chirps) to a scene 108. In some embodiments, the chirps are linear chirps (i.e., the instantaneous frequency of the chirps varies linearly with time).

[0034] The emitted radiation pulse 106 is reflected by an object (also referred to as a target) in the scene 108. For example, the reflected radiation pulse (at Figure 1 The echo signal (not shown) is also called an echo signal, which is detected by the millimeter wave radar 102 and processed by the processor 104 to determine the arrival angle of the echo signal.

[0035] For example, objects in scene 108 may include human hand 114. Scene 108 may include other objects, such as fingers, metal markers, and other objects.

[0036] For example, the processor 104 uses signal processing techniques to analyze the echo data to determine the position of the hand 114. For example, in some embodiments, the range FFT is used to estimate the range component of the position of the index finger tip of the hand 114 (i.e., the distance of the index finger tip of the hand 114 from the millimeter wave radar). An angle estimation technique can be used to determine the azimuth component of the position of the detected human.

[0037] Processor 104 can be implemented as a general-purpose processor, a controller or a digital signal processor (DSP), for example, processor 104 includes a combination circuit coupled to a memory. In some embodiments, for example, DSP can be implemented using an ARM architecture. In some embodiments, processor 104 can be implemented as a customized application-specific integrated circuit (ASIC). In some embodiments, processor 104 includes a plurality of processors, each of which has one or more processing cores. In other embodiments, processor 104 includes a single processor with one or more processing cores. Other implementations are also possible. For example, some embodiments can be implemented as a combination of a hardware accelerator and software running on a DSP or a general-purpose microcontroller.

[0038] The millimeter wave radar 102 is used as a frequency modulated continuous wave (FMCW) radar, which includes a millimeter wave radar sensor circuit, a transmitting antenna, and a receiving antenna. The millimeter wave radar 102 transmits and receives signals in the range of 20 GHz to 122 GHz. Alternatively, frequencies outside this range, such as: frequencies between 1 GHz and 20 GHz, or frequencies between 122 GHz and 300 GHz, can also be used.

[0039] In some embodiments, in a manner known in the art, a bandpass filter (BPF), a low-pass filter (LPF), a mixer, a low-noise amplifier (LNA), and an intermediate frequency (IF) amplifier are used to filter and amplify the echo signal received by the receiving antenna of the millimeter wave radar 102. Then, one or more analog-to-digital converters (ADCs) are used to digitize the echo signal for further processing. Other implementations are also possible.

[0040] Figure 2 1 shows a schematic diagram of a millimeter wave radar 102 according to an embodiment of the present invention. The millimeter wave radar 102 includes a millimeter wave radar sensor circuit 200, a transmitting antenna 212, and a receiving antenna 214. The millimeter wave radar sensor circuit includes a phase-locked loop (PLL) 2104, a voltage-controlled oscillator (VCO) 206, a frequency divider 208, an amplifier 210, a mixer 216, a low-pass filter (LPF) 218, and an ADC 220.

[0041] During normal operation, VCO 206 generates a linear frequency chirp (e.g., from 57 GHz to 64 GHz) that is transmitted by transmit antenna 212. VCO is controlled by PLL 204, which receives a reference clock signal (e.g., 80 MHz) from reference oscillator 202. PLL 204 is controlled by a loop including divider 208 and amplifier 210.

[0042] The linear chirp emitted by the transmit antenna 212 is reflected by the hand 114 and received by the receive antenna 214. The echo received by the transmit antenna 214 is mixed with the signal emitted by the transmit antenna 212 using the mixer 216 to produce an intermediate frequency (IF) signal x(t) (also referred to as a beat signal). The beat signal x(t) is low-pass filtered by the LPF 218 and then sampled by the ADC 220. The ADC 220 is advantageously capable of sampling the beat signal x(t) at a sampling frequency that is much lower than the frequency of the signal received by the receive antenna 214. Therefore, in some embodiments, the use of an FMCW radar advantageously allows a compact and low-cost implementation of the ADC 220.

[0043] The propagation delay between the signal transmitted by the transmit antenna 212 and the signal received by the receive antenna 214 may be identified by determining the beat frequency of the beat signal x(t). The beat frequency of the beat signal x(t) may be identified by spectrum analysis using the processor 104 (eg, using FFT).

[0044] In an embodiment of the present invention, the millimeter wave radar performs fine range estimation based on the frequency and phase of the beat signal.

[0045] The beat signal x(t) can be given by the following equation:

[0046]

[0047] where f b is the beat frequency and is the difference between the signal transmitted by the transmit antenna 212 and the signal received by the receive antenna 214 .

[0048] The beat signal can be given by the following equation:

[0049]

[0050] Where B is the rise bandwidth of the linear chirp emitted by the transmit antenna 212, R is the range or distance to the hand 114, c is the speed of light, and T c is the linear chirp duration.

[0051] The speed of light c is a known constant, and the bandwidth B and the duration T of the linear chirp c is a known quantity for a particular system. For example, in some embodiments, the millimeter wave radar 102 is characterized by having an elevation field of view (FoV) of 70°, an azimuth FoV of 70°, a rise start frequency fmin of 57 GHz, a ramp end frequency f of 63 GHz, and a slope end frequency f max , 6GHz(f max –f min ) bandwidth B, chirp time T of 171.2μs c , the sampling frequency f is 0.747664MHzs , and 128 per chirp N s The number of samples. Since the beat frequency f can be estimated b , for example, using spectrum analysis of the beat signal x(t), and the quantities B, c, and T c is known, so the range R can be estimated by using Equation 2.

[0052] Beat signal φ b The phase can be given by the following equation:

[0053]

[0054] Since Equation 3 also includes the range R, the range estimate can be improved by using Equations 2 and 3. For example, Figure 3 A flow chart of an embodiment method 300 of fine distance estimation according to an embodiment of the present invention is shown. For example, the method 300 may be implemented by the millimeter wave radar 102 .

[0055] During step 302, the beat frequency estimate is estimated, for example using equation 1 and a Fourier transform (such as an FFT). During step 304, the beat frequency estimated in step 302 is estimated, for example using equation 2. To estimate the rough range estimate R fb During step 306, for example, R is estimated using equation 3 and the rough range estimated in step 304. fb To estimate the beat signal phase

[0056] During step 308, the beat signal is demodulated. Beat signal It can be given by the following equation:

[0057]

[0058] in is the demodulated signal. Replacing the beat signal x(t) in Equation 4 with the beat signal x(t) in Equation 1 yields:

[0059]

[0060] or

[0061]

[0062] in and

[0063] When the beat frequency f b When there is only a single component, the demodulated signal in Equation 6 The phase Δφ b With slope Δf b and y intersection Δφ b During step 310, for example, the demodulated signal is used to determine the demodulated beat phase Δφ b .

[0064] During step 312, the range ΔR is determined, for example, using Equation 3. For example, ΔR may be determined by the following equation:

[0065]

[0066] During step 314 , a fine estimation range R is determined based on ΔR. fb,φb For example, the fine estimation range R fb,φb It can be estimated by the following equation:

[0067] R fb,φb =R fb +ΔR (8)

[0068] In an embodiment of the present invention, three millimeter wave radars form a millimeter wave radar network. Each of the three millimeter wave radars has a corresponding FoV pointing to the monitoring space. When an object such as a hand 114 enters the monitoring space, each of the three millimeter wave radars performs a fine range estimation to determine the range of the hand 114 relative to the corresponding millimeter wave radar. The trilateral measurement technology based on the fine range estimation of each of the three millimeter wave radars is used to determine the position of the hand 114 in the monitoring space.

[0069] In some embodiments, the radar network is non-coherent, and each of the millimeter wave radars in the millimeter wave radar operates independently of each other when monitoring the same target. Using data from the radar network, processing for locating a target (e.g., a finger) in a monitoring space (e.g., a finger) using data from the network of radars can be performed centrally (e.g., using a central processor) or distributedly (e.g., using multiple processors, such as processors of the millimeter wave radars of the radar network).

[0070] Figure 4 A millimeter wave radar 402 with a FoV 404 according to an embodiment of the present invention is shown. The millimeter wave radar 402 can be implemented in a similar manner to the millimeter wave radar 102. The FoV 404 is based on the elevation angle FoVθ ver and azimuth angle FoVθ hor As in Figure 4 As shown in the figure, the vertical distance D ver Can be Given, and the horizontal distance D hor Can be Given as, where h is the distance between the millimeter wave radar 402 and the plane 406, and where the vertical distance D ver and horizontal distance D hor In plane 406.

[0071] The FoV of the millimeter wave radar 402 can be extended by rotating the millimeter wave radar by an angle φ. For example, Figure 5 A millimeter wave radar 402 with a FoV 504 according to an embodiment of the present invention is shown. The distance d h d V1 and d V2 In plane 406. Distance d h Can be Given, the distance d V1 It can be given by the following equation:

[0072] And the distance d V2 It can be given by the following equation:

[0073]

[0074] Figure 6 A network of millimeter wave radars 600 according to an embodiment of the present invention and a corresponding monitoring space 608 are shown. The network of millimeter wave radars 600 includes millimeter wave radars 602, 604, and 606. Each of the millimeter wave radars 602, 604, and 606 can be implemented in a manner similar to the millimeter wave radar 102, and each millimeter wave radar can have a FoV based on rotation or without rotation. Although the network of millimeter wave radars 600 only shows three millimeter wave radars, it is understood that the network of millimeter wave radars 600 can include more than three millimeter wave radars (such as: four, five, six or more).

[0075] As in Figure 6 As shown in FIG. 1 , millimeter wave radars 602, 604, and 606 are located at points R1, R2, and R3, respectively, and millimeter wave radars 602, 604, and 606 have corresponding coordinates (x1, y1, z1), (x2, y2, z2), and (x3, y3, z3). In some embodiments, R1, R2, and R3 are not located on a straight line (i.e., not on a straight line).

[0076] The monitoring space 608 is formed at the corresponding intersections of the millimeter wave radars 602, 604 and 606 with the FoV 602 、FoV 604 and FoV 606 In some embodiments, the monitoring space 608 has a width of about 10 cm 3In some embodiments, the monitoring space 608 may have a volume greater than 10 cm 3 Volume (such as 11cm 3 、13cm 3 、15cm 3 or larger), or less than 10cm 3 Volume (such as 9cm 3 、7cm 3 or smaller).

[0077] During normal operation, millimeter wave radars 602, 604, and 606 monitor monitoring space 608. When an object, such as hand 114 or fingers of hand 114, enters monitoring space 608, millimeter wave radars 602, 604, and 606 are used to determine the position of the object using trilateration.

[0078] When the target is located at a point T (x, y, z), where the point T (x, y, z) is located at distances D1, D2, and D3 from the millimeter wave radars 602, 604, and 606, respectively, the coordinates of T can be obtained by the following set of equations:

[0079]

[0080] The matrix form of this system of equations is given by the following matrix:

[0081]

[0082] This matrix group can be written in the following form:

[0083] A x = b (11)

[0085] in x ∈E, and among them

[0086] When R1, R2, and R3 are not located on a straight line, the range (A) is 3 and dim(Kern(A))=1, and the solution of equation 11 can be given by the following equation:

[0087] x = xp +αx h (12)

[0089] in x p is a specific solution of Equation 11, x h is a homogeneous system of equations A x 0, and α is a real parameter.

[0090] The real parameter α can be obtained by using Equation 12 to generate Equation 13

[0091]

[0092] where x p =(x p0 ,x p1 ,x p2 ,x p3 ) T 、x h =(x h0 ,x h1 ,x h2 ,x h3 ) T ,and x =(x0,x1,x2,x3) T .because x ∈E, so

[0093] x p0 +α·x h0 =(x p1 +α·x h1 ) 2 +(x p2 +α·x h2 ) 2 +(x p3 +α·x h3 ) 2 (14)

[0094] And therefore

[0095]

[0096] Equation 15 is aα 2 +bα+c=0, the solution of which is given by the following equation:

[0097]

[0098] Some embodiments may implement trilateration techniques in other ways known in the art.

[0099] Figure 7 FIG. 6 is a schematic diagram of a network of millimeter wave radars 600 according to an embodiment of the present invention. When monitoring an object in a monitoring space 608, each millimeter wave radar in the network of millimeter wave radars 600 may operate in an independent and incoherent manner.

[0100] As in Figure 7, the network of millimeter wave radars 600 includes a processor 702. The processor 702 receives a fine distance estimate from each of the millimeter wave radars 602, 604, and 606, and the processor 702 determines the position of an object in the monitoring space 608 by using trilateration. In some embodiments, the processor 702 receives data from the millimeter wave radars 602, 604, and 606 wirelessly.

[0101] Processor 702 can be implemented as a general-purpose processor, a controller or a digital signal processor (DSP), for example, processor 702 includes a combination circuit coupled to a memory. In some embodiments, for example, ARM architecture can be used to implement DSP. In some embodiments, processor 702 can be implemented as a customized application-specific integrated circuit (ASIC). In some embodiments, processor 702 includes multiple processors, each processor having one or more processing cores. In other embodiments, processor 702 includes a single processor having one or more processing cores. Other implementations are also possible. For example, some embodiments can be implemented as a combination of a hardware accelerator and software running on a DSP or a general-purpose microcontroller. In some embodiments, processor 702 can be implemented together with processor 104 of a millimeter wave radar in millimeter wave radar 602, 604 or 606.

[0102] In some embodiments, the processor 702 determines multiple positions of an object over time based on data received from the millimeter wave radars 602, 604 and 606, and the processor 702 uses a filter such as a Kalman filter or any other smoothing filter to generate a smooth trajectory of the object in the monitoring space 608 over time in a two-dimensional (2D) plane.

[0103] When the data input (e.g., measurement values) contains unpredictable or random errors, uncertainties, or variations, the Kalman filter can be viewed as an iterative process that uses a set of equations and continuous data input to estimate the position, velocity, etc. of an object. In an embodiment, a constant acceleration model is used for motion sensing and trajectory generation in a 2D plane. For example, the position, velocity, and acceleration of an object monitoring space 608 in a 2D plane can be obtained by the vector x gives:

[0104]

[0105] as well as

[0106]

[0107] in x k is at time k x state, where δ tis the time step.

[0108] The observation model that maps the estimated state to the observation space can be given by the following equation:

[0109]

[0110] And the 2D position z (x, y) can be estimated by the millimeter wave radar 600 using the trilateration technique. Processing noise can make the uncertainty of the position of the sensed object estimated by the state transition model of the Kalman filter. Processing noise can be modeled using the processing covariance matrix Q, which can be given by the following equation:

[0111]

[0112] in and It is the acceleration processing noise.

[0113] The position estimated by trilateration may be noisy. The noise may be due to, for example, thermal noise from millimeter wave radars 602, 604, and 606. The measurement noise due to noisy sensor measurements (e.g., the noise associated with generating a fine range estimate) may be given by the following equation:

[0114]

[0115] where R represents the variance of the measurement noise on each axis.

[0116] Figure 8 A flow chart of an exemplary method 800 for tracking the trajectory of an object in the monitoring space 608 using a Kalman filter according to an embodiment of the present invention is shown. The method 800 includes an initialization step 802, a prediction step 804, and a calibration / update step 810. For example, the method 800 may be implemented by the processor 702.

[0117] During step 802, covariance matrices P, Q, and R are initialized, where covariance matrix P corresponds to the estimated covariance of the position of the object, covariance matrix Q corresponds to the covariance of the processing noise, and covariance matrix R corresponds to the covariance of the measurement noise. In some embodiments, the initial values ​​of matrices P, Q, and R are systematically determined based on, for example, an experimental setup. In other embodiments, the initial values ​​of matrices P, Q, and R are set to predetermined default values ​​(such as, a scaled identity matrix or other values).

[0118] During prediction step 804, a prediction of the position of the object is made based on the current state of the Kalman filter. Step 804 includes steps 806 and 808. During step 806, the state transition can be determined by the following equation:

[0119]

[0120] During step 808, the covariance projection of the position may be determined by the following equation:

[0121]

[0122] During correction / update step 810, the estimate of position is updated based on the fine range estimate from the network of millimeter wave radars 600 and the estimated position from step 804. Step 810 includes 812, 814, 816, and 818.

[0123] In step 812, a fine range estimate z of the location of the target is received from the network of millimeter wave radars 600. k During step 814, based on the covariance matrix P k and R to determine the Kalman gain K k , and the Kalman gain can be given by the following equation:

[0124]

[0125] During step 816, based on the fine range estimate z k and the Kalman gain K k To update the estimate x k .estimate x k It can be given by the following equation:

[0126]

[0127] During step 818, the covariance matrix Pk is updated, for example by the following equation:

[0128]

[0129] As in Figure 8 As shown, after step 818, k is incremented by one and the sequence is repeated.

[0130] Some embodiments may implement a Kalman filter in other ways known in the art.

[0131] Using a Kalman filter advantageously allows smoothing of the trajectory of the object in the detection space.

[0132] In an embodiment of the present invention, during a training phase, a neural network, such as an LSTM neural network (e.g., a unidirectional or bidirectional LSTM), is trained using a set of gestures of an object (e.g., a metal tag, a hand, or a finger) moving in the monitoring space 608. During normal operation, the LSTM neural network is then used to associate (e.g., assign labels) the smoothed trajectories generated by the Kalman filter with character symbols (e.g., Latin characters), wherein the smoothed trajectories generated by the Kalman filter correspond to gestures performed by objects in the monitoring space 608 during normal operation.

[0133] LSTM neural networks can be understood as recurrent networks that include a memory that is used to model temporal dependencies. The activation of a neuron is fed back to itself via a weight value and a unit time delay, which provides the neuron with a memory value (hidden value) of past activations, enabling the neuron to learn the temporal dynamics of sequence data. Given a time input sequence of length T A recurrent neural network (RNN) can map this to a sequence of hidden values And output activation sequence by iteration

[0134]

[0135] Where ρ is a nonlinear activation function, is the hidden bias vector and the W term represents the weight matrix, is the input hidden weight matrix, and is the hidden weight matrix. The activation of the recurrent unit can be given by the following equation:

[0136]

[0137] in, is the activation-hidden weight matrix, and is the bias activation vector.

[0138] LSTM extends RNNs with memory cells by using gating. Gating can be understood as a mechanism based on component-wise multiplication of inputs that controls the behavior of each individual memory cell of the LSTM. LSTM updates its cell state based on the activation of the gates. The inputs provided to the LSTM are fed into different gates that control which operations are performed on the memory cells. Operations include writing (input gate), reading (output gate), and resetting (forget gate). The vector representation of the LSTM update (a vector of all cells in a layer) can be given by the following equation:

[0139]

[0140] where i, f, o, and c are the input gate, forget gate, output gate, and unit activation vector, respectively, all of which can have the same size as the vector h representing the hidden values. The term σ represents a nonlinear function (e.g., a sigmoid function). The term a t is the input of the memory cell at time t, and in some embodiments, a t W may be the position output from the Kalman filter at time k. ai , W hi , W ci , W af , W hf , W cf , W ac , W hc , W ao , W ho and W co is a weight matrix, where the subscripts denote from-to relationships and the term b i , b f , b c and b o is the bias vector.

[0141] The input weights may be input states of the LSTM, and the input weights may be determined during the training phase. In some embodiments, the input weights of the LSTM are the variables x, y, and The Kalman filter state of the LSTM can be the loss determined in a single sequence step, which can be the average of the logarithmic losses determined individually for each label, which can be given by the following equation:

[0142]

[0143] in is the predicted classification likelihood or label likelihood, y is the actual classification or label, and L is the number of possible classifications or labels.

[0144] Fig. 9 FIG. 9 is a schematic diagram of an LSTM network 900 according to an embodiment of the present invention. The LSTM network 900 includes a plurality of units (unit 1, unit 2, ..., unit T).

[0145] As in Fig. 9 As shown in the figure, each cell i Receive data x from the output of the Kalman filter i(e.g., the position coordinates of the target at each frame.) For example, if the Kalman filter generates 100 position estimates over time (where the 100 position estimates are associated with the trajectory of the object in monitoring space 608), then T is equal to 100, and each cell i Receive the corresponding position estimate x i The LSTM network 900 then associates the trajectory received from the Kalman filter with a label (such as a Latin character symbol or a number symbol).

[0146] Fig.10 The trajectories of metal markers in the monitoring space 608 and the corresponding labels generated by the LSTM network 900 according to an embodiment of the present invention are illustrated. The x-axis and y-axis illustrate the distance (in meters) relative to the origin (0,0). In some embodiments, the origin (0,0) corresponds to the position of one of the millimeter wave radars (e.g., the millimeter wave radar 602). Other reference positions may also be used.

[0147] Although metal markers are used to generate Fig.10 In addition to the trajectories illustrated in FIG. 1 , other objects (such as fingers of hand 114 ) may be used.

[0148] As in Fig.10 As shown in FIG. 6 , the monitoring space 608 may be positioned between 0.2 m and 0.6 m from one or more millimeter wave radars. In some embodiments, the nearest edge of the monitoring space 608 may be closer to one or more millimeter wave radars (e.g., 0.15 m or closer), or farther away from one or more millimeter wave radars (e.g., 0.5 m or farther). In some embodiments, the farthest edge of the monitoring space 608 may be closer than 0.6 m (e.g., 0.5 m or closer) or farther than 0.6 m (e.g., 0.7 m or farther).

[0149] In some embodiments, the object's trajectory is a depiction of a character symbol, such as Fig.10 900. A neural network (such as LSTM network 900) maps such a depiction to a character symbol. In other embodiments, an approximate depiction of a character symbol, a shorthand depiction of a character symbol, or an arbitrary gesture may be mapped to a character symbol. For example, in some embodiments, a trajectory of a straight line may be associated with a character symbol α.

[0150] Some embodiments may implement the LSTM network in other ways known in the art.

[0151] Fig.11 The flowchart of an example method 1100 for character recognition in air writing according to an embodiment of the present invention is shown. The method 1100 includes a position extraction phase 1101 and a recognition phase 1111.

[0152] During normal operation, the millimeter wave radars 602, 604, and 606 respectively perform steps 1102. 602 ,1102 604 and 1102 606 During this time, radar signals are transmitted and received to the monitoring space 608. The received radar signals may be filtered and digitized using, for example, LPF 218 and ADC 220.

[0153] In step 1103 602 ,1106 604 and 1103 606 During this time, background removal and filtering are performed, for example, by the corresponding processor 104. For example, in step 1102 602 ,1102 604 and 1102 606 The digitized radar data generated during the period can be filtered, and for example, the DC component can be removed to remove the self-interference of the transmitter-receiver and optionally pre-filter the colored interference noise. In some embodiments, the filtering includes removing data outliers with values ​​significantly different from other adjacent range gate measurements. Therefore, the filtering is also used to remove background noise from the radar data. In a specific example, a Hampel filter applies a sliding window at each range gate to remove such outliers. Alternatively, other filtering methods known in the art for range preprocessing can be used.

[0154] When an object enters the monitoring space 608, the millimeter wave radars 602, 604, and 606 detect the object based on the detection result in step 1102. 602 ,1102 604 and 110 2606 The radar signals received during the period are respectively and in step 1103 602 ,1106 604 and 1103 606 During the filtering of the radar signal, in step 1104 602 ,1104 604 and 1104 606 For example, in some embodiments, objects are detected by using a range FFT on the radar data, and the object is detected when the range bin is above a predetermined threshold. Other object detection methods may also be used.

[0155] In some embodiments, not all detected objects are selected for tracking and further processing. For example, in some embodiments, the Doppler velocity of the detected object is determined. If the Doppler velocity is within a predetermined Doppler velocity range (e.g., the velocity range of a typical human hand movement), millimeter wave radars 602, 604, and 606 can select the object for further processing. If the Doppler velocity exceeds the predetermined Doppler velocity range, such as too slow (e.g., static) or too fast (e.g., 10m / s), the object can be ignored and not selected for further processing. In some embodiments, as an alternative or additional, in addition to the Doppler velocity, other attributes (such as the size of the detected object) can be used to select objects for further processing.

[0156] At step 1106 602 ,1106 604 and 1106 606 During this time, the millimeter wave radars 602, 604, and 606 each generate a fine distance estimate of the distance toward the detected and selected object (e.g., the fingertips of the hand 114) using, for example, method 300. During step 1107, the processor 702 uses trilateration to determine the position of the object in the monitoring space 608 (such as described with respect to equations 9 to 16). The processor 702 also generates a track of the determined positions over time (e.g., by storing each determined position in a local memory).

[0157] During step 1108, processor 702 smoothes the trajectory (track) generated during step 1107 and tracks the object over time (eg, using a Kalman filter, such as using method 800). In some embodiments, other smoothing filters may be used.

[0158] During step 1112 , processor 702 uses an LSTM network (such as LSTM network 900 ) to associate the trajectory generated by the Kalman filter in step 1108 with the character symbol and generates an output (label) based on the association.

[0159] In some embodiments, the neural network used in step 1112 is implemented as a unidirectional or bidirectional LSTM network. In other embodiments, other types of neural networks may be used. For example, in some embodiments, a convolutional neural network (CNN) is used during step 1112 to generate outputs (labels) based on the trajectories generated by the Kalman filter in step 1108. For example, in some embodiments, a CNN similar to that in step 1108 is generated during step 1108. Fig.10 Then, the CNN generates a label based on the 2D image generated in step 1108 during step 1112.

[0160] In some embodiments, the processor 702 determines the start and end of the trajectory based on fixed boundaries to generate a segment of the trajectory. The processor 702 then feeds the segment of the trajectory to the LSTM network 900 for classification purposes. For example, Fig.12 A flow chart of an embodiment method 1200 for character recognition in airwriting using frame-by-frame classification is shown in accordance with an embodiment of the present invention.

[0161] During step 1202, the network of millimeter wave radar 600 detects that an object (such as a fingertip of hand 114) enters monitoring space 608. During step 1204, a smooth trajectory of the object in monitoring space 608 is generated for a fixed number of frames (such as 100 frames in 2 seconds). For example, it can be as in Fig.11 A smooth trajectory is obtained as described in step 1101. A different number of frames may be used, such as 128, 200, 96, 64 or other number of frames. A different duration may also be used, such as 1.5 seconds, 2.5 seconds or other duration.

[0162] During step 1206, the LSTM network receives a fixed number of frames and, for example, Fig.11 As described in step 1112, an output is generated.

[0163] Fig.13 A flow chart is shown of an embodiment method 1300 for character recognition in airwriting using frame-by-frame classification in accordance with an embodiment of the present invention.

[0164] During step 1304, a smooth trajectory of the object in the monitoring space 608 is generated for a boundary number of frames. For example, the boundary may be determined by detecting that the object exits the monitoring space 608. The number of frames in such a boundary segment may be different for each trajectory.

[0165] In step 1306, the LSTM network receives a boundary number of frames and, for example, as described with respect to Fig.11 As described in step 1112, an output is generated.

[0166] In some embodiments, the processor 702 uses a connectionist temporal classification (CTC) layer to process the unsegmented data stream from the output of the LSTM network 900, wherein the unsegmented data stream is an unsegmented stream based on the position of an object that changes over time. The output of the LSTM network 900 is converted to a conditional probability distribution on a sequence of labels. The total probability of any one sequence of labels can be found by summing the probabilities of different arrangements. The processor 702 then segments the track based on the output of the CTC layer and associates the segmented track with a character symbol. The general operation of the CTC layer is known in the art.

[0167] Fig.14 A flow chart of an embodiment method 1400 for character recognition in air writing using CTC classification according to an embodiment of the present invention is shown.

[0168] During step 1404, a smooth trajectory of an object in the monitoring space 608 is continuously generated and fed to an LSTM network with a CTC layer. During step 1406, the LSTM network with a CTC layer finds frame arrangements with higher probabilities in real time based on possible LSTM labels (e.g., a set of possible character symbols plus blanks), segments the frames according to the arrangement, and outputs associated character symbols corresponding to the segmented trajectories. In this way, the processor 702 is advantageously able to generate a continuous sequence of character symbols based on the trajectory of the object in the monitoring space 608 without having to use predetermined time boundaries. By using an LSTM with CTC, the processor 702 is advantageously able to accurately associate character symbols with object trajectories in the monitoring space 608, regardless of whether the object is moving quickly or slowly.

[0169] Some embodiments may implement LSTM networks with CTC in other ways known in the art.

[0170] It should be understood that methods 1200, 1300, and 1400 can be combined in a variety of ways. For example, in some embodiments, processor 702 can begin monitoring an object when the object enters monitoring space 608, and continuously generate character symbols using LSTM with CTC for a predetermined number of frames (e.g., 500 or more) or until the object exits monitoring space 608.

[0171] Fig.15 A smartphone 1502 having a network of millimeter wave radars 600 for character recognition in air writing according to an embodiment of the present invention is shown. For example, the network of millimeter wave radars 600 can implement any of the methods 1100, 1200, 1300, and 1400 for character recognition in air writing.

[0172] As in Fig.15 As shown in FIG. 1 , the smartphone 1502 may have millimeter wave radars in three of the four corners of the smartphone 1502. In some embodiments, the smartphone 1502 may have more than three radars, such as four radars (e.g., one radar in each corner) or more. In some embodiments, the millimeter wave radars may be positioned in other locations (such as, positioned in the edge of the screen 1504).

[0173] In some embodiments, the outer edge of monitoring space 608 closest to screen 1504 is at 10 cm from screen 1504. In some embodiments, the distance between the outer edge of monitoring space 608 closest to screen 1504 and screen 1504 is less than 10 cm (such as 9 cm, 8 cm, or closer). In some embodiments, the distance between the outer edge of monitoring space 608 closest to screen 1504 and screen 1504 is greater than 10 cm (such as 11 cm, 14 cm, or more).

[0174] For example, screen 1504 may be a 5.2 inch screen. In some embodiments, screens such as 5.1 inches, 5 inches, 4.7 inches, or smaller may be used. Screens of 5.3 inches, 5.5 inches, or larger may be used.

[0175] In some embodiments, other devices, such as devices larger than the smartphone 1502, may implement a network of millimeter wave radars 600 for character recognition in airwriting. For example, in some embodiments, a tablet computer, a computer monitor, or a television may implement a network of millimeter wave radars 600 in a manner similar to the smartphone 1502. For example, Fig.16 A computer display 1602 for character recognition in airwriting with a network of millimeter wave radars 600 is shown according to an embodiment of the present invention.

[0176] As in Fig.16 As shown in FIG. 1 , the computer display 1602 may have millimeter wave radars in three of the four corners of the computer display 1602. In some embodiments, the computer display 1602 may have more than three radars, such as four radars (e.g., one radar in each corner) or more. In some embodiments, the millimeter wave radars may be positioned in other locations (such as, positioned in the edge of the screen 1604).

[0177] In some embodiments, the outer edge of monitoring space 608 closest to screen 1604 is 20 cm from screen 1504. In some embodiments, the distance between the outer edge of monitoring space 608 closest to screen 1204 and screen 1204 is less than 20 cm (such as 18 cm, 15 cm, or closer). In some embodiments, the distance between the outer edge of monitoring space 608 closest to screen 1604 and screen 1604 is greater than 20 cm (such as 25 cm, 30 cm, or more).

[0178] For example, screen 1604 may be a 27 inch screen. In some embodiments, screens such as 24 inches, 21 inches, 20 inches, or smaller may be used. Screens such as 34 inches or larger may be used.

[0179] Example embodiments of the present invention are summarized here. Other embodiments may also be understood from the entirety of the specification and the claims submitted herein.

[0180] Example 1. A method for air writing character recognition, the method comprising: determining a position of an object in a monitoring space using trilateration by using a plurality of millimeter wave radars, wherein each of the plurality of millimeter wave radars has a field of view, and wherein the intersection of the fields of view of the plurality of millimeter wave radars forms the monitoring space; tracking a position of an object that changes over time in the monitoring space using the plurality of millimeter wave radars; determining a character symbol depicted by the tracked position of the object that changes over time using a neural network (NN); and providing a signal based on the determined character symbol.

[0181] Example 2. The method according to example 1, wherein the character symbol is determined based on a tracked position of the object during a predetermined fixed time.

[0182] Example 3. The method of example 1 or 2, wherein the character symbol is determined based on the tracked position of the object during the boundary time.

[0183] Example 4. The method of any one of Examples 1 to 3 further includes determining a boundary time based on when the detected object enters and leaves the monitored space.

[0184] Example 5. The method according to any one of Examples 1 to 4, wherein the NN comprises a recursive NN (RNN).

[0185] Example 6. The method of any one of Examples 1 to 5, wherein the RNN comprises a Long Short Term Memory (LSTM) network.

[0186] Example 7. The method according to any one of Examples 1 to 6 further includes: receiving an unsegmented stream of positions of objects that change over time; and generating a sequence of character symbols using connectionist temporal classification (CTC) based on the unsegmented stream of positions.

[0187] Example 8. The method of any one of Examples 1 to 7, wherein the NN comprises a convolutional neural network (CNN).

[0188] Example 9. The method of any one of Examples 1 to 8, wherein the character symbol comprises a number or a Latin character.

[0189] Example 10. The method of any one of Examples 1 to 9, wherein tracking the position of the object comprises: determining a trajectory of the object based on a plurality of determined positions of the object over time; and smoothing the determined trajectory using a smoothing filter.

[0190] Example 11. The method according to any one of Examples 1 to 10, wherein the smoothing filter comprises a Kalman filter.

[0191] Example 12. The method of any one of Examples 1 to 11, wherein the object comprises a human hand.

[0192] Example 13. The method of any of Examples 1 to 12, wherein the object comprises a human finger.

[0193] Example 14. The method of any of Examples 1 to 13, wherein the object comprises a metallic object.

[0194] Example 15. The method of any one of Examples 1 to 14, wherein three of the plurality of millimeter wave radars are not located in a straight line.

[0195] Example 16. The method according to any one of Examples 1 to 15, wherein the monitoring space is located at a distance between 0.3 m and 0.6 m from each of the plurality of millimeter wave radars.

[0196] Example 17. The method according to any one of Examples 1 to 16 further includes: transmitting a chirp to the monitoring space; receiving an echo of the chirp from the monitoring space; generating a beat signal based on the echo of the chirp; determining a beat frequency based on the beat signal; determining a first range based on the beat frequency; determining a beat frequency signal phase based on the first range; generating a demodulated beat signal based on the determined beat frequency and the determined beat signal phase; determining a demodulated beat phase based on the demodulated beat signal; determining a demodulated beat range based on the demodulated beat phase; determining a fine range estimate based on the demodulated beat frequency range; and determining a position of the object based on the fine range estimate.

[0197] Example 18. An air-written character recognition system includes: a plurality of millimeter-wave radars, wherein each of the plurality of millimeter-wave radars is configured to have a field of view, and wherein the intersection of the fields of view of the plurality of millimeter-wave radars forms a monitoring space; and a controller configured to: determine a position of an object in the monitoring space based on outputs of the plurality of millimeter-wave radars by using trilateration, track a position of the object that changes over time in the monitoring space based on the position determined using the plurality of millimeter-wave radars, determine a character symbol depicted by the tracked position of the object that changes over time using a neural network (NN), and provide a signal according to the determined character symbol.

[0198] Example 19. The airwriting character recognition system according to Example 18, wherein the character symbol is determined based on the tracked position of the object during the boundary time.

[0199] Example 20. The air writing character recognition system according to any one of Examples 18 or 19, wherein the controller is further configured to: generate an unsegmented stream of positions of objects that change over time; and generate a sequence of character symbols using connectionist temporal classification (CTC) based on the unsegmented stream of positions.

[0200] Example 21. The airwriting character recognition system according to any one of Examples 18 to 20, wherein the controller is configured to track the position of the object using a Kalman filter.

[0201] Example 22. The airwritten character recognition system according to any one of Examples 18 to 21, wherein the NN comprises a long short-term memory (LSTM) network.

[0202] Example 23. A millimeter wave radar system, comprising: three millimeter wave radars, wherein each of the three millimeter wave radars is configured to have a field of view, wherein the intersection of the fields of view of the three millimeter wave radars forms a monitoring space; and a controller, which is configured to: determine a position of an object in the monitoring space based on outputs of the three millimeter wave radars by using trilateration, determine a trajectory of the object that changes over time based on the determined position of the object, apply a filter to the determined trajectory to generate a filtered trajectory, determine a character symbol depicted by the filtered trajectory using a long short-term memory (LSTM), and provide a signal based on the determined character symbol.

[0203] Example 24. A smart phone includes a millimeter wave radar system and a screen. The millimeter wave radar system includes: three millimeter wave radars. Each millimeter wave radar is positioned at a corner of the smart phone. Each of the three millimeter wave radars is configured to have a field of view, wherein the intersection of the fields of view of each of the three millimeter wave radars forms a monitoring space. The smart phone also includes a controller configured to: determine the position of an object in the monitoring space based on the outputs of the three millimeter wave radars by using trilateral measurement, determine the trajectory of the object that changes over time based on the determined position of the object, apply a filter to the determined trajectory to generate a filtered trajectory, determine the character symbol depicted by the filtered trajectory using a long short-term memory (LSTM), and provide a signal based on the determined character symbol.

[0204] Example 25. A computer display comprising a millimeter wave radar system and a screen. The millimeter wave radar system comprises: three millimeter wave radars. Each of the three millimeter wave radars is positioned at a corner of the computer display. Each of the three millimeter wave radars is configured to have a field of view, wherein the intersection of the fields of view of each of the three millimeter wave radars forms a monitoring space. The computer display also includes a controller configured to: determine the position of an object in the monitoring space based on the outputs of the three millimeter wave radars by using trilateral measurement, determine the trajectory of the object that changes over time based on the determined position of the object, apply a filter to the determined trajectory to generate a filtered trajectory, determine the character symbol depicted by the filtered trajectory using a long short-term memory (LSTM), and provide a signal based on the determined character symbol.

[0205] Although the present invention has been described with reference to illustrative embodiments, this description is not intended to be interpreted in a limiting sense. Various modifications and combinations with reference to the illustrative embodiments of the present invention and other embodiments of the present invention will be apparent to those skilled in the art. Therefore, the appended claims cover any such modifications or embodiments.

Claims

1. A method for identifying characters written in the air, the method comprising: Determining a position of an object in a monitoring space using trilateration by using a plurality of millimeter wave radars, wherein each of the plurality of millimeter wave radars has a field of view, and wherein an intersection of the fields of view of the plurality of millimeter wave radars forms the monitoring space, wherein determining the position of the object comprises: transmitting a chirp to the monitoring space; receiving an echo of the chirp from the monitoring space; generating a beat signal based on an echo of the chirp; determining a beat frequency based on the beat signal; determining a first range based on the beat frequency; determining a beat frequency signal phase based on the first range; generating a demodulated beat signal based on the determined beat frequency and the determined beat signal phase; determining a demodulated beat phase based on the demodulated beat signal; determining a demodulated beat range based on the demodulated beat phase; determining a fine range estimate based on the demodulated beat range; determining a position of the object based on the fine range estimate; Tracking the position of the object in the monitoring space as it changes over time using the plurality of millimeter-wave radars; determining, using a neural network NN, a character symbol depicted by the tracked position of the object over time; and A signal is provided based on the determined character symbol. 2 . The method of claim 1 , wherein the character symbol is determined based on the tracked position of the object during a predetermined fixed time. The method of claim 1 , wherein the character symbol is determined based on the tracked position of the object during a boundary time. 4 . The method of claim 3 , further comprising determining the boundary time based on detecting when the object enters and leaves the monitored space.

5. The method of claim 1, wherein the NN comprises a recursive NN (RNN).

6. The method of claim 5, wherein the RNN comprises a Long Short-Term Memory (LSTM) network.

7. The method according to claim 6, further comprising: receiving an unsegmented stream of positions of said objects as a function of time; as well as Based on the unsegmented stream of positions, a sequence of character symbols is generated using Connectionist Temporal Classification (CTC).

8. The method of claim 1, wherein the NN comprises a convolutional CNN.

9. The method of claim 1, wherein the character symbols include numbers or Latin characters.

10. The method of claim 1, wherein tracking the location of the object comprises: determining a trajectory of the object based on a plurality of determinations of the position of the object over time; as well as A smoothing filter is used to smooth the determined trajectory. The method of claim 10 , wherein the smoothing filter comprises a Kalman filter.

12. The method of claim 1, wherein the object comprises a human hand.

13. The method of claim 1, wherein the object comprises a human finger.

14. The method of claim 1, wherein the object comprises a metallic object. 15 . The method according to claim 1 , wherein three of the plurality of millimeter-wave radars are not located in a straight line. 16 . The method according to claim 1 , wherein the monitoring space is located at a distance between 0.3 m and 0.6 m from each of the plurality of millimeter-wave radars.

17. A system for recognizing characters written in the air, comprising: a plurality of millimeter wave radars, wherein each of the plurality of millimeter wave radars is configured to have a field of view, and wherein an intersection of the fields of view of the plurality of millimeter wave radars forms a monitoring space; as well as The controller is configured as: determining a position of an object in the monitoring space based on outputs of the plurality of millimeter-wave radars by using trilateration, tracking the position of the object changing over time in the monitoring space based on the position determined using the plurality of millimeter wave radars, determining, using a neural network NN, a character symbol depicted by the tracked position of the object over time, and providing a signal according to the determined character symbol, Each millimeter wave radar is configured as: transmitting a chirp to the monitoring space; receiving an echo of the chirp from the monitoring space; generating a beat signal based on an echo of the chirp; determining a beat frequency based on the beat signal; determining a first range based on the beat frequency; determining a beat frequency signal phase based on the first range; generating a demodulated beat signal based on the determined beat frequency and the determined beat signal phase; determining a demodulated beat phase based on the demodulated beat signal; determining a demodulated beat range based on the demodulated beat phase; determining a fine range estimate based on the demodulated beat range; A position of the object is determined based on the fine range estimate.

18. The airwriting character recognition system of claim 17, wherein the character symbol is determined based on the tracked position of the object during a boundary time.

19. The air writing character recognition system according to claim 18, wherein the controller is further configured to: generating an unsegmented stream of positions of the objects over time; and Based on the unsegmented stream of positions, connectionist temporal classification (CTC) is used to generate a sequence of character symbols.

20. The airwriting character recognition system of claim 18, wherein the controller is configured to track the position of the object by using a Kalman filter.

21. The air writing character recognition system according to claim 18, wherein the NN comprises a Long Short-Term Memory (LSTM) network.

22. A millimeter wave radar system, comprising: three millimeter-wave radars, each of the three millimeter-wave radars being configured to have a field of view, wherein an intersection of the fields of view of each of the three millimeter-wave radars forms a monitoring space; as well as The controller is configured as: determining the position of an object in the monitoring space based on the outputs of the three millimeter-wave radars by using trilateration, determining a trajectory of the object over time based on the determined position of the object, applying a filter to the determined trajectory to generate a filtered trajectory, determining the character symbol depicted by the filtered trajectory using a long short-term memory (LSTM), and providing a signal based on the determined character symbol, Each millimeter wave radar is configured as: transmitting a chirp to the monitoring space; receiving an echo of the chirp from the monitoring space; generating a beat signal based on an echo of the chirp; determining a beat frequency based on the beat signal; determining a first range based on the beat frequency; determining a beat frequency signal phase based on the first range; generating a demodulated beat signal based on the determined beat frequency and the determined beat signal phase; determining a demodulated beat phase based on the demodulated beat signal; determining a demodulated beat range based on the demodulated beat phase; determining a fine range estimate based on the demodulated beat range; A position of the object is determined based on the fine range estimate.

Images