Parametric cnn for radar processing

Abstract

Embodiments of the present disclosure relate to parameter CNNs for radar processing. In one embodiment, a method includes transmitting, using a millimeter wave radar sensor, a plurality of radar signals toward a target, receiving, using the millimeter wave radar, a plurality of reflected radar signals corresponding to the plurality of transmitted radar signals, mixing a copy of the plurality of transmitted radar signals with the plurality of received reflected radar signals to generate an intermediate frequency signal, generating, using an analog-to-digital converter, raw digital data based on the intermediate frequency signal, processing the raw digital data using a constrained L-dimensional convolutional layer of a neural network to generate intermediate digital data, where L is a positive integer greater than or equal to two, and where the neural network includes a plurality of additional layers, and processing the intermediate digital data using the plurality of additional layers to generate information about the target.

Description

Technical Field

[0001] This disclosure generally relates to an electronic system and method, and in a particular embodiment, to a parametric convolutional neural network (CNN) for radar processing. Background Technology

[0002] Over the past few years, applications in the millimeter-wave frequency domain have attracted significant interest due to the rapid development of low-cost semiconductor technologies, such as silicon-germanium (SiGe) and fine-geometry complementary metal-oxide-semiconductor (CMOS) processes. The availability of high-speed bipolar and metal-oxide-semiconductor (MOS) transistors has led to a growing demand for integrated circuits for millimeter-wave applications, such as 24 GHz, 60 GHz, 77 GHz, and 80 GHz, and beyond 100 GHz. Examples of such applications include automotive radar systems and multi-gigabit communication systems.

[0003] In some radar systems, the distance between the radar and the target is determined by transmitting a frequency-modulated (FM) signal, receiving the reflected FM signal (also known as the echo), and determining the distance based on the time delay and / or frequency difference between the transmission and reception of the FM signal. Therefore, some radar systems include: a transmitting antenna to transmit radio frequency (RF) signals; a receiving antenna to receive the reflected RF signals; and associated RF circuitry used to generate the transmitted signal and receive the RF signal. In some cases, multiple antennas can be used to implement directional beamforming using phased array technology. Multiple-input multiple-output (MIMO) configurations with multiple chip sets can also be used to perform coherent and incoherent signal processing. Summary of the Invention

[0004] According to an embodiment, a method includes: transmitting multiple radar signals toward a target using a millimeter-wave radar sensor; receiving multiple reflected radar signals corresponding to the multiple transmitted radar signals using a millimeter-wave radar; mixing copies of the multiple transmitted radar signals with the multiple received reflected radar signals to generate an intermediate frequency (IF) signal; generating raw digital data based on the IF signal using an analog-to-digital converter; processing the raw digital data using a constrained L-dimensional convolutional layer of a neural network to generate intermediate digital data, where L is a positive integer greater than or equal to 2, and where the neural network includes multiple additional layers; and processing the intermediate digital data using the multiple additional layers to generate information about the target.

[0005] According to an embodiment, a radar system includes: a millimeter-wave radar sensor, comprising: a transmitting antenna configured to transmit a plurality of radar signals toward a target; a receiving antenna configured to receive a plurality of reflected radar signals; a mixer configured to mix copies of the transmitted radar signals with the plurality of received reflected radar signals to generate an intermediate frequency (IF) signal; an analog-to-digital converter (ADC) configured to generate raw digital data at the output of the ADC based on the IF signal; and an artificial intelligence (AI) accelerator having an input coupled to the output of the ADC, and configured to: receive the raw digital data from the ADC, and process the raw digital data using an L-dimensional convolutional layer constrained by a neural network to generate intermediate digital data, wherein L is a positive integer greater than or equal to 2, and wherein the neural network includes a plurality of additional layers; and process the intermediate digital data using the plurality of additional layers to generate target-associated data at the output of the AI ​​accelerator.

[0006] According to an embodiment, a radar system includes: a millimeter-wave radar configured to transmit a plurality of chirped signals toward a target and receive a plurality of reflected chirped signals; a mixer configured to mix copies of the plurality of transmitted chirped signals with the plurality of received reflected chirped signals to generate an intermediate frequency (IF) signal; an analog-to-digital converter (ADC) configured to generate raw digital data at the output of the ADC based on the IF signal; and a processor having an input coupled to the output of the ADC and configured to: receive the raw digital data from the ADC and process the raw digital data using a neural network with a two-dimensional convolutional layer having a first constraint followed by a plurality of additional layers to generate data associated with the target at the output of the plurality of additional layers. Attached Figure Description

[0007] To gain a more complete understanding of the invention and its advantages, reference is now made to the following description taken in conjunction with the accompanying drawings, in which:

[0008] Figure 1 A schematic diagram of an exemplary radar system is shown;

[0009] Figure 2A The preprocessing and processing methods are shown. Figure 1 A flowchart illustrating the method for generating raw digital data using a millimeter-wave radar sensor;

[0010] Figure 2B The preprocessing and processing methods are shown. Figure 1 A flowchart of another method for generating raw digital data from millimeter-wave radar sensors;

[0011] Figure 3 A schematic diagram of a radar system according to an embodiment of the present invention is shown;

[0012] Figure 4 The invention is illustrated by an embodiment of the invention. Figure 3 A series of chirping signals emitted by the TX antenna;

[0013] Figure 5 A block diagram of a deep CNN (DCNN) for activity classification according to an embodiment of the present invention is shown;

[0014] Figure 6 The embodiments of the present invention are illustrated as being in the time domain and frequency domain. Figure 5 An example two-dimensional (2D) sinc filter used in a constrained convolutional layer;

[0015] Figure 7 The effect of the rotation angle α relative to the pseudo-range domain and the pseudo-Doppler domain according to an embodiment of the present invention is shown;

[0016] Figure 8 A block diagram of a DCNN according to an embodiment of the present invention is shown;

[0017] Figure 9 A block diagram of a DCNN for activity classification according to an embodiment of the present invention is shown;

[0018] Figure 10 The embodiments of the present invention are illustrated in the time and frequency domains, as shown. Figure 9 An exemplary two-dimensional (2D) Morlet wavelet filter used in a constrained convolutional layer;

[0019] Figures 11-17 The following is illustrated using an embodiment of the invention. Figure 3 The radar system achieves this. Figure 8 and Figure 9 Experimental setup or results of DCNN;

[0020] Figure 18 A block diagram of a DCNN for target and / or activity classification according to an embodiment of the present invention is shown;

[0021] Figure 19 A block diagram of a DCNN for target and / or activity classification according to an embodiment of the present invention is shown;

[0022] Figure 20 Embodiments of the present invention are shown respectively. Figure 20 An exemplary graph showing the frequency response of a first-constrained 2D convolutional layer and a second-constrained 2D convolutional layer;

[0023] Figure 21 An embodiment of the present invention is shown for generating 2D radar images. Figure 18 A block diagram of DCNN;

[0024] Figure 22An embodiment of the invention is shown in Figure 18 An example 2D radar image during the training of the DCNN;

[0025] Figure 23 A schematic diagram of a radar system according to an embodiment of the present invention is shown; and

[0026] Figure 24 A block diagram of a DCNN for generating target localization data according to an embodiment of the present invention is shown.

[0027] Unless otherwise specified, corresponding numbers and symbols in different figures generally refer to corresponding parts. The figures are drawn to clearly illustrate relevant aspects of the preferred embodiments and are not necessarily drawn to scale. Detailed Implementation

[0028] The manufacture and use of the disclosed embodiments are discussed in detail below. However, it should be understood that the present invention provides many applicable inventive concepts that can be embodied in a wide variety of specific environments. The specific embodiments discussed are merely illustrative of particular ways of making and using the invention and do not limit the scope of the invention.

[0029] The following description illustrates various specific details to provide a thorough understanding of several exemplary embodiments according to the description. Embodiments may be obtained without one or more specific details or in other ways, components, materials, etc. In other instances, known structures, materials, or operations have not been shown or described in detail so as not to obscure different aspects of the embodiments. References to "one embodiment" in this specification indicate that a particular configuration, structure, or feature described with respect to that embodiment is included in at least one embodiment. Therefore, phrases such as "in one embodiment" that may appear at different points in this specification do not necessarily refer precisely to the same embodiment. Furthermore, in one or more embodiments, particular forms, structures, or features may be combined in any suitable manner.

[0030] Embodiments of the invention will be described in a specific context. Deep CNNs (DCNNs) for (human) target classification or radar-based (human) target localization using millimeter-wave (mmWave) radar employ parametric two-dimensional (2D) or 3D CNN layers for receiving raw digital data from an analog-to-digital converter (ADC) of the millimeter-wave radar. In some embodiments, the DCNN uses an L-dimensional CNN layer for receiving the raw digital data, where L is a positive integer greater than or equal to 2. Some embodiments may additionally or alternatively generate other information about the target, either for target classification and / or target localization. For example, some embodiments may be implemented in radars with a different regime than millimeter waves and / or for targets other than human targets (such as animals or autonomous machines).

[0031] In embodiments of the invention, the DCNN receives raw digital data from the ADC of a millimeter-wave radar and processes the raw digital data to directly extract features for classification (e.g., human activity) without using conventional preprocessing methods (such as background mean subtraction, distance discrete Fourier transform, and / or Doppler fast Fourier transform). In some embodiments, the initial (first) layer of the DCNN (receiving raw digital data from the ADC) is implemented as a constrained 2D convolutional layer. In some embodiments, the constrained 2D convolutional layer uses a 2D sinc filter kernel. In other embodiments, the constrained 2D convolutional layer uses a 2D Morlet wavelet filter kernel. Other filter kernels, such as those based on fractional Fourier transform and discrete cosine transform, may also be used.

[0032] The potential applications of human perception and activity classification are growing across various sectors, such as personal safety, defense, and surveillance. For example, in the industrial and consumer sectors, human activity recognition can be used in smart homes, human-machine interfaces, and fall monitoring systems for the elderly. Understanding the activities performed in a room, for instance, can enable intelligent control of energy consumption, such as HVAC and lighting. Furthermore, by discerning, for example, user intent, understanding the human activities performed contributes to ubiquitous smart home solutions.

[0033] Human activity recognition can also sense and identify fall movements in older adults. Falls are a leading cause of death among older adults. In some cases, without immediate medical attention, a fall can severely limit an individual's overall mobility.

[0034] Some conventional human activity recognition systems are based on cameras and computer vision methods. These systems are generally accurate and relatively easy to implement. However, camera systems may lack privacy and may be sensitive to lighting conditions, which may make some camera systems unsuitable for certain applications.

[0035] Radar can also be used to effectively detect human activity. It offers privacy protection and illumination invariance, and can be aesthetically concealed within the operating environment.

[0036] Figure 1 A schematic diagram of an exemplary radar system 100 is shown. The radar system 100 includes a millimeter-wave radar sensor 102, a processor 120, and an artificial intelligence (AI) accelerator 122.

[0037] During normal operation, the millimeter-wave radar sensor 102 functions as a frequency-modulated continuous wave (FMCW) radar sensor and transmits multiple radar signals 106 (such as chirps) toward scene 130 using a transmitter (TX) antenna 114. The radar signals 106 are generated using RF and analog circuitry 104. The radar signals 106 can be in the range of 20 GHz to 122 GHz.

[0038] Objects in scene 130 may include idle people (such as a person lying down 134 and a person standing 136) and moving people (such as a person walking 132). Objects in scene 130 may also include static objects (not shown) (such as furniture, walls) and periodically moving devices (such as fans). Other objects may also be present in scene 130.

[0039] The transmitted radar signal 106 is reflected by an object in scene 130. The reflected radar signal 108 (also referred to as the echo signal) is received by receiver (RX) antenna 116. RF and analog circuitry 104 processes the received reflected radar signal 108 using, for example, a bandpass filter (BPF), a low-pass filter (LPF), a mixer, a low-noise amplifier (LNA), and / or an intermediate frequency (IF) amplifier in a manner known in the art to generate an analog signal x. out (t).

[0040] Use ADC 112 to convert the analog signal x out (t) is converted into raw digital data x out_dig (n). Raw numerical data x out_dig (n) is preprocessed by processor 120 and then processed by AI accelerator 122 to classify human activities in scene 130.

[0041] The controller 110 controls one or more circuits of the millimeter-wave radar sensor 102, such as RF and analog circuits 104 and / or ADC 112.

[0042] The processor 120 may be implemented in any manner known in the art, such as a general-purpose processor, controller, or digital signal processor (DSP) that includes, for example, combinational circuitry coupled to a memory.

[0043] AI Accelerator 122 is designed to accelerate artificial intelligence applications such as artificial neural networks and machine learning. AI Accelerator 122 can be implemented in any manner known in the art.

[0044] Figure 2A This illustrates the preprocessing and processing of raw digital data x. out_digA flowchart of method 200 (n). Method 200 includes: a preprocessing step 205, which includes steps 202 and 204; a feature extraction step 206; and a classification step 208. Steps 202, 204, and 206 are executed by processor 120. Step 208 is executed by AI accelerator 122.

[0045] In step 202, a 1D Moving Target Indication (MTI) filter is applied to the raw digital data x. out_dig (n) To eliminate responses from static targets (such as chairs, tables, and walls) and transmitter-receiver leakage, which affects the first few range chambers. The MTI filter can be performed by subtracting the average along the fast-time (time between chirps) to eliminate transmitter-receiver leakage interfering with the first few range chambers, and then subtracting the average along the slow-time (time between chirps) to eliminate reflections from static objects (or zero-Doppler targets).

[0046] During step 204, a 2D window is applied to the filtered digital data x along the fast and slow time dimensions. filterd_dig (n), and then a 2D Fast Fourier Transform (FFT) is performed to generate a 2D matrix representing the energy spectrum over the received distance and velocity, also known as a range-Doppler image (RDI).

[0047] During method 206, feature image extraction is performed on the range Doppler image to generate an RDI video. The RDI video can be represented as:

[0048] (1)

[0049] U st It is the size of the FFT along the slow path, U ft is the FFT size along the fast time, w(m,n) is the 2D weighting function along the fast and slow times, and s(m,n,k) is the ADC data (x) on the k-th frame. out_dig The indices n and m are scanned along the fast time axis and the slow time axis, respectively. The indices l and p are scanned along the distance axis and the Doppler axis, respectively.

[0050] During step 208, the RDI video is fed into a DCNN or LSTM, which classifies human activities of detected human targets based on the RDI video and training based on the DCNN or LSTM.

[0051] Figure 2B This illustrates the preprocessing and processing of raw digital data x. out_digA flowchart of method 220 (n). Method 220 includes: a preprocessing step 205, which includes steps 202 and 204; a feature extraction step 222; and a classification step 224. Steps 202, 204, and 222 are executed by processor 120. Step 224 is executed by AI accelerator 122. Steps 202 and 204 of method 220 are executed in a similar manner to method 200.

[0052] During step 222, feature image extraction is performed on the distance Doppler image by edge-mapping over the distance to generate a Doppler spectrum. Each generated Doppler frequency includes information about the macro-Doppler component and the micro-Doppler component caused by hand and leg movements of the human target detected during the performance of the activity.

[0053] Doppler spectra from consecutive frames are stacked one after another to generate a 2D image. The stacked Doppler spectra across consecutive frames are called a Doppler spectrogram and include information about the instantaneous Doppler spectral content and how the Doppler spectral content changes over time.

[0054] The Doppler spectrum of the slow-time data from the k-th radar frame on the selected L-range bin can be represented as:

[0055] (2)

[0056] Where V RDI It can be given by Equation 1.

[0057] During step 224, the Doppler spectrogram is fed into a DCNN or Long Short-Term Memory (LSTM) neural network, which classifies human activities of detected human targets based on the spectrogram and the training of the DCNN or LSTM.

[0058] In embodiments of the invention, preprocessing steps (e.g., 205) and feature extraction steps (e.g., steps 206 or 222) are omitted. Instead, a neural network is used to generate, for example, a human activity classification directly from the raw digital data from the ADC. By using a DCNN to operate directly on the raw digital data from the ADC, some embodiments advantageously reduce computational complexity and eliminate the need for a DSP for preprocessing.

[0059] Figure 3 A schematic diagram of a radar system 300 according to an embodiment of the present invention is shown. The radar system 300 includes a millimeter-wave radar sensor 102 and an artificial intelligence (AI) accelerator 322.

[0060] As shown in the figure Figure 3A possible implementation of the millimeter-wave radar sensor 102 according to an embodiment of the present invention is shown. Other implementations are also possible.

[0061] like Figure 3 As shown, in some embodiments, the millimeter-wave radar sensor 102 includes a reference oscillator 302, a phase-locked loop (PLL) 304, a voltage-controlled oscillator (VCO) 306, a frequency divider 308, an amplifier 310, a mixer 316, a low-pass filter (LPF) 318, and an ADC 112.

[0062] During normal operation, VCO 306 generates a linear frequency chirp signal (e.g., from 57 GHz to 64 GHz) transmitted by transmit antenna 114. VCO is controlled by PLL 304, which receives a reference clock signal (e.g., 80 MHz) from reference oscillator 302. PLL 304 is controlled by a loop including frequency divider 308 and amplifier 310.

[0063] The linear chirping signal transmitted by transmitting antenna 114 is reflected by an object in scene 130 and received by receiving antenna 116. Mixer 316 is used to mix the echo received by transmitting antenna 116 with a copy of the signal transmitted by transmitting antenna 114 to reduce the intermediate frequency (IF) signal intensity. IF (t) (also known as the beat frequency signal). In some embodiments, the beat frequency signal x IF (t) has a bandwidth between 10 kHz and 1 MHz. A beat frequency signal x has a bandwidth less than 10 kHz or greater than 1 MHz. IF (t) is also possible.

[0064] Beat frequency signal x IF (t) is filtered by a low-pass filter (LPF) 318 and then sampled by an ADC 112. The ADC 112 is advantageously capable of sampling the filtered beat frequency signal x at a sampling frequency much lower than the frequency of the signal received by the receiving antenna 116. out (t) Sampling is performed. Therefore, in some embodiments, the use of FMCW radar advantageously allows for a compact and low-cost implementation of the ADC 112.

[0065] Original digital data x out_dig (n) (in some embodiments, the filtered beat frequency signal x) out The digital version of (t) is (e.g., temporarily) stored (e.g., in N) C × N S In the matrix, where N C N is the number of chirping signals considered in the frame. S (This is the number of transmitted samples for each chirp signal) is used for further processing.

[0066] In some embodiments, ADC 112 is a 12-bit ADC. Higher resolution ADCs (such as 14-bit or higher) or lower resolution ADCs (such as 10-bit or lower) may also be used.

[0067] like Figure 3 As shown, in some embodiments, the AI ​​accelerator 322 is used to process the raw digital data x from the ADC 112. out_dig (n) is used to classify the activities of the target in scenario 130. The AI ​​accelerator 322 can be implemented in any manner known in the art.

[0068] like Figure 3 As shown, although some embodiments use AI accelerator 322 to implement neural networks to process raw digital data x out_dig (n) can be used to classify the activities of the target in scenario 130, but other hardware implementations, different from or besides AI accelerators, can also be used. For example, some embodiments may use a general-purpose processor, controller, or digital signal processor (DSP) (e.g., including combinational circuitry coupled to memory) to implement a neural network (such as DCNN 500, 800, or 900, as described in more detail below) for classifying the target. For example, in some embodiments, the neural network may be implemented using an ARM or x86 architecture. In some embodiments, the neural network may be implemented using a combination of custom application-specific integrated circuits (ASICs) and / or hardware accelerators and software running on a DSP or general-purpose microcontroller. Other implementations are also possible.

[0069] Figure 4 A series of chirp signals 106 transmitted by a TX antenna 114 according to an embodiment of the present invention are shown. Figure 4 As shown, the chirp signal 106 is organized across multiple frames and can be implemented as an upward chirp signal. Some embodiments may use a downward chirp signal or a combination of an upward chirp signal and a downward chirp signal.

[0070] The time between chirps in a frame is typically referred to as the pulse repetition time (PRT). In some embodiments, the PRT is 5 ms. Different PRTs may also be used, such as less than 5 ms (e.g., 4 ms, 2 ms or less) or greater than 5 ms (e.g., 6 ms or more).

[0071] A frame of chirp signal 106 may include multiple chirp signals. For example, in some embodiments, each frame of the chirp signal includes 16 chirp signals. Some embodiments may include more than 16 chirp signals per frame (such as 20 chirp signals, 32 chirp signals, or more) or less than 16 chirp signals per frame (such as 10 chirp signals, 8 chirp signals, or less). In some embodiments, each frame of the chirp signal includes only a single chirp signal.

[0072] The frame repeats once every FT time. In some embodiments, the FT time is 50 ms. Different FT times may also be used, such as greater than 50 ms (e.g., 60 ms, 100 ms, 200 ms or more) or less than 50 ms (e.g., 45 ms, 40 ms or less).

[0073] In some embodiments, the FT time is selected such that the time between the start of the last chirp signal of frame n and the start of the first chirp signal of frame n+1 is equal to the PRT. Other embodiments may be used or may result in different timing.

[0074] Some activities can be distinguished by analyzing their unique distance-velocity curves (e.g., ... Figure 2A and Figure 2B (As shown). Some activities can have very different distance-velocity curves, such as walking and idling. However, the distance-velocity curves of some activities may only show minute differences. For example, working on a laptop and idling in a chair may differ only from the slight hand movements exhibited by a human working on a laptop. Therefore, higher resolution in specific frequency bands may be needed to accurately distinguish these actions. However, when using preprocessing steps (such as steps 205, 206, and 222), the entire observable distance-velocity space is discretized in equal bins.

[0075] In embodiments of the invention, the first 2D convolutional layer of the neural network uses a plurality of trained temporal bandpass filters (e.g., using 2D sinc filter kernels) such that their respective cutoff frequencies are adjusted according to the needs of a particular application. Therefore, in some embodiments, replacing preprocessing steps (e.g., steps 205, 206, and 222) with a DCNN, which uses a first 2D convolutional layer constrained to optimize the cutoff frequencies of the plurality of temporal bandpass filters, advantageously results in improved accuracy during activity recognition compared to implementations using explicit preprocessing steps (e.g., steps 205, 206, and 222). By constraining the first 2D convolutional layer to a predetermined shape (e.g., using 2D sinc filter kernels or other types of predetermined kernels), some embodiments advantageously achieve faster training convergence times and help mitigate or avoid the problem of getting trapped in local minima.

[0076] Figure 5 A block diagram of a DCNN 500 for activity classification according to an embodiment of the present invention is shown. The DCNN 500 includes constrained 2D (conv2D) convolutional layers 502 and multiple additional layers 504. For example, the additional layers 504 may include one or more convolutional layers (including complex and non-complex convolutional layers), fully connected layers, recurrent layers, pooling layers, and / or dense layers. In some embodiments, the additional layers 504 may be implemented using known neural network architectures such as U-Net or atomic convolutional layers. For example, the DCNN 500 may be implemented in an AI accelerator 322.

[0077] like Figure 5 As shown, the 2D convolutional layer 502 is implemented by a constrained plurality of 2D sinc filters, such that only the cutoff frequency and / or bandwidth of each 2D sinc filter is trainable (as opposed to an unconstrained 2D convolutional layer, where all parameters of each filter are trainable, and where each filter is not limited to a specific type (such as a 2D sinc filter type)). For example, in some embodiments, each 2D sinc filter of the 2D convolutional layer 502 can be given by the following equation:

[0078] (3)

[0079] in It is a slow-time 1D sinc filter. N It is the length of the slow-time 1D sinc filter. It is the lower cutoff frequency of the slow-time 1D sinc filter, b st It is the bandwidth of the slow-time 1D sinc filter. n It is the index of the slow-time filter parameters (n is an integer between 0 and N, inclusive). It is a fast-time 1D sinc filter. M It is the length of the fast-time 1D sinc filter. It is the lower cutoff frequency of a fast 1D sinc filter. b ft It is the bandwidth of a fast 1D sinc filter. m It is the fast-time filter parameter index (m is an integer between 0 and M, inclusive), and w (n,m) It is a 2D cosine weighted function. The 2D cosine weighted function can be given by the following formula:

[0080] (4)

[0081] Furthermore, the slow-time 1D sinc filter and the fast-time 1D sinc filter can be given by the following formula:

[0082] (5)

[0083] in K It is the length of the 1D sinc filter. k It is 0 to K Integers between 0 and 1, including 0 and 1. K , f S The data to be filtered (e.g., x) out_dig The sampling frequency of (n) f l It is the lower cutoff frequency. b It is the bandwidth of the 1D sinc filter, and k It is the filter parameter index, where .

[0084] The 2D sinc filter sinc in the 2D convolutional layer 502 defined by Equation 3 2D In this context, the trainable parameters (also known as hyperparameters) are the lower cutoff frequencies of the slow-time 1D sinc filter and the fast-time 1D sinc filter, respectively. and ) and bandwidth (b st and b ft It should be understood that, for example, bandwidth (b st and b ft The lower cutoff frequency hyperparameter can be replaced by, for example, the upper cutoff frequency or center frequency of a slow-time 1D sinc filter and a fast-time 1D sinc filter, respectively, without affecting performance.

[0085] During the training of DCNN 500, a constrained 2D convolutional layer 502 is initialized according to a defined 2D sinc filter, and only hyperparameters are allowed to be learned. As a result, the filter trained on the constrained 2D convolutional layer 502 is a 2D bandpass filter (e.g., having a rectangular shape in the frequency domain) with its corresponding cutoff frequency optimized based on the training data.

[0086] Figure 6 An exemplary 2D sinc filter is shown in the time and frequency domains of a constrained 2D convolutional layer 502 according to an embodiment of the present invention. 2D .like Figure 6 As shown, the 2D sinc filter sinc 2D It is a 2D bandpass filter, which exhibits a clear cutoff frequency in the frequency domain, as shown by the rectangle in graph 620. Figure 6 As shown, the 2D sinc filter sinc 2D Able to extract the data to be filtered (e.g., x) out_dig The combined distance and velocity characteristics of (n).

[0087] In some embodiments, the 2D sinc filters used in the constrained 2D convolutional layer 502 include rotation parameters for rotating the 2D sinc filters relative to the pseudo-range domain and the pseudo-Doppler domain. In this embodiment, each 2D sinc filter of the 2D convolutional layer 502 can be given by the following equation:

[0088]

[0089]

[0090] (6)

[0091]

[0092]

[0093] in

[0094] (7)

[0095] (8)

[0096] (9)

[0097] Where α is the rotation angle. N It is the length of the 2D sinc filter in slow time. It is the lower cutoff frequency of the 2D sinc filter in slow time. b st It is the bandwidth of the 2D sinc filter in slow time. n It is the index of the slow-time filter parameters (n is an integer between 0 and N, inclusive). M It is the length of the 2D sinc filter in the fast time. It is the lower cutoff frequency of the 2D sinc filter in fast time. b ft It is the bandwidth of the 2D sinc filter in fast time. m It is the fast-time filter parameter index (m is an integer between 0 and M, inclusive), and w(n,m) It is a 2D cosine weighted function (for example, as given in Equation 4).

[0098] When the rotation angle α equals 0, equation 6 can be expressed as equation 3. Figure 7 The effect of an angle α of rotation relative to the pseudo-range domain and pseudo-Doppler domain is illustrated according to an embodiment of the invention. The pseudo-range domain and pseudo-Doppler domain are used herein to represent the feature map obtained after applying (e.g., 2D sinc) convolution.

[0099] like Figure 7 As shown, the exemplary 2D sinc filter 704 is rotated by a rotation angle α relative to the pseudo-range domain and the pseudo-Doppler domain. An exemplary 2D sinc filter 702 with a rotation angle α equal to 0 can be represented using Equation 4.

[0100] In some embodiments, classification cross-entropy is used as the loss function to implement DCNN 500. Other loss functions, such as mean squared error, can also be used.

[0101] In some embodiments, either convolutional layer 502 or additional layer 504 may use a rectified linear unit (ReLU) as the activation function. Other activation functions, such as sigmoid and leaky ReLU, may also be used.

[0102] Figure 8 A block diagram of a DCNN 800 according to an embodiment of the present invention is shown. The DCNN 800 is implemented as a DCNN 820 during training and as a DCNN 802 during normal operation (after training).

[0103] As shown in the figure, DCNN 800 includes a constrained 2D convolutional layer 502 and an additional layer 804, where the additional layer 804 represents a possible implementation of the additional layer 504.

[0104] like Figure 8 As shown, the additional layer 804 includes maxpool layers 806 and 810, an unconstrained 2D convolutional layer 808, a dense layer 812, and a softmax classifier layer 814. During training, after each convolutional and dense layer, the DCNN 800 additionally includes dropout layers 822, 824, and 826.

[0105] During training, the DCNN 800 receives batches of training data (e.g., 128 samples per batch) and uses a softmax classifier layer 814 to generate an M-element vector corresponding to the classification of the corresponding data, where M is equal to or greater than 2. For example, in some embodiments, M equals 6, corresponding to the following activity categories: “empty room”, “walking”, “idle”, “arm movement”, “waving”, and “working”. For example, the output vector can have the form [“empty room”, “walking”, “idle”, “arm movement”, “waving”, and “working”]. The training data includes a dataset containing records of five different human activities (“walking”, “idle”, “arm movement”, “waving”, and “working”) and records of empty rooms, where each record in the dataset is pre-labeled with a corresponding category (e.g., “empty room”, “walking”, “idle”, “arm movement”, “waving”, and “working”).

[0106] In some embodiments, the output vector includes confidence values ​​(i.e., the probability that a particular label is correct). In this embodiment, the output vector [0.01 0.75 0.1 0.05 0.04 0.05] can be interpreted as corresponding data with a probability of 1% for the "empty room" category, 75% for the "walking" category, 10% for the "idle" category, 5% for the "arm movement" category, 4% for the "waving" category, and 5% for the "working" category. In this case, the category with the highest confidence can be assigned to the corresponding data (in this non-limiting example, the category is "walking").

[0107] In some embodiments, during training, the constrained 2D convolutional layer 502 is initialized to cover the entire range Doppler space. For example, in some embodiments, the constrained 2D convolutional layer 502 can be initialized by directly defining a 2D sinc filter. In other embodiments, the constrained 2D convolutional layer 502 can be initialized by generating a 2D sinc filter using a 1D sinc filter. Other initialization schemes are also possible.

[0108] For example, in some embodiments, during training, a 2D sinc filter is used to initialize the constrained 2D convolutional layer 502. In some embodiments, N st The 1D sinc filter is initialized to equally divide the slow-time frequency into... The size of the frequency band, and N ft The 1D sinc filter is initialized to equally divide the fast-time frequency into... The size of the frequency band, of which fs,st and f s,ft These are the sampling frequencies of the slow-time data and the fast-time data, respectively. N st and N ft These represent the number of filters in the slow-time direction and the fast-time direction, respectively. This can be obtained by multiplying each 1D time-slot sinc filter by each 1D fast-time sinc filter. N st Second-rate N ft The initial set of filters thus covers the entire observable frequency domain. Therefore, there are no separate filters for slow and fast times. By initializing the 2D sinc filters of the constrained 2D convolutional layer 502 in this way, some embodiments avoid preferential frequency regions through initialization.

[0109] Due to the fast direction f s,ft The sampling frequency on the upper direction can be higher than that in the slower direction. f s,st The sampling frequency is several orders of magnitude higher, so in some embodiments, the cutoff frequencies and bandwidths of the slow-time and fast-time filters of the constrained 2D convolutional layer 502 are normalized to values ​​between 0 and 1 (inclusive) to allow, for example, equal training on both filter dimensions (slow-time and fast-time).

[0110] For example, the "Glorot" initialization scheme is used to initialize the unconstrained convolutional layer 808 and the dense layer 812.

[0111] During training, the hyperparameters of the constrained 2D convolutional layer 502 and the trainable parameters of the additional layer 804 are trained based on the training data. For example, during training, the output vector generated by the DCNN 800 is compared with the pre-labels of the corresponding data batches, and the trainable weights of the neural network are adjusted so that the classification of the corresponding batch of data corresponds to the corresponding pre-label. The model (neural network 800) is refined by running multiple training data batches (e.g., hundreds or thousands of training data batches).

[0112] In some embodiments, an optimizer (such as the RMSprop optimizer) is used to optimize DCNN 800. Other optimizers (such as gradient boolean optimizers and gradient boolean optimizers with momentum optimizers) may also be used. In some embodiments, the learning rate is l = 0.0001. r , and ρ and 10 for 0.9 -8 ε. Other learning parameter values ​​can also be used.

[0113] Dropout layers 822, 824, and 826 are used during training to help create redundancy in the neural network and to prevent overfitting by randomly removing nodes (e.g., randomly zeroing out the weights of the previous layer) and the corresponding edges to / from the removed nodes in the neural network. For example, the order of the layers during training is 502, 822, 806, 808, 824, 810, 812, 826, and 814. In some embodiments, 20% of the nodes are removed by each of the dropout layers 822, 824, and 826.

[0114] A constrained 2D convolutional layer 502 can be implemented using 65 filters in the slow-time dimension and 33 filters in the fast-time dimension. Different numbers of filters can also be used.

[0115] The maxpool layers 806 and 810 can be implemented using pooling sizes of 8×2 and 4×2, respectively. Other pooling sizes can also be used.

[0116] An unconstrained 2D convolutional layer 808 can be implemented using 50 filters of 3×3 size. In some embodiments, more than 50 filters (e.g., 60 or more) or fewer than 50 filters (e.g., 45 or fewer) can be used. In some embodiments, filters of different sizes than 3×3, such as 4×4 or 2×2, can also be used.

[0117] After the maxpool layer 810, the tensor is flattened and fed into a dense layer 812, which may be 32 in size. In some embodiments, the dense layer may have a size other than 32, such as greater than 32 (e.g., 35, 40 or larger) or less than 32 (e.g., 28, 24 or smaller).

[0118] Following the dense layer 812, the softmax classifier layer 814 generates categories. In an embodiment considering six categories, the softmax classifier 814 has a size of 6 (corresponding to each of the six categories). Some embodiments may implement the softmax classifier 814 with a size less than 6 (such as 5, 4, 3, or 2) or with a size greater than 6 (such as 7, 8, 10, or larger).

[0119] Once trained, the DCNN 800 can be used to classify objects, such as classifying human activities in scene 130. During normal operation (after training), the layers of the DCNN 800 are in the order of 502, 806, 808, 810, 812, and 814. The constrained 2D convolutional layer 502 receives raw digital data x from the ADC 112. out_dig(n) and filters the original digital data x by convolving it with multiple trained 2D sinc filters. out_dig (n). For example, a maxpool layer 806 is used to pool the filtered data to smooth it by applying an average. Then, a trained unconstrained 2D convolutional layer 808 is used to filter the pooled data. For example, a maxpool layer 810 is used to pool the output of the convolutional layer 808 to smooth the data and reduce dimensionality by applying an average. The tensor generated by the maxpool layer 810 is then flattened and fed into a dense layer 812, and then into a softmax classifier layer 814. The softmax classifier layer 814 generates an output vector with probabilities associated with each class.

[0120] Some embodiments offer advantages including that by constraining the first 2D convolutional layer to a specific filter shape (e.g., using a 2D sinc filter), some embodiments advantageously allow for faster convergence during the training process of the neural network when compared to an unconstrained convolutional layer. In some embodiments, constraining the first 2D convolutional layer to a specific filter shape (e.g., using a 2D sinc filter) has the additional advantage of helping to overcome the problem of getting trapped in local minima.

[0121] Some implementations can utilize filters other than 2D sinc filters to implement the first-constraint convolutional layer of a DCNN. For example, Figure 9 A block diagram of a DCNN 900 for activity classification according to an embodiment of the present invention is shown. The DCNN 900 operates in a similar manner to the DCNN 500. However, the DCNN 900 utilizes Morlet wavelet filters instead of 2D sinc filters to implement the 2D convolutional layer 902 with first constraints.

[0122] Morlet wavelets can be understood as the product of a fundamental frequency (carrier) and a Gaussian window (envelope). In some embodiments, each 2D Morlet wavelet filter of the 2D convolutional layer 902 can be given by the following equation:

[0123] (10)

[0124] in N It is the slow-time filter length. n It is the index of slow-time filter parameters ( n It is 0 to N Integers between 0 and 1, including 0 and 1. N ), M It is the length of the fast-time filter. m It is the fast-time filter parameter index ( m It is 0 to MIntegers between 0 and 1, including 0 and 1. M ), σ st It is the slow time standard deviation. σ ft It is the fast time standard deviation. It is the slow-time center frequency. It is the center frequency at high speed. It is the slow sampling frequency. It is a fast sampling frequency, and It can be given by the following formula:

[0125] (11)

[0126] The 2D Morlet wavelet filter of the 2D convolutional layer 902 defined by Equation 10 In this context, the trainable parameters (hyperparameters) are the wavelet's center frequency (…). and ) and standard deviation (σ) st and σ ft Similar to the 2D sinc filter used in the constrained 2D convolutional layer 502, for example, a wavelet filter can be adjusted by changing the center frequency of the filter ( and The frequency region of interest can be adjusted by adjusting the standard deviation (σ) of the Gaussian part of the wavelet. Additionally, however, the wavelet can be adjusted by adjusting the standard deviation (σ) of the Gaussian part of the wavelet. st and σ ft Use this to adjust the time-frequency resolution.

[0127] As shown in Equation 10, since the Morlet wavelet is the product of a cosine function and a Gaussian window function, the resulting Morlet wavelet's frequency response also has a Gaussian shape. For example, Figure 10 An exemplary 2D Morlet wavelet filter used in a constrained 2D convolutional layer 902 in both the time and frequency domains according to an embodiment of the present invention is shown. .like Figure 10 As shown, the Morlet wavelet filter does not exhibit a distinct cutoff frequency (as shown in curve 1020). The standard deviation of the Gaussian function in the time and frequency domains is indirectly proportional, where decreasing the width of the Gaussian function in the time domain leads to an increase in the width in the frequency domain (which illustrates the trade-off between time and frequency resolution).

[0128] Similar to the 2D sinc filter used in the constrained 2D convolutional layer 502, the 2D Morlet wavelet filter used in the constrained 2D convolutional layer 902 may include rotation parameters for rotating the 2D Morlet wavelet filter with respect to the pseudo-range domain and the pseudo-Doppler domain. In this embodiment, each 2D Morlet wavelet filter in the 2D convolutional layer 902 can be given by the following equation:

[0129] (12)

[0130] in and The results are given by Equations 8 and 9, respectively, where α is the rotation angle. N It is the slow-time filter length. M It is the length of the fast-time filter, σ st It is the slow time standard deviation, σ ft It is the fast time standard deviation. It is the slow-time center frequency. It is the center frequency at high speed. It is the slow sampling frequency. It is the fast sampling frequency. This can be given by Equation 11.

[0131] When the rotation angle α is equal to 0, equation 12 can be expressed as equation 10.

[0132] In some embodiments, during training, the constrained 2D convolutional layer 902 is initialized to cover the entire range Doppler space. For example, in some embodiments, the constrained 2D convolutional layer 902 can be initialized by directly defining a 2D Morlet wavelet filter. In other embodiments, the constrained 2D convolutional layer 902 can be initialized by generating a 2D Morlet wavelet filter using a 1D Morlet wavelet filter. Other initialization schemes are also possible.

[0133] For example, in some embodiments, N st 1D Morlet wavelet filter and N ft The 1D Morlet wavelet filters are initialized to equally divide the slow-time and fast-time dimensions into equal bands. Both time axes are normalized to values ​​between 0 and 1, inclusive. N is obtained by multiplying the outer product of each 1D slot Morlet wavelet filter by each fast-time Morlet wavelet filter. st Nth time ftThe initial set of filters is used to cover the entire observable frequency domain. In some embodiments, the standard deviation is initialized to 0.06 (other values ​​may also be used). In some embodiments, the trainable weights (hyperparameters) of the Morlet wavelet filter can also be normalized by mapping to a range of values ​​between 0 and 1 (inclusive).

[0134] By using wavelet filters in the first constrained 2D convolutional layer of the DCNN, some embodiments advantageously allow the time-frequency resolution of the filter in the first 2D convolutional layer to be adjusted, for example, by adjusting the standard deviation of the Gaussian distribution associated with the wavelet filter.

[0135] Figures 11-17 Experimental setups or results of DCNN 800 and DCNN 900 implemented using radar system 300 according to embodiments of the present invention are shown.

[0136] In order to generate experimental results, such as Figure 4 As shown, radar system 300 is implemented using an upward chirp signal with a single chirp signal per frame and a PRT of 1 ms, wherein each chirp signal is generated by 128 samples and has a duration of 64 μs. Each chirp signal has a ramp start frequency f of 59.5 GHz. min The ramp stop frequency f of 60.5 GHz max And a bandwidth of 1 GHz. The ADC 112 is implemented as a 12-bit ADC operating at a sampling frequency of 2 MHz. The range resolution is 15 cm, with a maximum range of 9.6 m and a maximum Doppler velocity of 1.25 m / s. The elevation angle and azimuth angle (direction of the beam center) of the millimeter-wave radar sensor 102 are 70° and 120°, respectively.

[0137] To generate experimental results, additional layer 504 of DCNN 900 was implemented in a similar manner to additional layer 804, and during training, DCNN 900, including dropout layers 822, 824 and 826, was implemented in DCNN 800.

[0138] To generate experimental results, DCNN 800 and 900 were trained for activity classification on a dataset that included recordings of five different human activities as well as empty rooms. Thus, DCNN 800 and 900 were trained to identify six categories: “empty room”, “walking”, “idle”, “arm movement”, “waving”, and “working”.

[0139] Figure 11 An experimental setup for data recording of an activity “work” performed by a tester, according to an embodiment of the present invention, is shown. Figure 11As shown, the millimeter-wave radar sensor 102 is located in a corner of a room, which includes a table in front of the millimeter-wave radar sensor 102 and a chair facing the millimeter-wave radar sensor 102.

[0140] To record the category "walking", a single human in Figure 11 The human walked randomly within the room. The category "idle" was divided into two records: in the first record, the human stood in front of the millimeter-wave radar sensor 102; in the second record, the human sat at a table facing the millimeter-wave radar sensor 102. To record the category "arm movement," the human was recorded standing... Figure 11 He randomly moved his arm while in the room. To record the category "waving", the human... Figure 11 Waving at the millimeter-wave radar sensor 102 from different locations in the room. To record the category "work," such as... Figure 11 The human shown is sitting in a chair working on his laptop. This is for the category of "empty room". Figure 11 The rooms were recorded as having human presence. During training to generate experimental results, each human activity was performed by the same human, and each category was recorded for approximately 18 minutes in total.

[0141] Figure 12 The diagram illustrates the number of samples per class in a training dataset 1200 used to train DCNNs 800 and 900 for generating experimental results, according to an embodiment of the invention. Each sample in the training dataset 1200 has 2048 chirps. Samples are generated by overlaying 2048 chirps from a recording with 512 chirps. Since the PRT is 1 ms, each sample is captured for 2.048 seconds. Figure 12 As shown, for each activity, there are approximately 700 samples available for training per category.

[0142] For comparison purposes, results were also generated for training three other neural networks using the same training dataset 1200. The first neural network was implemented in a similar manner to DCNN 800, but with unconstrained 2D convolutional layers instead of constrained convolutional layers 502. The second neural network was implemented to receive a Doppler spectrogram as input (from step 222) instead of raw digital data from ADC 112. The third neural network was implemented to receive RDI video (from step 206) instead of raw digital data from ADC 112.

[0143] Five neural networks (DCNN 800 and 900, and three other neural networks) were trained for 20 rounds using a training dataset of 1200 (except for the spectrogram-based neural network, which was trained for 100 rounds to allow convergence), where rounds indicate the number of times the machine learning algorithm has completed training across the entire training dataset.

[0144] Figure 13 The cumulative gains of the initial 2D filters of convolutional layers 502 and 902 according to an embodiment of the present invention are shown. Curves 1302, 1304, and 1306 respectively show the initial cumulative gains of the constrained convolutional layer 502's 2D sinc filter (after initialization but before training) in terms of range, velocity, and joint range-velocity gain. Curves 1312, 1314, and 1316 respectively show the initial cumulative gains of the constrained convolutional layer 902's 2D Morlet wavelet filter (after initialization but before training) in terms of range, velocity, and joint range-velocity gain. Curves 1322, 1324, and 1326 respectively show the initial cumulative gains of the unconstrained convolutional layer (after initialization but before training) in terms of range, velocity, and joint range-velocity gain when initialized using the Glorot initialization scheme. Figure 13 As shown, the initial gain is approximately constant throughout the entire space.

[0145] During training, the filter parameters are iteratively optimized. Figure 14 The cumulative gain of the 2D filters of convolutional layers 502 and 902 after 20 training epochs according to an embodiment of the present invention is shown. Curves 1402, 1404, and 1406 respectively show the cumulative gain of the 2D sinc filter of the constrained convolutional layer 502 trained on distance, velocity, and joint distance-velocity gain. Curves 1412, 1414, and 1416 respectively show the cumulative gain of the 2D Morlet wavelet filter of the constrained convolutional layer 902 trained on distance, velocity, and joint distance-velocity gain. Curves 1422, 1424, and 1426 respectively show the cumulative gain of the unconstrained convolutional layers trained on distance, velocity, and joint distance-velocity gain.

[0146] like Figure 14 As shown, the cumulative gain of the 2D sinc filter and the 2D Morlet wavelet filter exhibits bandpass characteristics. Therefore, except that the 2D Morlet wavelet gain is smoother due to the nature of its filter shape in the frequency domain, the cumulative gains obtained by the 2D sinc filter and the 2D Morlet wavelet filter are similar.

[0147] However, the shape of an unconstrained convolutional layer differs from that of a 2D sinc filter or a 2D Morlet wavelet filter. The reason for this difference is that the search space of an unconstrained 2D convolutional layer is not limited to a specific filter type. Therefore, although 2D sinc filters and 2D wavelet filters exist within the search space of an unconstrained 2D convolutional layer, this search space is several orders of magnitude larger than that of a 2D convolutional layer constrained to a specific filter type. Consequently, achieving a comparable solution using an unconstrained 2D convolutional layer may require significantly longer training times compared to using constrained 2D convolutional layers (such as 502 and 902). Additionally, during training at local minima, an unconstrained 2D convolutional layer may get stuck and may not achieve a solution with performance comparable to that using constrained 2D convolutional layers (such as 502 and 902).

[0148] Figure 15 The accuracy, standard deviation, and F1 score of DCNN 800, DCNN 900, and three additional networks according to embodiments of the present invention are shown. Figure 15 As shown, after 20 epochs of training, DCNN 800 and DCNN 900 achieve 98.9% and 99.5% accuracy, respectively, which is significantly better than the accuracy achieved by DCNNs implemented using unconstrained first 2D convolutional layers, using spectrogram-based neural networks (even if the spectrogram-based neural networks are trained for 100 epochs to allow convergence), or using neural networks based on RDI videos. As also shown, DCNN 800 and DCNN 900 converge to near 100% accuracy in less than 21 epochs of training.

[0149] The limitations of spectrogram-based and RDI-based video methods are partly due to their respective preprocessing steps. For example, the lack of distance information in spectrogram-based methods can adversely affect category prediction, which can be exacerbated when analyzing multiple human activities simultaneously. As another example, the STFT used to generate RDI discretizes both the range and velocity domains equally. However, some activities (such as "idle" and "working") exhibit very slight motion. As a result, their features share similar range Doppler binaries, so the data processed by the STFT is very similar for these two actions, making the classification task difficult.

[0150] Figure 16 The confusion matrix for classification of RDI-based videos and classification using DCNN 900 is shown according to an embodiment of the present invention. Figure 16As shown, the classification of "idle" and "working" using an RDI video-based neural network (as shown in confusion matrix 1602) exhibits significantly lower accuracy compared to using DCNN 900 (as shown in confusion matrix 1622).

[0151] Some of the advantages of the embodiments include the ability to achieve better accuracy by allowing the neural network to operate directly on (e.g., from ADC 112) the raw digital data, rather than using preprocessing steps (such as spectrogram-based or RDI video-based preprocessing).

[0152] While using unconstrained 2D convolutional layers that operate directly on (e.g., from ADC 112) raw digital data may not exhibit the limitations of spectrogram-based or RDI-based implementations, the use of specific types of filters (such as 2D sinc filters and 2D wavelet filters) results in a reduction of the search space and advantageously allows some embodiments to reach solutions that are at least close to the global minimum (global optimum), significantly reducing the training workload compared to using unconstrained convolutional layers.

[0153] Figure 17 The diagram illustrates a DCNN 800 / DCNN 900 according to an embodiment of the present invention, as well as the model dimensions of a DCNN implemented using unconstrained 2D convolutional layers. As shown, layer 502 is implemented with 64 filters of 65x33, layer 902 with 64 filters of 129x33, and the unconstrained convolutional layers with 64 filters of 65x33.

[0154] like Figure 17 As shown, DCNN 800 and DCNN 900 have a size less than half that of an unconstrained DCNN. This small size is because the first layer of DCNN 800 and DCNN 900 is significantly smaller than the first layer of an unconstrained 2D convolutional layer. The smaller size of layers 502 and 902 is due to the fact that only four hyperparameters are trainable (64 filters multiplied by 4 hyperparameters equals 256), while the remaining weights are fixed (unlike unconstrained convolutional layers, where all weights are trainable).

[0155] Figure 18 A block diagram of a DCNN 1800 for target and / or activity classification according to an embodiment of the present invention is shown. The DCNN 1800 includes constrained 2D convolutional layers 1802 and multiple additional layers 504. For example, the DCNN 1800 can be implemented in an AI accelerator 322. In some embodiments, the constrained convolutional layer 1802 is a constrained L-dimensional convolutional layer, where L is a positive integer greater than or equal to 2.

[0156] The constrained 2D convolutional layer 1802 can be implemented as a constrained 2D convolutional layer 502 or 902. In some embodiments, the constrained 2D convolutional layer 1802 can be implemented using a fractional Fourier transform filter or a discrete cosine transform filter. Other filter types that include global minima in their search space can also be used. In some embodiments, filters with search spaces that include only local minima can also be used.

[0157] In some embodiments, the filter kernel of the constrained 2D convolutional layer 1802 has a size of 10×10 or larger (e.g., 65×33).

[0158] For example, DCNN 1800 can be used for human activity classification in a similar manner to DCNN 800 and DCNN 900. DCNN 1800 can also be used for other types of classification. For example, in some embodiments, DCNN 1800 can be used for gesture sensing applications, where each gesture (e.g., human posture, such as gesturing with a hand) corresponds to a category. In this embodiment, DCNN 1800 is trained using a dataset based on the gestures to be recognized.

[0159] In some embodiments, DCNN 1800 can be used in person detection applications where objects are classified as human or non-human. In such embodiments, DCNN 1800 is trained using datasets based on either human or non-human data.

[0160] Some embodiments may implement other types of classification. For example, in some embodiments, the set of categories includes categories indicating the number of humans present (e.g., counting the number of humans in a room). In some embodiments, the set of categories includes categories indicating the presence of humans and categories indicating the absence of humans. Other classifications are also possible.

[0161] Some embodiments implement a multi-layer approach for the first layer of a DCNN 1800. For example, in some embodiments, additional constrained convolutional layers follow the first constrained convolutional layer. Figure 19 A block diagram of a DCNN 1900 for target and / or activity classification according to an embodiment of the present invention is shown. The DCNN 1900 is a possible implementation of the DCNN 1800 and includes constrained 2D convolutional layers 1802 and multiple additional layers 1904. For example, the DCNN 1900 can be implemented in an AI accelerator 322.

[0162] The additional layer 1904 includes a second-constrained 2D convolutional layer 1902 and multiple additional layers 1906. In some embodiments, additional layer 504 may be implemented as additional layer 1904. In some embodiments, additional layer 1906 may be implemented as additional layer 804. Other implementations are also possible.

[0163] In some embodiments, the filters implemented using constrained convolutional layers 1802 and / or constrained convolutional layers 1902 and / or additional layers 1906 can be complex. For example, in some embodiments, the learned cosine kernel is implemented as a complex kernel in which the same real kernel is phase-shifted by 90° and convolved with the input.

[0164] In some embodiments, the constrained convolutional layer 1902 can be implemented as an L-dimensional convolutional layer, where L is a positive integer greater than or equal to 2. The dimension of the constrained convolutional layer 1902 can be the same as or different from the dimension of the constrained convolutional layer 1802.

[0165] DCNN 1900 operates in a similar manner to DCNN 1800. However, DCNN 1900 includes a second-constrained 2D convolutional layer 1902 that operates in conjunction with a first-constrained 2D convolutional layer 1802. For example, in some embodiments, the first-constrained 2D convolutional layer 1802 implements multiple coarse filters that produce corresponding data channels. The data within each data channel is then downsampled (e.g., by using a stride greater than 1 in the convolutional or pooling layers) to reduce computational complexity. The second-constrained 2D convolutional layer 1902 then operates only within the previously filtered frequency region of the corresponding channel, rather than covering the entire observable search space.

[0166] Figure 20 Exemplary frequency response curves 2002 and 2022 of a first-constrained 2D convolutional layer 1802 and a second-constrained 2D convolutional layer 1902, respectively, according to embodiments of the present invention, are shown. Figure 20 In one embodiment, a 2D sinc filter is used to implement the first-constrained 2D convolutional layer 1802 and the second-constrained 2D convolutional layer 1902. Other embodiments may use other filter types.

[0167] In some embodiments, the first-constrained 2D convolutional layer 1802 and the second-constrained 2D convolutional layer 1902 can be implemented using different types of filters. For example, in some embodiments, the first-constrained 2D convolutional layer 1802 can be implemented using a 2D sinc filter, while the second-constrained 2D convolutional layer 1902 can be implemented using a 2D Morlet wavelet filter. Other implementations are also possible.

[0168] As shown by curve 2002, the 2D convolutional layer 1802 of the first constraint uses a coarse filter that is trained to adjust its cutoff frequency based on the training dataset. As shown by curve 2022, the filters of the 2D convolutional layer 1902 of the second constraint are also trained to adjust their cutoff frequencies based on the training dataset, but their search space is limited to the cutoff frequency determined by the 2D convolutional layer 1802 of the first constraint.

[0169] In some embodiments, the first-constrained 2D convolutional layer 1802 and the second-constrained 2D convolutional layer 1902 are trained simultaneously. For example, in some embodiments, the L2 norm restricts the search space of the filter of the second-constrained 2D convolutional layer 1902 (as a soft constraint). In some embodiments, explicit boundaries restrict the search space of the filter of the second-constrained 2D convolutional layer 1902 (hard constraints).

[0170] For example, such as Figure 18 As shown, the DCNN 1800 can generate outputs corresponding to the classification of a target (e.g., human activity classification, gesture recognition, people detection, people counting, etc.) based on a predetermined set of categories. In some embodiments, the DCNN 1800 can generate outputs other than or replacing the output indication of the target's classification, for example, based on a predetermined set of categories. Figure 21 A block diagram of a DCNN 1800 for generating 2D radar images according to an embodiment of the present invention is shown.

[0171] like Figure 21 As shown, the DCNN 1800 can be trained based on raw digital data x from the ADC 112. out_dig (n) to generate radar images. For example, in some embodiments, the DCNN 1800 can generate 2D radar images, such as range-Doppler images (RDI), range-angle images (RAI), and Doppler-angle images (DAI). Therefore, in some embodiments, the preprocessing and feature extraction steps can be implicitly implemented by the DCNN 1800 (e.g., Figure 2A Steps 205 and 206, and Figure 2B Steps 205 and 222).

[0172] DCNN 1800 can be trained to generate radar images using a training dataset generated from (e.g., radar and / or cameras) (e.g., one or more humans performing one or more activities (such as walking, working, standing, waving, arm movements, etc.) in scene 130). Each sample in the training dataset is pre-labeled with a corresponding 2D radar image, for example, by manually removing ghosted targets using artificial filters and adding real targets if they are missed in the generated 2D radar image. The difference between the acquired 2D image (at the output of DCNN 1800) and the corresponding pre-labeled 2D image is used as the error for training DCNN 1800.

[0173] Figure 22 Exemplary 2D radar images 2202 and 2222 during training of a DCNN 1800 according to an embodiment of the present invention are shown. Radar image 2222 is a pre-labeled 2D RDI corresponding to a recording of four humans walking in a room.

[0174] As shown in the figure, radar image 2222 shows four identified targets 2204, 2206, 2208, and 2210. For example, in some embodiments, radar image 2222 is a matrix, where 1 indicates that a target is located at the identified range-velocity point, and 0 indicates that no target is located at the identified range-velocity point.

[0175] For example, during training, recordings of four humans walking around a room are fed into DCNN 1800, which generates a 2D radar image 2202, for instance. As shown, there is a difference between the generated radar image 2202 and a pre-labeled radar image 2222. For example, no human 2210 is detected in radar image 2202. The difference between radar images 2202 and 2222 represents an error, which is used to optimize DCNN 1800, for example, by adjusting the hyperparameters of the constrained convolutional layer 1802 and / or other weights of the additional layer 504 based on a loss function (such as mean squared error).

[0176] In some embodiments, pre-labeled radar images are generated by using, for example, a camera that records images simultaneously with radar recordings to generate a training dataset. In some embodiments, for example, pre-labeled radar images are generated by performing steps 205 and 206 on radar recordings instead of using camera images or in addition to using camera images.

[0177] In some embodiments, pre-labeled images are manually labeled by a user based on knowledge of the target and the corresponding activity. In some embodiments, at least some of the pre-labeled radar images are synthetically generated. For example, in some embodiments, radar images with multiple people are synthesized by performing a data augmentation step during training based on multiple single-person radar images (e.g., by overlaying multiple single-person radar images). For example, two single-person images are overlaid, where the humans are not close together in the corresponding radar images, to generate a radar image with two people.

[0178] Although RDI has been used to demonstrate a possible method for training a DCNN 1800 for generating radar images, it should be understood that other radar images, such as RAI and DAI, can also be used.

[0179] In some embodiments, the output of the DCNN 1800 is further processed, for example, for tracking targets, counting people, or other applications. Figure 23 A schematic diagram of a radar system 2300 according to an embodiment of the present invention is shown. The radar system 2300 includes a millimeter-wave radar sensor 102, an AI accelerator 322, and a processor 1910.

[0180] like Figure 23 As shown, processor 2310 can post-process the output of DCNN 1800, which includes information about targets in scene 130, such as target classification information and / or radar images. For example, in some embodiments, during normal operation, radar images with target location information (such as radar image 2222) are fed to processor 2310, for example, targets are clustered in processor 2310 to group detected targets into individual targets (e.g., to group detected hands, torsos, and feet into individual human targets).

[0181] The processor 2310 can cluster targets using, for example, density-based spatial clustering (DBSCAN) for noise-based applications, based on the output of DCNN 1800, or other clustering methods.

[0182] In some embodiments, the processor 2310 is used to track targets and their activities. For example, in some embodiments, the processor 2310 may track which activities a detected human target is performing over time, such as sitting, then standing, then walking, then working, etc.

[0183] In some embodiments, a Kalman filter can be used to track one or more targets based on radar images received from a DCNN 1800. In some embodiments, the Kalman filter can also track the relevant target classification (e.g., related human activities, gestures, etc.) of the detected targets based on the classification output of the DCNN 1800. Tracking methods other than those using a Kalman filter, or other tracking methods besides using a Kalman filter, can also be used. For example, some embodiments may use a particle filter instead of a Kalman filter to track targets.

[0184] In addition to or in lieu of clustering and / or tracking targets, processor 2310 can be used for other post-processing activities. Processor 2310 can be implemented in a similar manner to processor 120. In some embodiments, processor 2310 and AI accelerator 322 are integrated into a single integrated circuit (IC).

[0185] Some embodiments offer advantages including minimizing the data flow of the radar system. For example, in radar system 2300, data flows from millimeter-wave radar 102 to AI accelerator 322 (e.g., for classification), and then to processor 2310 (for post-processing). In contrast, a method based on radar system 100 would exhibit a data flow from millimeter-wave radar 102 to processor 120 (for pre-processing), then to AI accelerator 122 (for classification), and then back to processor 120 (for post-processing).

[0186] In some embodiments, the convolutional layer with the first constraint can be used to feed its output to multiple additional layer paths. For example... Figure 24 A block diagram of a DCNN 2400 according to an embodiment of the present invention is shown. For example, the DCNN 2400 can be used to generate target localization data.

[0187] The DCNN 2400 includes a constrained 3D convolutional layer 2402 and additional layers 2404, 2406, and 2408. It should be understood that the constrained 3D convolutional layer 2402 is a possible implementation of the constrained convolutional layer 1802. For example, each of the additional layers 2404, 2406, and 2408 can be implemented as additional layer 504. For example, in some embodiments, additional layers 2404, 2406, and 2408 have the same architecture (e.g., the same order of the same layers) and can be initialized in a similar manner (although the weights of the corresponding trainable parameters may differ after training), but perform pointwise convolutions along different axes. For example, in some embodiments, additional layers 2404, 2406, and 2408 keep all layer parameters the same, but additional layer 2404 performs pointwise convolution along the channel dimension to generate the reconstructed RDI, additional layer 2406 performs pointwise convolution along the fast time dimension to generate the reconstructed DAI, and additional layer 2408 performs pointwise convolution along the slow time dimension to generate the reconstructed RAI.

[0188] In other embodiments, some or all of the additional layers 2404, 2406 and 2408 may have different architectures and / or may be initialized in different ways from each other.

[0189] Integration step 2410 can be performed on the outputs of additional layers 2404, 2406, and 2408 to generate positioning data. For example, in some embodiments, integration step 2410 is implemented by utilizing soft information transfer between layers, such as by using a softmax layer (e.g., leveraging connections between additional layers 2404, 2406, and / or 2408). Other embodiments may implement integration step 2410 by performing signal processing operations to integrate reconstructed RDI, DAI, and / or RAI to generate positioning data.

[0190] In some embodiments, a constrained 3D convolutional layer 2402 is implemented using a plurality of 3D sinc filters. In some embodiments, each dimension of the 3D sinc filter corresponds to the slow time, the fast time, and the channel, respectively, wherein each channel corresponds to a data stream from a (real or virtual) RX antenna associated with the millimeter-wave radar sensor 102. For example, in an embodiment using two real antennas to receive the reflected radar signal 108, the number of channels is 2. More than two antennas may also be used.

[0191] In some embodiments, the 3D sinc filter of the c-constrained 3D convolutional layer 2402 is phase-shifted. For example, if a channel has a kernel w 1, then the next channel is applied. The next channel is applied And so on, where θ is the angle of the target.

[0192] The RX antenna for each channel can be implemented in any manner known in the art. For example, in some embodiments, one TX antenna and three TX antennas are implemented in an L-shaped configuration for a 3-channel implementation. Other implementations are also possible.

[0193] During normal operation, the output of the constrained 3D convolutional layer is fed into one or more paths of additional layers (such as additional layers 2404, 2406, and / or 2408). The output of one or more additional layers can be integrated via integration step 2410 to generate localization data along distance, Doppler, and angle. Figure 24 As shown, the positioning data can have three dimensions, such as distance, Doppler, and angle. In some embodiments, the generated positioning data can be in the form of a 2D range-doppler image (RDI), a 2D range-angle image (RAI), and / or a 2D Doppler-angle image (DAI).

[0194] The additional layer 2404 receives the output from the constrained 3D convolutional layer 2402 and generates a reconstructed RDI (e.g., similar to RDI 2222), where the location of the detected target is identified in the range-Doppler domain. Figure 24 As shown, the additional layer 2404 operates on the received input by pointwise convolution along the channel domain, thereby generating slices along the fast time domain to the slow time domain, where pointwise convolution along the first domain can be understood as a weighted summation of learning along this first domain.

[0195] The additional layer 2406 receives the output from the constrained 3D convolutional layer 2402 and generates a reconstructed DAI, in which the location of the detected target is identified in the Doppler angle domain. Figure 24 As shown, the additional layer 2406 generates slices along the Doppler channel domain by performing pointwise convolution on the received input along the fast time.

[0196] The additional layer 2408 receives the output from the constrained 3D convolutional layer 2402 and generates a reconstructed RAI, in which the location of the detected target is identified in the range-angle domain. For example... Figure 24 As shown, the additional layer 2408 operates on the received input by pointwise convolution along the slow time, thereby generating slices along the fast time channel domain.

[0197] In some embodiments, only two of the additional layers are implemented (e.g., only additional layers 2404 and 2408).

[0198] In some embodiments, integration layer 2410 may be omitted. For example, in some embodiments, only one of the additional layers (such as one of additional layers 2404, 2406 or 2408) is implemented.

[0199] In an embodiment with two channels, a constrained 3D convolutional layer 2402 can be implemented using, for example, an F1×F22D sinc filter associated with the first channel and an F1×F22D sinc filter associated with the second channel filter, wherein the second F1×F22D sinc filter associated with the second channel is implemented as follows: Multiply by the F1×F22D sinc filter associated with the first channel. In some embodiments, the sum of the results of the F1×F22D sinc filters associated with the first channel and the F1×F22D sinc filters associated with the second channel is additionally provided to the F1×F2 matrix by the K kernels. Thus, the K kernels will transform the input from 2 to K along the channel. In some embodiments, F1 equals 36 and F2 equals 65. Other values ​​for F1 and F2 are also possible.

[0200] In some embodiments, the kernel value is normalized such that the 2D sinc filter associated with the first channel has a value of w1=1 for all angles θ. Each of the F1×F2 filters associated with the second channel can be given by the following equation:

[0201] (13)

[0202] in d It is the distance between the two RX antennas. λ It is the wavelength of the emitted signal (e.g., the center frequency of a chirp signal), and θ It's from the perspective of the goal.

[0203] In some embodiments, and ,in and Therefore, in some embodiments, each k th The kernels that can be trained are And the spatial frequency that can be trained is , K It is the number of angular kernels, and These are learnable parameters. Therefore, the kernel used for learning the second channel is... .

[0204] In some embodiments, instead of applying the learned kernel in the fast and slow time domains via convolution operations, the learned kernel is applied along the antenna via 1×1 convolutional / fully connected connections. K In the embodiment with 29 (angular kernels) and M total kernels, the 4D kernel mapping after the constrained 3D convolutional layer 2402 is 128×32×29×M.

[0205] In some embodiments, these kernels can be implemented using complex weights, where k th The kernel can be represented as:

[0206] (14)

[0207] Exemplary embodiments of the invention are summarized herein. Other embodiments may also be understood from the entire specification and claims submitted herein.

[0208] Example 1. A method comprising: transmitting multiple radar signals toward a target using a millimeter-wave radar sensor; receiving multiple reflected radar signals corresponding to the multiple transmitted radar signals using a millimeter-wave radar; mixing copies of the multiple transmitted radar signals with the multiple received reflected radar signals to generate an intermediate frequency (IF) signal; generating raw digital data based on the IF signal using an analog-to-digital converter; processing the raw digital data using a constrained L-dimensional convolutional layer of a neural network to generate intermediate digital data, where L is a positive integer greater than or equal to 2, and where the neural network includes multiple additional layers; and processing the intermediate digital data using the multiple additional layers to generate information about the target.

[0209] Example 2. Following the method of Example 1, generating information about the target includes classifying the target based on a set of categories.

[0210] Example 3. A method based on one of Examples 1 or 2, wherein generating information about the target includes providing the target's location using radar imagery.

[0211] Example 4. The method according to one of Examples 1 to 3, wherein the radar image is a range-Doppler image (RDI), a range-angle image (RAI), or a Doppler-angle image (DAI).

[0212] Example 5. Following the method of one of Examples 1 through 4, where L equals 2.

[0213] Example 6. The method of one of Examples 1 through 5, wherein the kernel size of the filter of the constrained L-dimensional convolutional layer is greater than 10×10.

[0214] Example 7. A method based on one of Examples 1 through 6, wherein processing raw digital data using constrained L-dimensional convolutional layers includes processing the raw digital data using a 2D sinc filter of constrained L-dimensional convolutional layers.

[0215] Example 8. Following the method of one of Examples 1 through 7, where the 2D sinc filter is defined by the following equation:

[0216]

[0217] in It is a slow-time 1D sinc filter. N It is the length of the slow-time 1D sinc filter. It is the lower cutoff frequency of the slow-time 1D sinc filter, b st It is the bandwidth of the slow-time 1D sinc filter. n It is 0 to N Between and including 0 and N integers, It is a fast-time 1D sinc filter, where M is the length of the fast-time 1D sinc filter. It is the lower cutoff frequency of a fast 1D sinc filter. b ft It is the bandwidth of a fast 1D sinc filter. m It is 0 to M Between and including 0 and M integers, w(n,m) It is a 2D cosine weighting function, where the slow-time 1D sinc filter and the fast-time 1D sinc filter are defined by the following equation:

[0218]

[0219] Where K is the length of the 1D sinc filter. k It is 0 to K Between and including 0 and K integers, f s It is the sampling frequency of the signal to be filtered. f l It is the lower cutoff frequency. b It is the bandwidth of the 1D sinc filter, and k It is the filter parameter index.

[0220] Example 9. Following the method of one of Examples 1 through 7, where the 2D sinc filter is defined by the following equation:

[0221]

[0222]

[0223]

[0224]

[0225]

[0226] in,

[0227]

[0228]

[0229]

[0230] Where α is the rotation angle, and α is not 0. N It is the length of the 2D sinc filter in slow time. It is the lower cutoff frequency of the 2D sinc filter in slow time. b st It is the bandwidth of the 2D sinc filter in slow time. n It is 0 to N Integers between and including 0 and N, M It is the length of the 2D sinc filter in the fast time. It is the lower cutoff frequency of the 2D sinc filter in fast time. b ft It is the bandwidth of the 2D sinc filter in fast time. m It is 0 to M Between and including 0 and M an integer, and w(n,m) It is a 2D cosine weighted function.

[0231] Example 10. A method according to one of Examples 1 through 5, wherein processing raw digital data using constrained L-dimensional convolutional layers includes processing the raw digital data using a 2D Morlet wavelet filter with constrained L-dimensional convolutional layers.

[0232] Example 11. A method based on one of Examples 1 through 5 or Example 10, where the 2D Morlet wavelet filter is defined by the following equation:

[0233]

[0234] in N It is the slow-time filter length. n It is 0 to N Between and including 0 and N integers, M It is the length of the fast-time filter. m It is 0 to M Between and including 0 and M integers, σ st It is the slow time standard deviation, σ ft It is the fast time standard deviation. It is the slow-time center frequency. It is the center frequency at high speed. It is the slow sampling frequency. It is the fast sampling frequency, and among them Defined by the following formula:

[0235] .

[0236] Example 12. A method based on one of Examples 1 through 5 or Example 10, where the 2D Morlet wavelet filter is defined by the following equation:

[0237]

[0238] in

[0239]

[0240]

[0241] Where α is the rotation angle, and α is not 0. N It is the slow-time filter length. n It is 0 to N Between and including 0 and N integers, M It is the length of the fast-time filter. m It is 0 to M Between and including 0 and M integers, σ st It is the slow time standard deviation, σ ft It is the fast time standard deviation. It is the slow-time center frequency. It is the center frequency at high speed. It is the slow sampling frequency. It is the fast sampling frequency, and among them It is given by the following formula:

[0242] .

[0243] Example 13. The method according to one of Examples 1 to 12 further includes training the neural network by the following steps: initializing the neural network; and after initializing the neural network, feeding training data into a constrained L-dimensional convolutional layer, wherein the trainable weights of the constrained L-dimensional convolutional layer include at least one of the following: slow time cutoff frequency, slow time bandwidth, fast time cutoff frequency, and fast time bandwidth.

[0244] Example 14. The method according to one of Examples 1 to 13 further includes normalizing the slow time cutoff frequency and the fast time cutoff frequency, and normalizing the slow time bandwidth and the fast time bandwidth.

[0245] Example 15. The method according to one of Examples 1 to 14 further includes training the neural network by the following steps: initializing the neural network; and after initializing the neural network, feeding training data into a constrained L-dimensional convolutional layer, wherein the trainable weights of the constrained L-dimensional convolutional layer include the center frequency and standard deviation for each dimension in the L-dimensional space.

[0246] Example 16. The method of one of Examples 1 through 15 also includes training the neural network for less than 21 rounds.

[0247] Example 17. A method based on one of Examples 1 through 16, wherein multiple additional layers include a first maxpool layer, followed by an unconstrained 2D convolutional layer, followed by a second maxpool layer, followed by a dense layer, and then a softmax layer.

[0248] Example 18. Based on the approach of one of Examples 1 through 17, where the target is a human target.

[0249] Example 19. A method based on one of Examples 1 through 18, wherein generating information about a target includes classifying the target based on a set of categories, and wherein the set of categories includes categories indicating human activity.

[0250] Example 20. A method according to one of Examples 1 through 19, wherein the set of categories includes: a walking category indicating human walking; an idle category indicating an idle human; a random arm movement category indicating a human exhibiting random arm movements; a waving category indicating a human performing a waving motion; and a work category indicating a seated human using a computer to work.

[0251] Example 21. A method based on one of Examples 1 through 20, wherein the set of categories includes categories that indicate human gestures.

[0252] Example 22. A method based on one of Examples 1 through 21, wherein the set of categories includes a first category indicating the presence of humans and a second category indicating the non-existence of humans.

[0253] Example 23. A method based on one of Examples 1 through 22, wherein the set of categories includes categories indicating the number of humans present.

[0254] Example 24. Based on the method of one of Examples 1 through 23, further include tracking the target based on information about the generation of the target.

[0255] Example 25. The method of one of Examples 1 through 24, wherein the multiple radar signals are multiple chirp signals.

[0256] Example 26. A method based on one of Examples 1 through 25, wherein multiple additional layers include convolutional layers with a first additional constraint.

[0257] Example 27. A method according to one of Examples 1 to 25, wherein generating information about a target includes generating a range-Doppler radar image indicating the location of the target, and the method further includes processing intermediate digital data using a second plurality of additional layers to generate a range-angle radar image indicating the location of the target.

[0258] Example 28. A radar system comprising: a millimeter-wave radar sensor including: a transmitting antenna configured to transmit a plurality of radar signals toward a target; a receiving antenna configured to receive a plurality of reflected radar signals; a mixer configured to mix copies of the plurality of transmitted radar signals with a plurality of received reflected radar signals to generate an intermediate frequency (IF) signal; an analog-to-digital converter (ADC) configured to generate raw digital data at the output of the ADC based on the IF signal; and an artificial intelligence (AI) accelerator having an input coupled to the output of the ADC, and configured to: receive the raw digital data from the ADC, and process the raw digital data using an L-dimensional convolutional layer constrained by a neural network to generate intermediate digital data, wherein L is a positive integer greater than or equal to 2, and wherein the neural network includes a plurality of additional layers; and process the intermediate digital data using the plurality of additional layers to generate target-associated data at the output of the AI ​​accelerator.

[0259] Example 29. The radar system according to Example 28 also includes a digital signal processor (DSP) with inputs coupled to the output of an AI accelerator, wherein the AI ​​accelerator is directly connected to the output of the millimeter-wave radar sensor.

[0260] Example 30. A radar system based on one of Examples 28 or 29, wherein the DSP is configured to track a target based on the output of an AI accelerator.

[0261] Example 31. A radar system according to one of Examples 28 to 30, wherein the AI ​​accelerator and DSP are integrated in the same integrated circuit.

[0262] Example 32. A radar system comprising: a millimeter-wave radar configured to transmit a plurality of chirped signals toward a target and configured to receive a plurality of reflected chirped signals; a mixer configured to mix copies of the plurality of transmitted chirped signals with a plurality of received reflected chirped signals to generate an intermediate frequency (IF) signal; an analog-to-digital converter (ADC) configured to generate raw digital data at the output of the ADC based on the IF signal; and a processor having an input coupled to the output of the ADC and configured to: receive the raw digital data from the ADC and process the raw digital data using a neural network having a two-dimensional convolutional layer with a first constraint, followed by a plurality of additional layers to generate data associated with the target at the output of the plurality of additional layers.

[0263] Example 33. A radar system based on Example 32, wherein the processor is an artificial intelligence (AI) accelerator.

[0264] Although the invention has been described with reference to illustrative embodiments, this description is not intended to be limiting. Various modifications and combinations thereof will be apparent to those skilled in the art from the description, exemplary embodiments, and other embodiments of the invention. Therefore, it is intended that the appended claims cover any such modifications or embodiments.

Claims

1. A method for processing data, comprising: Use millimeter-wave radar sensors to transmit multiple radar signals toward the target; The millimeter-wave radar is used to receive multiple reflected radar signals corresponding to the multiple transmitted radar signals; The copies of the plurality of transmitted radar signals are mixed with the plurality of received reflected radar signals to generate an intermediate frequency signal; The intermediate frequency signal is digitized using an analog-to-digital converter to produce raw digital data representing the digitized intermediate frequency signal; The original digital data is processed by an L-dimensional convolutional layer constrained by a neural network to generate intermediate digital data, where L is a positive integer greater than or equal to 2, and wherein the neural network includes multiple additional layers, wherein processing the original digital data includes filtering the digitized intermediate frequency signal in the fast time dimension and the slow time dimension using a time-domain 2D filter. as well as The intermediate digital data is processed using the multiple additional layers to generate information about the target.

2. The method of claim 1, wherein generating information about the target includes classifying the target based on a set of categories.

3. The method of claim 1, wherein generating information about the target includes providing the location of the target using radar imagery.

4. The method according to claim 3, wherein the radar image is a range-Doppler image, a range-angle image, or a Doppler-angle image.

5. The method according to claim 1, wherein L equals 2.

6. The method according to claim 5, wherein the kernel size of the filter of the constrained L-dimensional convolutional layer is greater than 10×10.

7. The method of claim 5, wherein processing the original digital data using the constrained L-dimensional convolutional layer comprises processing the original digital data using a 2D sinc filter of the constrained L-dimensional convolutional layer.

8. The method of claim 7, wherein the 2D sinc filter is defined by the following formula: wherein is a slow-time 1D sinc filter, N is a length of the slow-time 1D sinc filter, is a lower cutoff frequency of the slow-time 1D sinc filter, b st is a bandwidth of the slow-time 1D sinc filter, n is an integer between 0 and N inclusive and N inclusive, is a fast-time 1D sinc filter, M is a length of the fast-time 1D sinc filter, is a lower cutoff frequency of the fast-time 1D sinc filter, b ft is a bandwidth of the fast-time 1D sinc filter, m is an integer between 0 and M inclusive and M inclusive, w(n,m) is a 2D cosine weighting function, wherein the slow-time 1D sinc filter and the fast-time 1D sinc filter are defined by wherein K is the length of the 1D sinc filter, k is an integer between 0 and K inclusive, and K f s is the sampling frequency of the signal to be filtered, f l is the lower cut-off frequency, b is the bandwidth of the 1D sinc filter, and k is a filter parameter index.​ 9. The method of claim 7, wherein the 2D sinc filter is defined by the following formula: in, wherein a is the angle of rotation and wherein a is different from 0, wherein N is the length of the 2D sinc filter in slow time, is the lower cut-off frequency of the 2D sinc filter in slow time, b st is the bandwidth of the 2D sinc filter in slow time, n is an integer between 0 and N including 0 and N , and M is the length of the 2D sinc filter in fast time, is the lower cut-off frequency of the 2D sinc filter in fast time, b ft is the bandwidth of the 2D sinc filter in fast time, m is an integer between 0 and M including 0 and M , and w(n,m) is a 2D cosine weighting function.

10. The method of claim 5, wherein processing the original digital data using the constrained L-dimensional convolutional layer comprises processing the original digital data using a 2D Morlet wavelet filter of the constrained L-dimensional convolutional layer.

11. The method of claim 10, wherein the 2D Morlet wavelet filter is defined by the following formula: wherein N is a slow-time filter length, n is an integer between 0 and N inclusive, and N M is a fast-time filter length, m is an integer between 0 and M inclusive, and M σ st is a slow-time standard deviation, σ ft is a fast-time standard deviation, is a slow-time center frequency, is a fast-time center frequency, is a slow-time sampling frequency, is a fast-time sampling frequency, and wherein is defined by the following equation:​​ 。 12. The method of claim 10, wherein the 2D Morlet wavelet filter is defined by the following formula: in wherein a is the angle of rotation and wherein a is different from 0, wherein N is the slow time filter length, n is 0 to N and includes 0 and N an integer, M is the fast time filter length, m is 0 to M and includes 0 and M an integer, σ st is the slow time standard deviation, σ ft is the fast time standard deviation, is the slow time center frequency, is the fast time center frequency, is the slow time sampling frequency, is the fast time sampling frequency, and wherein is given by 。 13. The method of claim 1, further comprising training the neural network by the following steps: Initialize the neural network; and After initializing the neural network, training data is fed into the constrained L-dimensional convolutional layer, wherein the trainable weights of the constrained L-dimensional convolutional layer include at least one of the following: slow time cutoff frequency, slow time bandwidth, fast time cutoff frequency, and fast time bandwidth.

14. The method of claim 13, further comprising normalizing the slow cutoff frequency and the fast cutoff frequency, and normalizing the slow bandwidth and the fast bandwidth.

15. The method of claim 1, further comprising training the neural network by the following steps: Initialize the neural network; and After initializing the neural network, training data is fed into the constrained L-dimensional convolutional layer, wherein the trainable weights of the constrained L-dimensional convolutional layer include the center frequency and standard deviation for each of the L dimensions.

16. The method of claim 1, further comprising training the neural network for less than 21 rounds.

17. The method of claim 1, wherein the plurality of additional layers comprises a first maxpool layer, followed by an unconstrained 2D convolutional layer, followed by a second maxpool layer, followed by a dense layer, and followed by a softmax layer.

18. The method of claim 1, wherein the target is a human target.

19. The method of claim 18, wherein generating information about the target includes classifying the target based on a set of categories, and wherein the set of categories includes categories indicating human activity.

20. The method of claim 19, wherein the set of categories comprises: Walking category indicating human movement; Idle category indicating idle human movement; Random arm movement categories that indicate random arm movements exhibited by humans; The categories of waving that instruct humans to perform waving motions; and the categories of work that instruct seated humans to work using computers.

21. The method of claim 18, wherein the set of categories includes categories indicating human gestures.

22. The method of claim 18, wherein the set of categories includes a first category indicating the presence of humans and a second category indicating the absence of humans.

23. The method of claim 18, wherein the set of categories includes categories indicating the number of humans present.

24. The method of claim 1, further comprising tracking the target based on the generated information about the target.

25. The method according to claim 1, wherein the plurality of radar signals is a plurality.

26. The method of claim 1, wherein the plurality of additional layers includes a first additional constrained convolutional layer.

27. The method of claim 1, wherein generating information about the target includes generating a range-Doppler radar image indicating the location of the target, the method further comprising processing the intermediate digital data using a second plurality of additional layers to generate a range-angle radar image indicating the location of the target.

28. A radar system, comprising: Millimeter-wave radar sensors, including: The transmitting antenna is configured to transmit multiple radar signals toward the target; The receiving antenna is configured to receive multiple reflected radar signals; A mixer is configured to mix copies of the plurality of transmitted radar signals with the plurality of received reflected radar signals to generate an intermediate frequency signal. An analog-to-digital converter (ADC) is configured to digitize the intermediate frequency (IF) signal to produce raw digital data representing the digitized IF signal at the output of the ADC; and An artificial intelligence (AI) accelerator, having an input coupled to the output of the ADC, is configured to: Receive the raw digital data from the ADC, and The original digital data is processed using L-dimensional convolutional layers constrained by a neural network to generate intermediate digital data, where L is a positive integer greater than or equal to 2, and the neural network includes multiple additional layers, and the AI ​​accelerator is configured to process the original digital data using a temporal 2D filter to filter the digitized intermediate frequency signal in both fast and slow time dimensions; and The intermediate digital data is processed using the multiple additional layers to generate data associated with the target at the output of the AI ​​accelerator.

29. The radar system of claim 28, further comprising a digital signal processor (DSP) having inputs coupled to the output of the AI ​​accelerator, wherein the AI ​​accelerator is directly connected to the output of the millimeter-wave radar sensor.

30. The radar system of claim 29, wherein the DSP is configured to track a target based on the output of the AI ​​accelerator.

31. The radar system of claim 29, wherein the AI ​​accelerator and the DSP are integrated in the same integrated circuit.

32. A radar system, comprising: The millimeter-wave radar is configured to transmit multiple chirps toward the target and to receive multiple reflected chirps. A mixer is configured to mix copies of the plurality of transmitted chirped signals with the plurality of received reflected chirped signals to generate an intermediate frequency signal; An analog-to-digital converter (ADC) is configured to digitize the intermediate frequency (IF) signal to produce raw digital data representing the digitized IF signal at the output of the ADC. as well as The processor, having an input coupled to the output of the ADC, is configured to: Receive the raw digital data from the ADC, and The raw digital data is processed using a neural network having a first-constrained two-dimensional convolutional layer followed by multiple additional layers to generate data associated with the target at the output of the multiple additional layers, wherein the processor is configured to process the raw digital data using a temporal 2D filter to filter the digitized intermediate frequency signal in both fast and slow time dimensions.

33. The radar system of claim 32, wherein the processor is an artificial intelligence (AI) accelerator.

Images