Target detection method, integrated circuit, sensor and equipment

Abstract

The embodiment of the invention relates to the technical field of electromagnetic wave sensors, and discloses a target detection method, an integrated circuit, a sensor and equipment. The target detection method comprises the steps that FFT processing and beam forming processing are carried out on echo signals to generate feature vectors of the echo signals, and the feature vectors comprise energy features representing the echo signals on different distance units; and processing the feature vector based on a deep learning model to realize target detection of the region of interest. The accuracy and efficiency of target detection can be improved at least.

Description

Technical Field

[0001] The embodiments of the present application relate to the technical field of electromagnetic wave sensors, and in particular to a target detection method, an integrated circuit, a sensor, and a device. Background Art

[0002] When performing target detection in a sealed or relatively sealed space area, such as detecting targets in a car cabin, there may be technical problems such as low detection rate, a large number of false alarm targets, and inaccurate angle estimation. Summary of the invention

[0003] The embodiments of the present application provide a target detection method, integrated circuit, sensor and device, which are at least beneficial for effectively improving the detection rate and reducing the number of false alarm targets, while also significantly improving the accuracy of angle estimation and other effects.

[0004] According to some embodiments of the present application, a first aspect of the embodiments of the present application provides a target detection method, comprising: performing FFT processing on an echo signal to generate a feature vector of the echo signal, wherein the feature vector includes energy characteristics characterizing the echo signal at different distance units; processing the feature vector based on a deep learning model to achieve target detection in a region of interest, such as determining whether a target exists in the region of interest and distinguishing the type of target (for example, for in-cabin detection, it can identify whether there is someone in the cabin, and if so, whether it is an adult or a non-adult, etc.).

[0005] Optionally, after performing FFT processing on the echo signal, beam synthesis processing may be continued to obtain a feature vector of the echo signal, so that the detected information contains distinguishing information, etc.

[0006] Optionally, when the embodiment of the present application directly performs deep learning on FFT data to identify the target, all channels, or selected parts of the channels, or the results of non-coherent merging and combining of these channels can be used to perform CNN recognition operations, thereby achieving corresponding target detection.

[0007] According to some embodiments of the present application, the second aspect of the embodiments of the present application further provides an integrated circuit, comprising a radio frequency module, an analog signal processing module and a digital signal processing module connected in sequence; wherein the radio frequency module is used to generate a radio frequency transmission signal and receive a radio frequency reception signal; the analog signal processing module is used to down-convert the radio frequency reception signal to obtain an intermediate frequency signal; the digital signal processing module is used to perform analog-to-digital conversion on the intermediate frequency signal, and the digital data obtained by the analog-to-digital conversion is processed using the target detection method as described in any one of the first aspect of the embodiments of the present application to achieve target detection.

[0008] According to some embodiments of the present application, the third aspect of the embodiments of the present application further provides an electromagnetic wave sensor, comprising: a carrier; the integrated circuit as described in the second aspect of the embodiments of the present application, which is arranged on the carrier; an antenna, which is arranged on the carrier, or the antenna and the integrated circuit are integrated into an integrated device and arranged on the carrier; wherein the integrated circuit is connected to the antenna for transmitting radio frequency transmission signals and / or receiving radio frequency reception signals.

[0009] According to some embodiments of the present application, the fourth aspect of the embodiments of the present application also provides a terminal device, including: a device body; and an electromagnetic wave sensor as described in the third aspect of the embodiments of the present application, which is arranged on the device body; wherein the electromagnetic wave sensor is used for target detection and / or communication to provide reference information for the operation of the device body.

[0010] The technical solution provided in the embodiment of the present application has at least the following advantages:

[0011] Through beamforming processing, the signal-to-clutter ratio of human targets and static strong clutter is effectively improved, and the feature vectors of empty vehicles (with or without interference), scenes with adults and scenes with children are effectively constructed. The target information contained in the feature vectors is deeply mined through a deep learning model to achieve more accurate target detection in spatial areas such as car cabins, indoors, and factories. While effectively improving the detection rate and reducing the number of false alarm targets, it can also greatly improve the accuracy of angle estimation and other effects, thereby enabling special targets or weak targets such as infants in the cabin to be accurately detected, realizing applications such as CPD (Child Presence Detection) and SBR (Safety Belt Reminder). BRIEF DESCRIPTION OF THE DRAWINGS

[0012] One or more embodiments are exemplarily described by pictures in the corresponding drawings, and these exemplified descriptions do not constitute limitations on the embodiments. Elements with the same reference numerals in the drawings represent similar elements, and unless otherwise stated, the figures in the drawings do not constitute proportional limitations.

[0013] Figure 1 is a flow chart of the target detection method provided in an embodiment of the present application;

[0014] Figure 2 is a flow chart of the target detection method provided in the embodiments of the present application;

[0015] Figure 3 is a flow chart of the target detection method provided in an embodiment of the present application;

[0016] Figure 4is a flow chart of the target detection method provided in an embodiment of the present application;

[0017] Figure 5 is a flow chart of the target detection method provided in an embodiment of the present application;

[0018] Figure 6 is a schematic diagram of dividing the region of interest involved in the target detection method provided in the embodiment of the present application;

[0019] Figure 7 is a schematic diagram of data rearrangement involved in the target detection method provided in the embodiments of the present application;

[0020] Figure 8 is a schematic diagram of the structure of a deep learning model involved in the target detection method provided in the embodiments of the present application;

[0021] Fig. 9 is a flow chart of the target detection method provided in an embodiment of the present application;

[0022] Fig.10 is a flow chart of the target detection method provided in the embodiments of the present application;

[0023] Fig.11 is a flow chart of the target detection method provided in the embodiments of the present application;

[0024] Fig.12 is a flow chart of the target detection method provided in the embodiments of the present application;

[0025] Figure 13 to Figure 22 is a schematic diagram of a method for target detection provided in another embodiment of the present application;

[0026] Fig.23 It is a schematic diagram of data rearrangement involved in the target detection method provided in the embodiment of the present application. DETAILED DESCRIPTION

[0027] In order to make the purpose, technical scheme and advantages of the embodiments of the present application clearer, the embodiments of the present application will be described in detail below in conjunction with the accompanying drawings. However, it will be appreciated by those skilled in the art that in the embodiments of the present application, many technical details are provided in order to enable the reader to better understand the present application. However, even without these technical details and various changes and modifications based on the following embodiments, the technical scheme claimed in the present application can be implemented.

[0028] The division of the following embodiments is for the convenience of description and shall not constitute any limitation on the specific implementation of the present application. The embodiments may be combined with each other and referenced to each other without contradiction.

[0029] In one aspect, the present application provides a method for detecting a target. In some embodiments, Figure 1 As shown, the target detection method includes the following steps:

[0030] Step 101 : Perform FFT processing and beamforming processing on the echo signal to generate a feature vector of the echo signal, wherein the feature vector includes energy characteristics representing the echo signal at different distance units.

[0031] Step 102: Process the feature vector based on the deep learning model to achieve target detection in the region of interest.

[0032] In this way, through beamforming processing, the signal-to-clutter ratio of human targets and static strong clutter is effectively improved, and the feature vectors of empty vehicles (with or without interference), adult scenes, and child scenes are effectively constructed. The target information contained in the feature vectors is deeply mined through the deep learning model to achieve more accurate detection. The solution for target detection in spatial areas such as car cabins, indoors, and factories can effectively improve the detection rate and reduce the number of false alarm targets. At the same time, it can also greatly improve the accuracy of angle estimation, so that special targets or weak targets such as infants in the cabin can be accurately detected, and applications such as CPD and SBR can be realized. At the same time, the introduction of deep learning modules can also reduce the required digital signal processing. For example, after extracting the features, there is no need to perform CFAR, DOA, etc.

[0033] To better understand Figure 1 The embodiment shown in FIG. 1 is shown, and its steps are described below.

[0034] In step 101, the implementation of FFT processing and beamforming processing is not limited. It can be understood that since beamforming processing is essentially to focus the directional gain (capable of reflecting energy) of the signal in a specified direction, beamforming is actually equivalent to extracting energy features. In other words, different implementations of FFT processing and beamforming processing do not affect the extraction of energy features, and thus do not affect the generation of feature vectors including energy features characterizing the echo signal at different distance units.

[0035] To facilitate understanding of the implementation of the FFT processing and beamforming processing of the echo signal in step 101, the following will be combined with Figure 2 The target detection process (without using a deep learning model) implemented based on multi-frame accumulation technology is illustrated.

[0036] like Figure 2As shown, after the radar (taking the target detection of the frequency modulated continuous wave (FMCW) radar as an example) performs analog-to-digital conversion, sampling and other operations to process the antenna receiving the echo signal, it will provide further processing through the main control module (for example, it can be implemented based on a microcontroller unit (MCU)) and a baseband (BB) module (for example, it can be implemented based on a baseband chip) to achieve target detection: range DC removal (Range DC removal) and range Fourier transform (Range FFT, which can also be expressed as 1D-FFT or fast time FFT) are performed on the baseband module in sequence, and then the baseband module sends the signal to the main control module to perform multi-frame accumulation on the main control module. When the data accumulates to a preset number of frames, such as 128 frames, the main control module moves the accumulated multi-frame data from the CPU static random access memory (SRAM) to the baseband module, and continues to perform Doppler DC removal (Doppler DC The method further comprises the following steps: performing noise estimation (such as using a noise variance estimator) in the baseband module to obtain a constant false alarm rate (CFAR) such as peak-based CFAR (detection with peak selection), and finally performing angle detection and target classification to obtain the target's distance, speed and / or angle information.

[0037] in, Figure 2 The relevant steps in can be implemented as follows:

[0038] Distance-dimensional DC removal: This is for the data corresponding to each chirp in each frame of data. At this time, the distance-dimensional DC removal is specifically as follows: the data collected by each receiving (RX) channel of each chirp is averaged along the fast time dimension, and then the DC component of all sampling points of each RX channel is subtracted.

[0039] Distance dimension FFT: This can be achieved by windowing the data after the DC in the distance dimension is removed and then performing 1D-FFT.

[0040] It should be emphasized that in the process of target detection, each chirp data in each frame of data obtained by radar through ADC or sampling based on the echo signal is sent to the main control module through the above operation. The data is sent to the main control module by moving the data from Direct Memory Access (DMA) to CPU SRAM.

[0041] Doppler DC removal: This can be achieved by calculating the complex average along the frame dimension of the multi-frame data accumulated on the main control module (i.e. averaging the data of different frames in the same distance unit (bin) of the same channel), obtaining the average values ​​of different distances in different channels as the DC components of the corresponding distance units of the corresponding channels, and subtracting the respective DC components from each distance unit of each channel.

[0042] Inter-frame Fourier transform: It can be implemented by windowing the data after the Doppler dimension DC is removed along the frame dimension, and performing 2D-FFT along the frame dimension, at which time the distance dimension-Doppler information of each transmitting and receiving channel will be obtained. For example, in some embodiments, it can be implemented as follows: after the echo signal is processed by the distance dimension FFT, the result obtained by the distance dimension FFT processing is sliding windowed, and the windowed data is subjected to 2D FFT to obtain a distance-Doppler spectrum; wherein the length of the sliding window corresponds to a duration of the same order of magnitude as the period of the periodic motion of the target. In this way, by performing digital signal processing on multiple frames of data with similar periods of the periodic motion of the target, it is possible to better detect the target in combination with the characteristics of the target motion period, avoid errors and omissions, and also help to eliminate the influence of multipath, which is conducive to improving the accuracy of target detection.

[0043] Incoherent accumulation: It can be realized by the following expression:

[0044]

[0045] or,

[0046]

[0047] Wherein, P(r,v) and A(r,v) are the power and amplitude at the r-th range unit and the v-th Doppler unit, respectively, and s(r,t,a,v) is the complex value obtained by inter-frame Fourier transform of the signal echo of the r-th range unit and the v-th Doppler unit on the t-th transmission channel and the a-th receiving channel.

[0048] The constant false alarm detection based on the noise floor obtained by noise estimation has been described in the existing CFAR technology, and will not be described in detail here.

[0049] Angle detection: It can be achieved by performing azimuth digital beam forming (DBF) and DOA on the target point obtained by CFAR, as well as elevation DBF and DOA. DBF and DOA have been described in the existing DBF and DOA technologies, so they will not be described here one by one.

[0050] Target classification: It can be achieved by counting the target points detected in each frame (or preset data volume, i.e. each time) according to the preset (or divided) area unit design, and the counting results can be judged according to preset rules to obtain whether there is a physical target of interest (such as an adult, child, infant or pet, etc.) in this processing, and determine its location area.

[0051] It should be noted that for radars with baseband modules and main control modules, Figure 2 The above is only an example, that is, frame data accumulation, target classification (which can be implemented by Region HIST, classification, etc.) and other operations are performed in the main control module, and other operations are performed in the baseband module, such as range dimension DC removal and Doppler dimension DC removal, etc. In other embodiments, frame data accumulation can also be performed in the baseband module, and Doppler dimension DC removal can be performed in the main control module.

[0052] It should also be noted that Figure 2 Only caching 128 frames of data is used as an example for explanation. In other embodiments, 56 frames of data or 32 frames can also be cached. The embodiments of the present application are not limited to this and can be determined according to demand or hardware support capabilities. In this way, the cached multi-frame data will continue to perform Doppler dimension DC removal and inter-frame FFT to obtain a range-Doppler (Range-Doppler, RD) diagram.

[0053] It should also be noted that Figure 2 In the process shown, some steps can be skipped, such as distance-dimensional DC removal or distance-dimensional DC removal or windowing; some steps can be simplified, such as non-coherent accumulation can be skipped, and only one of the channels is subjected to CFAR detection, or only some channels are selected for non-coherent accumulation, and only these channels are subjected to CFAR detection.

[0054] Depend on Figure 2 As can be seen from the process shown, the target detection process performs beamforming processing to achieve angle estimation, and subsequent processing such as target classification is required to output intuitive and accurate results. Figure 1In the illustrated embodiment, beamforming processing is used to generate feature vectors, and the subsequent angle estimation, target classification, etc. are replaced by deep learning model processing, which is more efficient. In addition, since the deep learning model does not directly use time series data as input, but further extracts target information from the echo signal through FFT processing and beamforming processing to form feature vectors with higher dimensions and richer information, the deep learning model can better perform analysis and processing, and output more accurate results, which can improve the accuracy and efficiency of target detection and achieve more accurate detection and identification of targets in the cabin. Among them, since the noise and target reflection signals have different energy characteristics, the use of feature vectors that characterize the energy characteristics will help the deep learning model to better detect targets and output more accurate and reliable results.

[0055] It is understandable that the application of deep learning models is to replace CFAR, angle estimation and other processing to directly output the detection results. Figure 2 Based on the process shown in FIG. 1 , beam synthesis is performed after inter-frame FFT, and the processing of correlation accumulation, CFAR, angle estimation, etc. is replaced by the processing of the deep learning model. That is, in some embodiments, such as Figure 3 As shown, the FFT processing and beamforming processing of the echo signal can be achieved through the following steps:

[0056] Step 1011, perform FFT processing on the echo signal to generate a range-Doppler spectrum.

[0057] Step 1012: Perform beamforming processing according to the range-Doppler spectrum.

[0058] It is also understandable that Figure 2 As shown, the FFT processing includes the distance dimension FFT and the inter-frame FFT. After the distance dimension FFT, the beamforming processing can be supported without waiting for the inter-frame FFT. Moreover, as mentioned above, the implementation order of the beamforming processing is different, and the energy characteristics can still be obtained and the feature vector can still be generated. Therefore, in some embodiments, Figure 4 As shown, the FFT processing and beamforming processing of the echo signal can also be achieved through the following steps:

[0059] Step 1013, performing range FFT processing on the echo signal;

[0060] Step 1014, performing beam synthesis processing on the result obtained by the range FFT processing;

[0061] Step 1015: Sliding windowing is performed on the result of the beamforming process, and 2D FFT is performed on the windowed data. Optionally, the 2D-FFT here may also be a sliding window 2D-FFT.

[0062] It can also be understood that after the distance dimension FFT processing, beamforming processing is already supported, and the subsequent inter-frame FFT is to better detect targets through digital signal processing. The deep learning model has the ability to learn implicit features, so even if the inter-frame FFT is not used, good target detection results can still be obtained, and good results can still be achieved without inter-frame FFT. Based on this, in some embodiments, such as Figure 5 As shown, the FFT processing and beamforming processing of the echo signal can be achieved through the following steps:

[0063] Step 1016, performing range FFT processing on the echo signal;

[0064] Step 1017: Perform beam synthesis processing on the result obtained by the range FFT processing.

[0065] It should be noted that Figure 3-Figure 5 The related processing (including range FFT, frame data accumulation, inter-frame FFT, etc.) performed in the illustrated embodiment is Figure 2 The embodiments shown have been described and will not be described in detail here.

[0066] Of course, the above is only an example of the implementation of FFT processing and beamforming processing. In some embodiments, other suitable methods may be used for processing, which will not be listed here one by one.

[0067] It is understandable that in Figure 4 and Figure 5 In the embodiment shown, the beamforming process is performed based on the result of the range-dimensional FFT process. Figure 3 In the illustrated embodiment, the beamforming process is performed based on the result of the inter-frame FFT process. Therefore, the implementation method of the beamforming process may also be different. Specifically, in Figure 4 and Figure 5 In the embodiment shown, beam synthesis is implemented in the distance dimension, that is, synthesis accumulation is performed on different distance units, and Figure 3 In the illustrated embodiment, beamforming can be implemented in the range dimension and the Doppler dimension, that is, beamforming accumulation is performed in different Doppler units of different range units. Figure 3 The implementation in the illustrated embodiment is described by taking an example.

[0068] In some examples, beamforming is performed based on the range-Doppler spectrum obtained by inter-frame FFT (to perform N c Taking spot beamforming as an example) it can be realized by the following expression:

[0069]

[0070] Wherein, P(r,v,b) is the energy characteristic of the vth Doppler unit of the rth range unit based on beam b, sv(c,b) is the steering vector of beam b on virtual channel c, x(c,r,v) is the data corresponding to the vth Doppler unit of the rth range unit on virtual channel c in the range-Doppler spectrum, N c is the total number of beams.

[0071] Correspondingly, when beamforming is performed based on data obtained by distance dimension FFT, x(c, r, v) in the above expression is replaced by x(c, r), that is, the Doppler dimension is not included. Of course, in other embodiments, other methods can be used for implementation, which will not be described here one by one.

[0072] Regarding the steering vector of the beam used in beam synthesis, the embodiments of the present application do not limit it, and it can be specified by the user, or it can be automatically generated according to the region of interest, etc. For example, in some embodiments, beam synthesis can be achieved in the following manner: determine the target direction pointing to the region of interest and generate a steering vector according to the target direction; perform beam synthesis according to the steering vector. Among them, regarding the determination of the target direction, the embodiments of the present application do not limit it. For example, in some examples, determining the target direction pointing to the region of interest includes: determining the target direction according to the azimuth and elevation angles of the region of interest relative to the radar. In other examples, determining the target direction pointing to the region of interest includes: obtaining the measured data of the target in the region of interest; clustering according to the measured data of the target, so as to use the azimuth and elevation angles of the cluster center points in each region of interest as the target direction.

[0073] For example, Figure 6 For the interior space shown in the figure, the scenario requirement is to determine whether there are people in the six areas of the three rear seats and the aisle in front of the three seats. If there are people, determine whether they are adults or children. Then, 6-point DBF can be performed, that is, 6 target directions can be determined, pointing to Figure 6 As shown in the figure, there are 3 seats and 3 aisles. At this time, P(r,v,b) corresponding to 6 beams will be obtained. Of course, other directions can also be added, such as adding three auxiliary beams pointing to the other three areas. At this time, a 9-point DBF is performed to obtain P(r,v,b) corresponding to nine beams.

[0074] The generation of the steering vector can be achieved through the following expression:

[0075] sv(c,b)=exp(1j·2π / λ·(x c cosφ b sinθ b +z c sinφ b)),

[0076] Where sv(c,b) is the steering vector of beam b on virtual channel c, λ is the central wavelength of the radar detection signal, and x c and z c are the horizontal and vertical positions of the antenna corresponding to the virtual channel c, θ b is the azimuth corresponding to the target direction, φ b is the pitch angle corresponding to the target direction.

[0077] It should be noted that, since the coverage of the radar is usually larger than the range of the target detection required by the user, the target detection usually occurs in the user's area of ​​interest. Based on this, in the embodiments of the present application, the target detection in the area of ​​interest is mainly described as an example, but it does not mean that the target detection method provided in the embodiments of the present application can only be used for the area of ​​interest. For example, it can also be used for all detectable areas, etc., which will not be described one by one here and later.

[0078] It is also understandable that the input of the deep learning model will also affect the accuracy of the deep learning model. Considering that the information of the target in the echo signal also has characteristics in a certain arrangement order. Therefore, in order to help the deep learning model better process the feature vector. The energy features in the feature vector can also be arranged. Based on this, in some embodiments, the feature vector of the echo signal is generated. It can be achieved in the following way: according to the preset rules, the energy features obtained by FFT processing and beamforming processing of the echo signal are rearranged to obtain the feature vector. In this way, by rearrangement, the correlation of the data in the feature vector is strengthened, which is conducive to better detection of the model.

[0079] It should be noted that the embodiments of the present application do not limit the specific rules for rearrangement. For example, in some embodiments, the preset rules include: the data with the same first dimension are arranged continuously in the changing direction of the second dimension, and the whole is arranged continuously in the changing direction of the first dimension, wherein the first dimension and the second dimension are different dimensions in the beam dimension and the distance dimension, respectively. At this time, it can correspond to the situation where only the energy characteristics in the distance dimension are generated. For example, in some embodiments, the preset rules include: the data with the same third dimension and the same fourth dimension are arranged continuously in the changing direction of the fifth dimension, the data with the same third dimension are arranged continuously in the changing direction of the fourth dimension, and the whole is arranged continuously in the changing direction of the third dimension, wherein the third dimension, the fourth dimension and the fifth dimension are different dimensions in the beam dimension, the distance dimension and the Doppler dimension, respectively. At this time, it can correspond to the situation where the energy characteristics in the distance dimension and the Doppler dimension are generated.

[0080] For ease of understanding, the following will be combined Figure 7The rearranged feature vector shown illustrates the preset rules of the rearrangement.

[0081] like Figure 7 As shown, the energy features are arranged in physical memory continuously along the v dimension, followed by the r dimension, and finally the b dimension. In the figure, v(0) and v(Nv) represent the 0th and Nvth Doppler bins (a total of Nv+1 Doppler bins), r(0) and r(Nr) represent the 0th and Nrth range bins (a total of Nr+1 rangebins), and b0 represents the 0th beam, with a total of N beams. This array of (Nv+1)×(Nr+1)×N energy features is the feature vector input to the deep learning model.

[0082] It is also understandable that, as mentioned above, target detection is implemented based on hardware, and therefore, in order to adapt to the capabilities of the hardware, data processing may also be performed to adapt to the capabilities of the hardware.

[0083] For example, in some embodiments, generating a feature vector of an echo signal can be achieved by performing logarithmic normalization on the energy features obtained by performing FFT processing and beamforming processing on the echo signal. Thus, by logarithmic normalization, data is converted from floating point numbers to fixed point numbers, which is conducive to improving the efficiency of subsequent processing, thereby improving the efficiency of target detection.

[0084] For example, in some embodiments, the energy features obtained by performing FFT processing and beamforming processing on the echo signal are logarithmically normalized, which can also be achieved by taking the logarithm of the energy features obtained by performing FFT processing and beamforming processing on the echo signal, and linearly normalizing the logarithm result using preset upper and lower bound constraints. That is, on the basis of logarithmic normalization, the upper and lower bounds, as well as the linear parameters, are further introduced, so that the data format is further optimized, which is more conducive to forming the feature data used subsequently.

[0085] Among them, the energy characteristics obtained by performing FFT processing and beamforming processing on the echo signal are logarithmized, and the result after taking the logarithm is linearly normalized using the preset upper and lower bound constraints, which is achieved through the following expression:

[0086]

[0087] Among them, P norm (r, v, b) is the logarithmic normalization result, P(r, v, b) is the energy characteristic of beam b at range unit r and Doppler unit v, a0, b0, P max and P min These are preset parameters.

[0088] It should be noted that the embodiments of the present application do not limit the specific values ​​of the above preset parameters. In some possible embodiments, b0 can be set to the mean or median of all possible log2(P(r,v,b)). Alternatively, in some embodiments, a0 is the difference between the 95% quantile and the 5% quantile of all possible log2(P(r,v,b)) values ​​divided by 256. In this way, P norm (r, v, b) are normalized to between -127 and 127, which is the range of fixed-point numbers.

[0089] It should be noted that the above is only an example of the processing of relevant data. In some embodiments, other processing methods can be used. For example, when taking the logarithm, the logarithm of 10 is taken instead of 2. Here, the logarithm of 2 can be taken to normalize the data to -127~127 through the above expression. Other logarithm taking methods or other processing methods can also achieve similar effects. Alternatively, instead of quantization, conversion into double-precision floating point or single-precision floating point, such as int32, int16, int8 or uint32, uint16, uint8, etc. can also improve subsequent processing efficiency, which will not be elaborated here.

[0090] Among them, regarding the normalization embodiments provided in the above embodiments, their implementation can be implemented in various forms, for example, the complex data is transferred to the MCU Mem via DMA and then performed in the MCU, or it can be implemented by the hardware engine in the BB and then directly transferred to the MCU Mem via DMA, or it can be partially implemented in the BB and transferred to the MCU Mem via DMA and then the remaining calculation is performed in the MCU (such as the modulo square operation and the logarithm operation of 2 can be implemented in the BB, and the subtraction, division and saturation truncation can be implemented in the MCU), etc., which will not be described one by one here.

[0091] In step 102, there is no limitation on the deep learning model used, which can be any deep learning model, such as convolutional neural networks (CNN) or Transformer, etc., which will not be described in detail here.

[0092] As for the processing process and result output of the deep learning model, it can be understood that it can be adjusted by setting the structure of the deep learning model and training the model.

[0093] For example, in some embodiments, the target detection result is a confidence matrix, and the elements in the confidence matrix correspond to indicating at least one of the following information: the confidence that the target is detected, the confidence that the target exists in the corresponding region of interest, the confidence that an adult target exists in the corresponding region of interest, and the confidence that a child target exists in the corresponding region of interest.

[0094] In some embodiments, the structure of the deep learning model is a CNN with three layers of convolution kernels, and the type, input and output size of each layer are as follows: Figure 8 As shown. At this time, the input feature dimension of the deep learning model is 6×64×20, corresponding to 6 beam pointings, 64 Doppler units and 20 range units. Each convolution operation contains a two-dimensional convolution layer, a batch normalization layer, a ReLu nonlinear activation layer and a max pooling layer. After three convolution operations, the resulting 128×2 feature vector is flattened to one dimension to obtain a 256×1 one-dimensional feature vector. After random dropout, it passes through a 256×10 fully connected layer to obtain the detection result. For example, whether there is someone in the car, and if so, the confidence that there are adults and children in each seat and aisle in the car.

[0095] At this point, it is understandable that, based on the output of the deep learning model, further processing can be performed before outputting it to the user. Figure 6 Taking the interior space of the car shown as an example, assume that the output of the deep learning model is as follows: a 1×10 vector yi formed by the confidence levels of whether there is someone in the car, whether there are children in the three seats, whether there are children in the three aisles, and whether there are adults in the three seats. At this point, the following judgments can be made: if yi(1)>0, it is determined that there is a person in the car without considering the other bits of yi, or if yi(k)=0, k=1, 2, ..., 10, it is also determined that there is no one in the car; if there is k=2, 3, ..., 10, so that yi(k)>0, it is determined that there are children or adults in the corresponding area; if the confidence levels of adults and children in seat A and aisle A are both greater than 0, it is determined that there is an adult in seat A, and so on for seat B and aisle B and seat C and aisle C. If yi=[-0.1, 7.2, -5, -7.2, -1.5, -1.9, -3.1, -6.4, -6.2, -5.5], it is determined that there is a child in seat A; if yi=[-0.1, -5, -4.2, 7.2, -1.5, -1.9, -3.1, 6.4, -6.2, -5.5], it is determined that there is a child in aisle C and an adult in seat A; if yi=[-0.1, -5, -4.2, 7.2, -1.5, -1.9, -3.1, -6.4, -6.2, 5.5], it is determined that there is only an adult in seat C; if yi=[0.1, -5, -4.2, 7.2, -1.5, -1.9, -3.1, 6.4, -6.2, -5.5], it is determined that the car is empty. At this point, it can be determined that there are multiple people in the car, and false alarms can be greatly reduced, but the risk of missed detection is slightly increased.

[0096] Regarding the training method of the deep learning model, in some embodiments, such as Fig. 9 As shown, before processing the feature vector based on the trained deep learning model to obtain the target detection result output by the deep learning model, the method also includes the following steps:

[0097] Step 103 obtains a training feature vector, wherein the training feature vector carries a one-hot encoding label and is used to characterize energy features of the signal at different distance units.

[0098] Step 104, using a preset data enhancement method to process the training feature vector to obtain a training set, the preset data enhancement method includes at least one of the following: randomly adding white noise, flipping the data along the Doppler dimension, and translating the data along the distance dimension.

[0099] Step 105: train the deep learning model according to the training set.

[0100] In order to help those skilled in the art better understand the above embodiments, the training of the deep learning model is illustrated below.

[0101] First, we obtain the training feature vectors by following the method of obtaining feature vectors mentioned above, and accumulate the multiple processed data of multiple groups of experiments to construct the input feature data set. Among them, each element P i It is the aforementioned (Nv+1)×(Nr+1)×N array.

[0102] Then, according to the experimental conditions, Marking, that is, P i Generate the corresponding label L i , obtain the labeled input feature dataset At this time, the generated label L i One-hot encoding can be used. Figure 6 For the in-car scene shown in the figure, if we consider judging whether there are people in the three rear seats and three aisle areas in the car, and classify adults and children at the same time, and do not consider the situation where adults squat in the three aisle areas, then there are 10 classification problems in total. i is a 1×10 vector, with values ​​of 0 or 1. i The first bit indicates whether there is someone in the car. If no one is in the car, the bit is 1, if there is someone, the bit is 0; the 2nd to 4th bits indicate whether there is a child in seat A-seat C. If yes, the bit is 1, if not, the bit is 0; the 5th to 7th bits indicate whether there is a child in aisle A-aisle C. If yes, the bit is 1, if not, the bit is 0; the 8th to 10th bits indicate whether there is an adult in seat A-seat C. If yes, the bit is 1, if not, the bit is 0. If no one is in the car, the corresponding L iis [1,0,0,0,0,0,0,0,0,0]; if there is an adult in seat A, then the corresponding value is L i is [0,0,0,0,0,0,0,1,0,0]; if there is a child in aisle A and an adult in seat B, then L i is [0,0,0,0,1,0,0,0,1,0]. Note that L i The first digit of the vehicle, i.e., whether there is someone in the vehicle, can also be omitted. i It is a 1×9 vector. When it is all zero, it means there is no one in the car.

[0103] Next, The enhanced data set is obtained by randomly adding white noise, randomly flipping along the Doppler dimension, and randomly translating a small amount along the distance dimension. As a result, the number of samples in the enhanced dataset is greatly increased compared to the distant dataset. There are 5,000 samples of hollow cars, children, and adults. Ten rounds of data augmentation can be performed to obtain an augmented dataset with 50,000 samples of each type. (You can choose to include or not include the original dataset ). Among them, you can also Random sorting, taking α = 0.3 ratio As a validation dataset The remaining data is used as the training data set

[0104] Then, deep learning model training is performed as follows:

[0105] Build an uninitialized CNN model and randomly initialize the parameters of each layer;

[0106] The following steps are repeated 20 times (each time is called an epoch):

[0107] Will Randomly divide into N b batches, each batch is N s For each batch of samples, perform the following steps:

[0108] Each batch of data is again subjected to one or more data enhancement processes including randomly adding white noise, randomly flipping along the Doppler dimension, and randomly translating a small amount along the distance dimension;

[0109] The enhanced data is sent to CNN for forward calculation to obtain three types of confidence: no one exists, adult exists, and child exists, and the corresponding loss CE is obtained based on the loss function t ;

[0110] Perform back propagation on the CNN network model and update the parameters of each layer of CNN based on the gradient descent optimization algorithm;

[0111] In all After the above operations, the updated CNN is used to Forward calculation is performed on all samples of The loss of CE v , if CE v In the setting N es If the value fails to decrease in the next cycle, the cycle will be terminated early;

[0112] Get CE v The CNN model with the smallest cycle is output as the final model.

[0113] In the training process, Adam Optimizer can be used as a gradient descent optimizer, and cross entropy can be used as the loss function. However, sigmoid is used for normalization operation before cross entropy is used for loss calculation at the network output. The learning rate is set to 0.001, the weight decay is set to 0.0001, and N s Set to 128, N es Set to 5.

[0114] Of course, the above is only an example for deep learning models. In other embodiments, models with other structures or CNNs with different parameters, or different training parameters or training methods, etc. can be used, which will not be described in detail here.

[0115] It should be noted that the aforementioned embodiments are only examples. In some embodiments, the output and label assignment may not use one-hot encoding but may take other forms. For example, if only the scene of no one in the car or only one person is considered, Li = 1, 2, ..., 10 is directly generated as the label, and the generated yi is searched for the position of its maximum value as the detection and recognition result, etc., which will not be elaborated here.

[0116] It can also be understood that the radar has a large coverage range, but the user does not have a detection requirement for all of its coverage ranges. Therefore, in some embodiments, the feature vector is used to characterize the energy characteristics of the echo signal at different distance units within a preset distance range and / or a preset Doppler range. For example, when performing beamforming, only the data of some Doppler units and distance units are used, such as 0-32 and 96-127 Doppler units and 10-42 distance units. In this way, only the part that the user is interested in is retained to reduce the occupation of resources such as calculation and storage, and improve detection efficiency.

[0117] In order to facilitate those skilled in the art to better understand the application of the target detection method provided by the above embodiment in the radar target detection process, the following will be combined with Figure 3-Figure 5 The application of the illustrated embodiment is described by way of example.

[0118] for Figure 3 The embodiment shown in the figure, after being applied to the complete target detection process, will be as follows Fig.10 As shown, that is, Figure 2 On the basis of the process shown, after the inter-frame FFT (Frame FFT) processing, two-dimensional digital beamforming (2D-DBF) processing and DL-based target classification processing (DL-based Classification) operations are directly performed. That is, after the inter-frame FFT, CFAR processing is not performed, but 2D-DBF operations are performed on the azimuth and elevation dimensions for each region of interest (or unit interval) to obtain a number of RD spectra corresponding to the region of interest; subsequently, the above-mentioned RD spectra are input into the constructed deep learning model (DL) for target judgment. In some optional embodiments, the relationship between the above-mentioned region of interest and the RD spectra can also be one-to-many, that is, at least two RD spectra can be obtained based on one region of interest, and the specific number can be adjusted according to actual needs.

[0119] The above DL input can be linear or dB domain amplitude or power, and normalization processing and other operations can also be performed. At the same time, the deep learning model constructed above can be classified according to the input, and the categories distinguished can be: whether there is a target, the attributes of the target, the specific unit interval location (i.e., regional location) of each target, etc.

[0120] For example, the rear area of ​​the car is the area of ​​interest or the target area (i.e. Figure 6 In the case of an application scenario where the 6 regions of interest (the regions of interest are also set as 3 seat regions) are divided and the corresponding 3 aisle regions are used, after the inter-frame FFT, the azimuth and elevation 2D-DBF can be performed on the center positions of the above 6 regions of interest (the regions of interest are also set as 3 seat regions) to obtain 6 RD spectra (or 3 RD spectra) corresponding to the regions of interest, and at least part of the above 6 RD spectra (or 3 RD spectra) are input into the constructed deep learning model for target judgment and differentiation. For example, the constructed deep learning model performs classification based on the input to determine whether there is a living target in the above region of interest. If there is a living target, it can be further determined whether the living target is an adult, a child, an infant or a pet, etc., and can further determine the specific location information of the living target, such as which seat or aisle region the living target is in.

[0121] Fig.10In the process shown, the target detection method is a processing process based on dual drive of model data. It has low requirements for signal processing and can effectively avoid steps such as selection of signal processing parameters, regional parameters, design of CFAR logic and regional judgment logic, thereby greatly reducing the difficulty of solution implementation and design.

[0122] for Figure 4 The embodiment shown in the figure, after being applied to the complete target detection process, will be as follows Fig.11 As shown. Fig.11 As shown, it can be Figure 2 Based on the process shown, after the range Fourier transform (Range FFT, also called 1D-FFT), 2D-DBF is performed first, and then frame data accumulation and inter-frame FFT processing are performed. That is, before the inter-frame FFT, the azimuth and elevation angles of the center position of the region of interest (such as the three-seat area) are firstly performed 2D-DBF. This embodiment can realize the solution of parallel processing of data processing and wave transmission time, thereby effectively reducing the processing time.

[0123] for Figure 5 The embodiment shown in the figure, after being applied to the complete target detection process, will be as follows Fig.12 As shown. Fig.12 As shown, it can be Figure 2 On the basis of the process shown, that is, after the range-dimensional Fourier transform (Range FFT, also called 1D-FFT), the frame data is accumulated after 2D-DBF, and the DL-based target classification operation (ComplexDL-based Classification) is directly performed. That is, in the entire target detection process, no inter-frame FFT processing operation is performed, and 2D-DBF is directly performed on the center position of the area of ​​interest (such as the 3 seat area) after 1D-FFT to obtain the corresponding number of distance-frame maps (such as 3 range-frames), and then the above distance-frame maps are input into the constructed (for example, complex) deep learning model for target differentiation and judgment and other operations.

[0124] exist Fig.12 In the embodiment shown, since the FFT processing step between frames is avoided, the amount of calculation can be effectively reduced; at the same time, if a complex deep learning model is adopted, a higher processing gain can be obtained compared to FFT, thereby achieving the purpose of improving target detection performance; in addition, this embodiment also effectively shortens the processing level, which can facilitate its efficient scheduling operations, etc.

[0125] From the above, we can see that Figure 4 and Figure 6In the process shown, the combination of 2D-FFT and DL recognition (DL-based Classification) can effectively improve the accuracy of target recognition. At the same time, the 2D-FFT steps and the DL recognition steps can be flexibly set between various steps in signal processing according to actual needs, such as setting the 2D-FFT step after Range FFT, or after Frame FFT, etc.

[0126] It should be noted that in the above embodiments, all FFT processing can be windowed FFT, and can also be implemented by SVA combined with FFT; alternatively, multi-frame sliding window FFT can replace other time-frequency transform processing, such as short-time Fourier transform, fractional Fourier transform, etc., to achieve effective detection of the target of interest; alternatively, multi-frame sliding window FFT can be replaced by high-pass FIR, Comb FIR, or a filter optimized and synthesized according to an optimization function, so as to achieve the purpose of obtaining similar or even higher performance of multi-frame FFT processing with fewer hardware resources; alternatively, DBF can be replaced by Capon, MUSIC, ESPRINT, etc. and their variant algorithms, and the beam synthesis at the center of the area of ​​interest can be optimized and synthesized and / or the antenna arrangement can be optimized and synthesized to further improve the system target detection performance.

[0127] It should also be noted that Figure 10-12 The embodiment shown is only a scheme for combining multi-frame accumulation technology to perform target detection, but this does not mean that multi-frame accumulation is combined, for example, in Fig.12 In the illustrated embodiment, inter-frame accumulation is omitted, and of course, the details are not described here.

[0128] like Figures 13 to 21 As shown, the embodiment of the present application also provides another method for target detection, which uses inter-frame accumulation combined with deep learning to perform coherent accumulation on the frequency of human breathing, improve the signal-to-noise ratio of human targets and static strong clutter, and effectively construct feature data of empty vehicles (with or without interference), scenes with adults, and scenes with children; at the same time, combined with a deep learning model, deep information mining is performed on the constructed feature data, thereby achieving effective detection of whether there are people in the car and accurate judgment of adults and children, and more accurate detection and identification of targets in spatial areas such as car cabins, indoors, and factories. Compared with point cloud information based on signal processing, the method proposed in this embodiment uses data features with higher dimensions and richer information, which can achieve more accurate detection and identification of targets in the cabin. Specifically:

[0129] like Fig.13 As shown, a method for target detection may include:

[0130] Step S11, for each chirp data of each frame. For example, the downlink ADC data can be DC filtered in the BB. Optionally: the data collected by each RX channel of each chirp can be averaged along the fast time dimension, and the DC component of all sampling points of each RX channel can be subtracted; then, the DC filtered data is windowed, and then 1D-FFT is performed to obtain the distance dimension information; then, the 1D-FFT data is moved from the BB to the CPU SRAM cache through DMA; finally, the same above operations are performed on different chirps.

[0131] Step S12, when the data cached in the CPU SRAM reaches a certain number of frames, such as 128 frames, the 128 frames of data are moved from the CPU SRAM to the BB via DMA, and the following steps are performed:

[0132] First, DC filtering is performed on multiple frames of 1D data: the complex average along the frame dimension is calculated for the 128 frames of data, the average values ​​of different channels and different distances are obtained as their DC components, and the DC components are subtracted from each distance unit of each channel.

[0133] Secondly, the DC-filtered data is windowed along the frame dimension, and then a 2D-FFT calculation is performed along the frame dimension to obtain the range-Doppler complex value x(c, r, v) of each transmitting and receiving channel.

[0134] After that, some Doppler units and range units are intercepted, such as 0-32 and 96-127 Doppler units and 10-42 range units; the energy of the complex value x(c, r, v) of each range unit and each Doppler unit of each channel is calculated, and then the log operation is performed and normalized. The mathematical form is as follows:

[0135]

[0136] Where P(c,r,v) is the normalized “energy” of virtual channel c, range unit r, Doppler unit v, b0 and a0 are preset normalization parameters, and P max and P min is the preset upper and lower limits of P(c, r, v). Generally speaking, the same b0, a0, and P are selected for different c, r, and v. max and P min Values ​​(possible variations can be different c and different v and r partitions to select different b0, a0, P max and P min In one possible embodiment, b0 can be set to all possible log2(|x(c,r,v)| 2 ) mean or median, a0 is the value of all possible log2(|x(c,r,v)| 2) is obtained by dividing the difference between the 95% quantile and the 5% quantile by 256. This normalizes P(c,r,v) to between -127 and 127.

[0137] Then, rearrange P(c,r,v) as Figure 7 The form shown is continuous along the v dimension in physical memory, followed by the r dimension, and finally the c dimension. In the figure, v0 and vNv represent the 0th and Nvth Doppler bins (a total of Nv+1 Doppler bins), r0 and rNr represent the 0th and Nrth range bins (a total of Nr+1 range bins), and c0 represents the 0th virtual channel, with a total of Nc+1 virtual channels. This (Nv+1)×(Nr+1)×(Nc+1) array is the input feature data of the deep learning model:

[0138] Finally, the above data is sent to the constructed deep learning model to determine whether there is someone in the cabin and whether it is an adult or a child, and the result is output.

[0139] It should be noted that the deep learning model can be either CNN or Transformer. An implementation of the model building process is as follows: Fig.14 As shown, the main steps are:

[0140] Step S21, obtain input feature data in the same way as above, and accumulate multiple processing data of multiple groups of experiments to construct an input feature data set Each element P i is an (Nv+1)×(Nr+1)×(Nc+1) array, defined the same as in step 2.c) above.

[0141] Step S22, according to the experimental conditions Marking, that is, P i Generate the corresponding label L i , obtain the labeled input feature dataset

[0142] Step S23, The enhanced data set is obtained by randomly adding white noise, randomly flipping along the Doppler dimension, and randomly translating a small amount along the distance dimension. Note that the number of samples in the augmented dataset is much larger than that in the original dataset. For example, There are 5,000 samples of hollow cars, children, and adults. Ten rounds of data augmentation can be performed to obtain an augmented dataset with 50,000 samples of each type. (You can choose to include or not include the original dataset

[0143] Step S24, Random sorting, taking α = 0.3 ratio As a validation dataset The remaining data is used as the training data set

[0144] Step S25, based on and Train deep learning models.

[0145] Optionally, the main steps of an embodiment are:

[0146] First, an uninitialized CNN model is constructed and the parameters of each layer are randomly initialized.

[0147] Secondly, repeat the following steps 20 times (each time is called an epoch): Randomly divide into N b batches, each batch is N s The following steps are performed for each batch: first, the data of each batch is subjected to one or more data enhancement processing including random addition of white noise, random flipping along the Doppler dimension, and random small translation along the distance dimension; then, the enhanced data is sent to CNN for forward calculation to obtain three types of confidence: no people, adults, and children, and the corresponding loss CE is obtained based on the loss function. t ,; finally, back propagate the CNN network model and update the parameters of each layer of CNN based on the gradient descent optimization algorithm. After the above operations, the updated CNN is used to Forward calculation is performed on all samples of The loss of CE v , if CE v In the setting N es If the value fails to decrease in the next cycle, the loop is terminated early.

[0148] Then, take CE v The CNN model with the smallest cycle is output as the final model.

[0149] It should be noted that, in the model training process, a possible implementation example can use Adam Optimizer as a gradient descent optimizer, select cross entropy as the loss function, set the learning rate to 0.001, and the weight decay to 0.0001. s Set to 128, N es Set to 5

[0150] like Fig.15As shown in the figure, a possible CNN structure is shown below. A CNN with three layers of convolution kernels is shown here, and the type, input and output size of each layer are also shown in the figure. Here, the input feature dimension is 16×64×20, corresponding to 16 virtual channels, 64 Doppler units, and 20 range units. Each convolution operation includes a two-dimensional convolution layer, a batch normalization layer, a ReLu nonlinear activation layer, and a max pooling layer. After three convolution operations, the resulting 128×2 feature vector is flattened to one dimension to obtain a 256×1 one-dimensional feature vector. After random dropout, it passes through a 256×3 fully connected layer to obtain the three types of confidence: empty car, child presence, and adult presence. The confidence can be calculated by softmax to obtain the maximum value as the classification of the input, that is, it is judged as one of the categories of empty car, child presence, and adult presence.

[0151] It should be noted that:

[0152] 1. The method in this embodiment is different from the slow time processing between chirps in the existing radar signal processing. It accumulates multiple frames of data and performs FFT processing between frames in the form of a sliding window.

[0153] 2. The method in this embodiment proposes a process of using inter-frame FFT processing and CNN for detection and classification, without the need for traditional signal processing CFAR and DoA to obtain point clouds for processing.

[0154] 3. The method in this embodiment introduces in detail the innovative CNN input feature construction, among which the noteworthy points and possible changes include:

[0155] ① When calculating P(c,r,v), the log2 operation can be skipped or log 10 or ln operation, etc.

[0156] ② To calculate P(c, r, v), the complex data can be transferred to MCU Mem via DMA and then performed in MCU, or the hardware engine in BB can be used to implement it and then directly transferred to MCU Mem via DMA, or part of it can be implemented in BB and then transferred to MCU Mem via DMA and the remaining calculation can be performed in MCU (e.g., modulo square operation and log2 operation can be implemented in BB, and subtraction, division and saturation truncation can be implemented in MCU)

[0157] ③ P(c, r, v) can be quantized in double-precision floating point or single-precision floating point form without quantization, or int32, int16, int8 or uint32, uint16, uint8.

[0158] ④ The above data arrangement can be continuous along the r dimension, followed by the v dimension and the c dimension (such as Fig.23 as shown); it can also be a vcr or cvr dimension:

[0159] ⑤ In the deep learning model, you can select the virtual channel as the channel of the model, or you can flatten the virtual channel dimension and use a single-channel deep learning model input. For example, the input of the deep learning model is 1×64×320, which contains 64 Doppler units, and the 20 distance units of 16 channels are flattened to 320.

[0160] ⑥ Only some channels can be selected to avoid the influence of channels with large noise or channels with poor RF analog characteristics on the processing effect. For example, 13 channels out of 16 virtual channels with 4 transmit and 4 receive channels are used to construct 13×64×20 CNN input feature data to judge empty vehicles, children and adults.

[0161] ⑦ The data after inter-frame FFT can be truncated to save memory and reduce the amount of calculation, or it can be not truncated to obtain the best performance; if truncated, it can be selected at any time or a combination of after 1D-FFT or after 2D-FFT or when calculating P.

[0162] like Fig.16 As shown, the target detection method in the embodiment of the present application can also be used Fig.16 The processing flow shown is to perform incoherent accumulation on the data after inter-frame FFT by channel to obtain an array and perform operations similar to the above-mentioned steps of intercepting part of the Doppler unit and the distance unit, and then send it as input feature data to the deep learning model for processing to obtain the classification result; the corresponding input channel number of the deep learning model is selected as 1.

[0163] like Fig.17 As shown, the target detection method in the embodiment of the present application can also be used Fig.17 The processing flow shown in the figure is to obtain two-dimensional data by incoherently accumulating the data after inter-frame FFT by channel, and then select only the energy of the points that exceed the set threshold through CFAR, and use the steps similar to intercepting some Doppler units and distance units as input feature data, fill in the distance units and Doppler units that do not exceed the threshold, and then send the input feature data to the deep learning model for processing to obtain the classification result; the corresponding input channel number of the deep learning model is selected as 1.

[0164] It should be noted that the above Fig.16 and 17The two embodiments are combined, that is, the data after inter-frame FFT is CFAR-ed by channel, and for each channel, only the energy of the points that exceed the set threshold is selected, and the operation after the above-mentioned interception of part of the Doppler unit and distance unit is used as the input feature data, and the distance units and Doppler units that do not exceed the threshold are filled in, and then the data of all channels are arranged in a similar manner to the above-mentioned rearrangement of P(c, r, v) and the input feature data are sent to the deep learning model for processing to obtain the classification result; the number of input channels of the deep learning model is selected as the number of virtual channels used.

[0165] The deep learning model in the embodiment of the present application can select other CNN or Transformer, etc., as long as it adapts to its input. Figure 18-19 As shown in the figure, the CNN model modified based on ResNet18 and the expansion of the Sequential module in it. The output here can also be two types: empty car and non-empty car.

[0166] Fig. 20 The two figures in the figure show an example of CNN input feature maps corresponding to adults and children respectively. The left figure shows the input features of adults, and the right figure shows the input features of children. Each sub-image corresponds to the data of each channel. The horizontal axis is the Doppler unit and the vertical axis is the distance unit. In this example, 20 distance units and 64 Doppler units are intercepted.

[0167] Fig.21 The figure shows the changes of model loss and model classification accuracy during training with the epoch of training. It can be seen that the model converges in about 7 epochs and can reach 98% accuracy.

[0168] Fig. 22 The confidence of the test on adult and child samples is shown. The horizontal axis represents the sample number, and the vertical axis represents the confidence (i.e., score). In the experiment, the first 150 samples are samples of adults in the car, and the next 150 samples are samples of children in the car. For each sample, a possible deep learning model calculates its confidence corresponding to adults and children respectively, that is, it generates two blue and orange points for each sample; and when the confidence of the adult is greater than the confidence of the child, it is determined that the person in the car is an adult, and when the confidence of the child is greater than the confidence of the adult, it is determined that the person in the car is a child. As can be seen from the figure, the trained CNN model can correctly determine whether the person in the car is an adult or a child.

[0169] The step division of the above methods is only for the purpose of clear description. When implemented, they can be combined into one step or some steps can be split and decomposed into multiple steps. As long as they include the same logical relationship, they are all within the scope of protection of this patent; adding insignificant modifications to the algorithm or process or introducing insignificant designs without changing the core design of the algorithm and process are all within the scope of protection of this patent.

[0170] The embodiment of the present application also provides an integrated circuit, which may include a radio frequency module, an analog signal processing module and a digital signal processing module connected in sequence; the radio frequency module is used to generate a radio frequency transmission signal and receive an echo signal; the analog signal processing module is used to down-convert the echo signal to obtain an intermediate frequency signal; the digital processing module is used to perform analog-to-digital conversion on the intermediate frequency signal to obtain a digital signal; and based on the target detection method in the embodiment of the present application, the digital signal is processed to achieve the purpose of target detection, for example, the integrated circuit may be a millimeter wave radar chip (chip or die). Among them, the above-mentioned digital processing module may include a baseband module and a main control module, etc., and each module may be configured to execute each corresponding step in the target detection method in the above-mentioned embodiment.

[0171] In some optional embodiments, the integrated circuit may be an AiP (Antenna-In-Package) chip structure, an AoP (Antenna-On-Package) chip structure, or an AoC (Antenna-On-Chip) chip structure.

[0172] According to other embodiments of the present application, an electromagnetic wave sensor is also proposed. The electromagnetic wave sensor may include an antenna, and an integrated circuit as described above. The integrated circuit is electrically connected to the antenna for receiving and transmitting electromagnetic wave signals. For example, the electromagnetic wave sensor may include: a carrier, an integrated circuit and an antenna as described in any of the above embodiments, and the integrated circuit may be arranged on the carrier; the antenna may be arranged on the carrier, or the integrated circuit may be integrated with the integrated circuit as an integrated device arranged on the carrier (that is, the antenna may be an antenna provided in an AiP, AoP or AoC structure at this time); the integrated circuit is connected to the antenna (that is, the sensor chip or integrated circuit is not integrated with an antenna at this time, such as a conventional SoC, etc.), for receiving and transmitting electromagnetic wave signals. The carrier may be a printed circuit board PCB, and the corresponding transmission line may be a PCB trace.

[0173] It should be noted that the electromagnetic waves in the present application may include radio waves and light waves. Radio waves include short waves, medium waves, long waves and microwaves, etc. Microwaves include centimeter waves (i.e. electromagnetic waves in the range of 3GHz to 30GHz, such as electromagnetic waves in the range of 3.1GHz to 10.6GHz, electromagnetic waves in the frequency band of 24GHz, etc.) and millimeter waves (i.e. electromagnetic waves in the range of 30GHz to 300GHz, such as electromagnetic waves in the frequency band of 60GHz, electromagnetic waves in the frequency band of 77GHz (such as 77GHz to 81GHz, etc.)), and light waves may include ultraviolet rays, visible light, infrared rays, and lasers, etc., among which the electromagnetic wave frequency band of lasers is (3.846 to 7.895)*10^5GHz, i.e., lasers are included in part of the frequency bands of ultraviolet rays and visible light.

[0174] An embodiment of the present application provides a device, which may include: a device body; and an electromagnetic wave sensor as described above, which is arranged on the device body; wherein the electromagnetic wave sensor is used for target detection and / or communication to provide reference information for the operation of the device body.

[0175] The embodiment of the present application also provides an electronic device, which can be expressed in the form of a general-purpose computing device. The components of the electronic device may include, but are not limited to: at least one processing unit, at least one storage unit, a bus connecting different system components (including a storage unit and a processing unit), a display unit, etc. Among them, the storage unit stores a program code, and the program code can be executed by the processing unit so that the processing unit executes the method described in this specification according to various exemplary embodiments of the present application. The storage unit may include a readable medium in the form of a volatile storage unit, such as a random access memory unit (RAM) and / or a cache memory unit, and may further include a read-only memory unit (ROM).

[0176] The storage unit may also include a program / utility having a set (at least one) of program modules, such program modules including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.

[0177] The bus may represent one or more of several types of bus structures, including a memory unit bus or memory unit controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus architectures.

[0178] The electronic device may also communicate with one or more external devices (e.g., keyboards, pointing devices, Bluetooth devices, etc.), may communicate with one or more devices that enable a user to interact with the electronic device, and / or may communicate with any device that enables the electronic device to communicate with one or more other computing devices (e.g., routers, modems, etc.). This communication may be performed through an input / output (I / O) interface. In addition, the electronic device may also communicate with one or more networks (e.g., local area networks (LANs), wide area networks (WANs), and / or public networks, such as the Internet) through a network adapter. The network adapter may communicate with other modules of the electronic device through a bus. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the electronic device, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems, etc.

[0179] For example, the electronic device in the embodiment of the present application may also include: a device body; and an electromagnetic wave sensor as described in any of the above embodiments arranged on the device body; wherein the electromagnetic wave sensor can be used to realize functions such as target detection and / or wireless communication.

[0180] Specifically, based on the above embodiments, in an optional embodiment of the present application, the electromagnetic wave sensor can be arranged outside the device body, or arranged inside the device body, and in other optional embodiments of the present application, the electromagnetic wave sensor can also be arranged partially inside the device body and partially outside the device body. The embodiments of the present application are not limited to this, and the specific situation can be determined.

[0181] In an optional embodiment, the device body can be a component or product used in fields such as smart cities, smart homes, transportation, smart homes, consumer electronics, security monitoring, industrial automation, in-cabin detection (such as smart cockpits), medical devices, and health care. For example, the device body can be intelligent transportation equipment (such as cars, bicycles, motorcycles, ships, subways, trains, etc.), security equipment (such as cameras), liquid level / flow rate detection equipment, smart wearable devices (such as bracelets, glasses, etc.), smart home devices (such as sweeping robots, door locks, televisions, air conditioners, smart lights, etc.), various communication devices (such as mobile phones, tablet computers, etc.), and gates, smart traffic lights, smart signs, traffic cameras, and various industrialized robotic arms (or robots), etc. It can also be various instruments for detecting life characteristic parameters and various devices equipped with the instruments, such as life characteristic detection in car cabins, indoor personnel monitoring, smart medical equipment, consumer electronic equipment, etc.

[0182] The embodiment of the present application also provides a non-transitory computer-readable storage medium on which computer-readable instructions are stored. When the instructions are executed by a processor, the processor executes the feeder unequal length compensation method as described above.

[0183] Through the description of the above implementation methods, it is easy for those skilled in the art to understand that the example implementation methods described here can be implemented by software or by combining software with necessary hardware. The technical solution according to the implementation method of the present application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, including several instructions to enable a computing device (which can be a personal computer, a server, or a network device, etc.) to execute the above method according to the implementation method of the present application.

[0184] The software product may use any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, a system, device or device of electricity, magnetism, light, electromagnetic, infrared, or semiconductor, or any combination thereof. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof.

[0185] Computer readable storage media may include data signals propagated in baseband or as part of a carrier wave, wherein readable program codes are carried. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. The readable storage medium may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, device, or device. The program codes contained on the readable storage medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination thereof.

[0186] Program code for performing the operations of the present application may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In the case of a remote computing device, the remote computing device may be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0187] The computer-readable medium carries one or more programs. When the one or more programs are executed by a device, the computer-readable medium implements the aforementioned functions.

[0188] Those skilled in the art will appreciate that the above modules can be distributed in the device according to the description of the embodiment, or can be changed accordingly and only used in one or more devices different from the embodiment. The modules of the above embodiments can be combined into one module, or further divided into multiple sub-modules.

[0189] According to an embodiment of the present application, a computer program is proposed, including a computer program or an instruction, which, when executed by a processor, can execute the method described above. In an optional embodiment, the above-mentioned integrated circuit can be a millimeter wave radar chip. The type of digital function module in the integrated circuit can be determined according to actual needs. For example, in a millimeter wave radar chip, the data processing module can be used for functions such as distance-dimensional Doppler transform, velocity-dimensional Doppler transform, constant false alarm detection, wave direction detection, point cloud processing, etc., to obtain information such as the distance, angle, speed, height, micro-Doppler motion characteristics, shape, size, surface roughness and dielectric properties of the target.

[0190] It should be noted that radio devices can achieve functions such as target detection and / or communication by transmitting and receiving radio signals to provide the device body with detection target information and / or communication information, thereby assisting or even controlling the operation of the device body.

[0191] For example, when the above-mentioned device body is applied to an advanced driver assistance system (i.e., ADAS), the radio device (such as millimeter-wave radar) as a vehicle-mounted sensor can assist the ADAS system to realize application scenarios such as adaptive cruise control, automatic braking assistance (i.e., AEB), blind spot detection warning (i.e., BSD), assisted lane change warning (i.e., LCA), reversing assistance warning (i.e., RCTA), parking assistance, rear vehicle warning, collision avoidance, pedestrian detection, and in-cabin liveness detection (i.e., CPD).

[0192] The technical features of the above-described embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above-described embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0193] The above-described embodiments only express the preferred embodiments of the present invention and the technical principles used. The description is relatively specific and detailed, but it cannot be understood as limiting the scope of the invention patent. It is possible for those skilled in the art to make various obvious changes, readjustments and substitutions without departing from the protection scope of the present invention. Therefore, although the present invention is described in more detail through the above embodiments, the present invention is not limited to the above embodiments, and may also include more other equivalent embodiments without departing from the concept of the present invention, and the protection scope of the patent of the present invention is determined by the scope of the attached claims.

Claims

1. A target detection method, characterized in that: include: Performing FFT processing on the echo signal to generate a feature vector of the echo signal, wherein the feature vector includes energy characteristics characterizing the echo signal at different distance units; The feature vector is processed based on a deep learning model to achieve target detection in the region of interest.

2. The target detection method according to claim 1, characterized in that: The feature vector includes energy features characterizing the echo signal in different Doppler dimensions at different distance units.

3. The target detection method according to claim 2, characterized in that: The performing FFT processing on the echo signal may include: Perform FFT processing on the echo signal to generate a range-Doppler spectrum; A beam synthesis process is performed according to the range-Doppler spectrum to obtain a feature vector of the echo signal.

4. The target detection method according to claim 3, characterized in that: The method of performing FFT processing on the echo signal to generate a range-Doppler spectrum includes: Performing distance-dimensional FFT processing on the echo signal; Performing sliding windowing on the result obtained by the range dimension FFT processing, and performing 2D FFT on the windowed data to obtain the range-Doppler spectrum; The length of the sliding window corresponds to a duration of the same order of magnitude as the period of the periodic motion of the target.

5. The target detection method according to claim 3, characterized in that: The beam synthesis processing according to the range-Doppler spectrum is implemented by the following expression: Wherein, P(r,v,b) is the energy characteristic of the vth Doppler unit of the rth range unit based on beam b, sv(c,b) is the steering vector of beam b on virtual channel c, x(c,r,v) is the data corresponding to the vth Doppler unit of the rth range unit on virtual channel c in the range-Doppler spectrum, N c is the total number of beams.

6. The target detection method according to claim 2, characterized in that: The performing FFT processing and beamforming processing on the echo signal comprises: Perform range FFT processing on the echo signal; Perform beam synthesis processing on the result obtained by range FFT processing; Sliding windowing is performed on the result of the beamforming process, and 2D FFT is performed on the windowed data.

7. The target detection method according to claim 1, characterized in that: The performing FFT processing and beamforming processing on the echo signal comprises: Perform range FFT processing on the echo signal; The beamforming process is performed on the result obtained by the range FFT process.

8. The target detection method according to any one of claims 3 to 7, characterized in that: The beamforming comprises: Determining a target direction pointing to the region of interest and generating a steering vector according to the target direction; Beam synthesis is performed according to the steering vector.

9. The target detection method according to claim 8, characterized in that: The determining a target direction pointing to the region of interest includes: The target direction is determined according to the azimuth and elevation angle of the region of interest relative to the radar.

10. The target detection method according to claim 8, characterized in that: The determining a target direction pointing to the region of interest includes: Acquire measured data of the target in the region of interest; Clustering is performed according to the target measured data, so as to use the azimuth and elevation angles of the cluster center points in each of the regions of interest as the target direction.

11. The target detection method according to claim 8, characterized in that: The generating of the steering vector according to the target direction is realized by the following expression: sv(c,b)=exp(1j·2π / λ·(x c cosφ b sinθ b +z c sinφ b )), Where sv(c,b) is the steering vector of beam b on virtual channel c, λ is the central wavelength of the radar detection signal, and x c and z c are the horizontal and vertical positions of the antenna corresponding to the virtual channel c, θ b is the azimuth corresponding to the target direction, φ b is the pitch angle corresponding to the target direction.

12. The target detection method according to any one of claims 1 to 7, characterized in that: The step of generating a feature vector of the echo signal comprises: According to a preset rule, the energy features obtained by performing FFT processing and beamforming processing on the echo signal are rearranged to obtain the feature vector.

13. The target detection method according to claim 12, characterized in that: The preset rules include: The data with the same first dimension are arranged continuously in the changing direction of the second dimension, and the whole data are arranged continuously in the changing direction of the first dimension; or, The data with the same third dimension and the same fourth dimension are arranged continuously in the changing direction of the fifth dimension, the data with the same third dimension are arranged continuously in the changing direction of the fourth dimension, and the whole is arranged continuously in the changing direction of the third dimension; The first dimension and the second dimension are different dimensions in the beam dimension and the distance dimension, respectively; the third dimension, the fourth dimension and the fifth dimension are different dimensions in the beam dimension, the distance dimension and the Doppler dimension, respectively.

14. The target detection method according to any one of claims 1 to 7, characterized in that: The step of generating a feature vector of the echo signal comprises: The energy characteristics obtained by performing FFT processing and beamforming processing on the echo signal are logarithmically normalized.

15. The target detection method according to claim 14, characterized in that: The logarithmic normalization of energy features obtained by performing FFT processing and beamforming processing on the echo signal includes: The energy characteristics obtained by performing FFT processing and beamforming processing on the echo signal are logarithmized, and the logarithm result is linearly normalized using preset upper and lower bound constraints.

16. The target detection method according to claim 15, characterized in that: The energy characteristics obtained by performing FFT processing and beamforming processing on the echo signal are logarithmized, and the result after logarithmization is linearly normalized using preset upper and lower bound constraints, which is achieved by the following expression: Among them, P norm (r, v, b) is the logarithmic normalization result, P(r, v, b) is the energy characteristic of beam b at range unit r and Doppler unit v, a0, b0, P max and P min These are preset parameters.

17. The target detection method according to any one of claims 1 to 7, characterized in that: Before processing the feature vector based on the deep learning model, the method further includes: Obtaining a training feature vector, wherein the training feature vector carries a one-hot encoding label and is used to characterize energy characteristics of a signal at different distance units; The training feature vector is processed by a preset data enhancement method to obtain a training set, wherein the preset data enhancement method includes at least one of the following: randomly adding white noise, flipping the data along the Doppler dimension, and translating the data along the distance dimension; The deep learning model is trained according to the training set.

18. The target detection method according to any one of claims 1 to 7, characterized in that: The target detection result is a confidence matrix, and the elements in the confidence matrix correspond to at least one of the following information: the confidence of detecting the target, the confidence that the target exists in the corresponding region of interest, the confidence that an adult target exists in the corresponding region of interest, and the confidence that a child target exists in the corresponding region of interest.

19. The target detection method according to any one of claims 1 to 7, characterized in that: The feature vector includes energy features characterizing the echo signal at different distance units within a preset distance range and / or a preset Doppler range.

20. An integrated circuit, characterized in that: It includes a radio frequency module, an analog signal processing module and a digital signal processing module connected in sequence; Wherein, the RF module is used to generate RF transmission signals and receive RF reception signals; The analog signal processing module is used to down-convert the RF received signal to obtain an intermediate frequency signal; The digital signal processing module is used to perform analog-to-digital conversion on the intermediate frequency signal, and to process the digital data obtained by the analog-to-digital conversion using the target detection method described in any one of claims 1 to 19 to achieve target detection.

21. An electromagnetic wave sensor, characterized in that: include: Carrier; The integrated circuit of claim 20, disposed on a carrier; An antenna is arranged on the carrier, or the antenna and the integrated circuit are integrated into one device and arranged on the carrier; The integrated circuit is connected to the antenna and is used to transmit a radio frequency transmission signal and / or receive a radio frequency reception signal.

22. A terminal device, characterized in that: include: Equipment body; and an electromagnetic wave sensor as claimed in claim 21 disposed on the device body; The electromagnetic wave sensor is used for target detection and / or communication to provide reference information for the operation of the device body.

Images