A deep learning fine-grained material identification method based on millimeter wave radar

Abstract

The application discloses a kind of deep learning fine-grained material identification methods based on millimeter wave radar, comprising the following steps: 1, construct fine-grained material identification system RF-MID based on deep learning network model, including data reorganization module, signal pre-processing module, material feature extraction module and classifier module;2, data reorganization module is received by millimeter wave radar and is reorganized into three-dimensional data block reflection signal;3, signal pre-processing module uses variational autoencoder model to the three-dimensional data block of input pre-processing denoising;4, material feature extraction module utilizes three-dimensional complex value inflation convolution network to extract high-dimensional feature information from the three-dimensional data block after pre-processing, generates the feature map of the difference of characteristic of material;5, the feature map of extraction is compressed by classifier module, obtains one-dimensional material category feature for output material category score, to identify fine-grained material, can accurately and efficiently identify the fine-grained material of multiple different shape materials.

Description

Technical Field

[0001] This invention relates to material identification technology, specifically a deep learning-based fine-grained material identification method based on millimeter-wave radar. Background Technology

[0002] With the development of wireless sensing technology, we can achieve non-contact physical world perception through intelligent sensing devices, such as positioning, action recognition (including activity monitoring and gesture recognition), identity authentication, and human health monitoring in daily life scenarios, as well as the identification of object materials. Whether it is autonomous driving technology, robotic systems, home smart devices, or security inspection, there is a need for non-invasive, fine-grained material perception in their working scenarios. Material perception technology can be applied to autonomous driving to improve the accuracy of target detection, enhance discrimination capabilities, and improve safety. It can also be applied to intelligent sorting and waste classification to improve efficiency. Furthermore, it can be applied to the quality inspection of liquids and food, as well as the health monitoring of cultural relics and building structures. Moreover, in the virtual reality world, accurate and detailed material mapping can greatly enhance the realism of interaction. All of this is sufficient to prove that accurate material recognition is crucial to daily life.

[0003] Traditional material identification methods employ techniques such as spectral analysis, X-rays, ultrasound, and CT / MRI. These methods not only rely on expensive proprietary instruments for analysis, but the bulky specialized equipment and invasive detection methods limit their use to laboratory environments, making them unsuitable for ubiquitous everyday applications. Therefore, based on the need for non-invasive features, invisible material identification, and material identification in more common environments, target material sensing methods based on wireless signals and material identification methods based on image processing have emerged.

[0004] Material recognition methods based on image processing analyze the shape, color, texture, and other features of an object's image to determine its material. However, this vision-based approach has certain limitations, such as sensitivity to lighting conditions and the risk of privacy leaks. Moreover, inferring material from images does not truly perceive the object's material information, making it easy to be deceived by appearance, especially when the material's surface is painted, covered, or obscured. In such cases, image recognition cannot accurately determine the target object's material. Therefore, we need to consider new material-related physical properties for material recognition.

[0005] Wireless signal-based target material sensing methods utilize wireless signals to perceive the material properties of target objects, enabling flexible, non-contact, and low-cost material identification applicable to everyday scenarios. These methods primarily include those based on ultra-wideband (UWB), Wi-Fi, radio frequency identification (RFID), and millimeter-wave radar. However, some of these methods still require direct contact with the target, such as attaching RFID tags to the object's surface, thus failing to meet the need for non-contact sensing. Therefore, some researchers have analyzed the changes in the dielectric constant of materials after wireless signals penetrate the target to distinguish between different materials. While this achieves the non-invasive requirement, this method is mostly applied to liquids and requires placing multiple signal transceivers on both sides of the container, limiting its deployment scenarios and application range.

[0006] Currently, based on signal utilization, target material sensing methods based on wireless signals can be divided into target material sensing methods based on penetrating signals and target material sensing methods based on reflected signals. Penetrating signal-based methods require placing multiple devices on both sides of the target material to analyze the attenuation of the penetrating signal. Since different liquids have different dielectric constants, the wave velocity and attenuation of the signal as it passes through the liquid also vary. For example, LiquID uses UWB (Ultra-Wideband) signals to penetrate a fixed container, analyzes the signal amplitude and phase to obtain the complex dielectric constant of the liquid, and achieves high-precision liquid identification. Vi-Liquid uses mechanical waves to establish a liquid viscosity characteristic model, also achieving accurate identification of various liquids. WiMi uses Wi-Fi signals penetrating the target to distinguish liquids, utilizing dual-antenna devices to eliminate the limitations of container width. The impact of liquid level; TagScan places the target between an RFID (Radio Frequency Identification) antenna and a tag, using the RSS value and phase of the reflected RFID signal penetrating the target for liquid classification; LiqRay and PackquID use multi-frequency RF signals, which not only solves the problem of liquid level height but also enables the identification of unknown liquids under arbitrary container and other obstruction conditions; Wi-Fruit measures the CSI change of the Wi-Fi signal penetrating the fruit to estimate the fruit's moisture content and SSC value (soluble solids content) biological characteristics; Although the above target material sensing methods based on penetrating signals are gradually improving the external constraints of the target, such as loading containers and obstructions, and reducing deployment complexity, they still require the placement of signal transmitting and receiving devices on both sides of the target material, resulting in the sensing location still being limited to a fixed area.

[0007] Therefore, more flexible and portable target material sensing methods based on reflected signals have begun to be extensively studied. TagTag, RFIQ, and RF-EATS all involve attaching RFID tags to the target surface. Based on the near-field coupling principle, the impedance of the RFID tag changes differently depending on the material, thus identifying liquid materials for food quality inspection and authenticity verification. IntuWition uses reflected Wi-Fi signals to achieve coarse-grained identification of five materials (copper, aluminum, plywood, birch, and humans), but it is only suitable for large targets. Recent research has begun to explore the application of lightweight millimeter-wave radar in sensing tasks. For example, RSA uses 60GHz millimeter-wave radar for imaging and coarse-grained material classification, but it is not suitable for fine-grained material identification. RadarCat utilizes Google... Soli was used to identify everyday objects by contact with targets. OLS, under the premise of fixed target shape, coarsely identified four different materials (carpet, tile, liquid, and laminate), but the types of materials that could be identified were very limited. Later, many scholars began to focus on the advantages of millimeter waves, such as short wavelength, high bandwidth, high precision and high resolution, as well as portability and low cost, and established material identification methods based on millimeter radar. For example, mSense used a 60GHz millimeter-wave radar antenna array to achieve high-precision identification of five flat surface materials (wood, plastic, ceramic, water, and aluminum) at a distance of 20cm in a moving scene, and could distinguish 21 kinds of everyday objects. It solved the problems of portability and non-invasiveness. However, it required a large antenna array, and because the reflected signal depends not only on the material of the object. The reflection coefficient also depends on the size and geometry of the target itself, the directionality of the incident wave, and the different multipath environments around it, resulting in a limited range of target materials that can be classified and a single scene. FG-LiquID uses multi-antenna data calibration from millimeter-wave radar to train neural networks, eliminating the influence of small displacements and identifying 30 different liquids with fine granularity. However, it only targets liquid materials and does not need to consider the influence of the shape of the target object. RFVibe adds a speaker to make the object vibrate, thereby affecting the radar echo signal. Based on the different vibration frequencies of different materials, it achieves stable material identification in different scenes. However, the added acoustic device offsets the portability advantage of millimeter-wave radar, and it focuses on the identification capability in different environments without fully considering fine-grained differentiation and the influence of shape.

[0008] While existing material identification methods have achieved good results, they rely excessively on frequency domain transformation and expert-defined features, fail to fully consider the relationship between the target surface shape and the reflected signal, and require complex feature engineering methods to obtain usable features for material differentiation. Therefore, there is an urgent need for a method that can identify fine-grained materials without manual feature engineering and can extract the most relevant features from high-dimensional complex data. Summary of the Invention

[0009] To address the shortcomings of existing technologies, the purpose of this invention is to provide a deep learning-based fine-grained material identification method based on millimeter-wave radar, which can accurately and efficiently identify fine-grained materials of various shapes.

[0010] To achieve the above objectives, the present invention employs the following technical solution:

[0011] A deep learning-based fine-grained material recognition method based on millimeter-wave radar includes the following steps:

[0012] Step 1: Construct a fine-grained material identification system RF-MID based on a deep learning network model, including a data reconstruction module, a signal preprocessing module, a material feature extraction module, and a classifier module;

[0013] Step 2: The data reconstruction module reconstructs the reflected signal received by the millimeter-wave radar into a three-dimensional data block containing time-domain feature variation information, multi-angle information of different antennas, and frequency distribution feature information of multiple samples.

[0014] Step 3: The signal preprocessing module uses a variational autoencoder model to preprocess the input three-dimensional data block, thereby removing noise interference caused by the material shape.

[0015] Step 4: The material feature extraction module uses a three-dimensional complex-valued dilated convolutional network to extract high-dimensional feature information from the preprocessed three-dimensional data blocks, thereby generating feature maps that characterize material differences.

[0016] The three-dimensional complex-valued dilated convolutional network includes dilated convolution, four three-dimensional complex-valued convolutional layers, and two residual blocks. The three-dimensional complex-valued convolutional layers are three-dimensional complex-valued convolutional layers used to capture subtle changes and diverse features of signals. They can preserve local features under short-time sampling, extract global information from multiple data segments, and introduce multi-angle features from multiple antennas. The dilated convolution introduces at least one dilation rate inside the convolutional kernel to expand the receptive field of the kernel. The residual blocks include two complex-valued dilated convolutional layers for extracting depth features and a max-pooling layer for dimensionality reduction.

[0017] Step 5: The classifier module compresses the extracted feature map to obtain a one-dimensional material category feature for outputting material category scores, thereby identifying fine-grained materials.

[0018] Furthermore, the process of reassembling the three-dimensional data block is as follows:

[0019] The number of antennas, sampling points, chirps, and frames in the millimeter-wave radar parameter settings are obtained. The reflected signal data received by the millimeter-wave radar is reassembled into a three-dimensional data block in the form of frames. The first dimension of the three-dimensional data block is the multi-angle information of different antennas, the second dimension is the sampling time-domain information within a chirp, and the third dimension arranges all chirps in each frame side by side, thereby expanding the multi-sample information of static objects or recording the change trend information of dynamic objects over a long period of time.

[0020] Furthermore, the variational autoencoder model preprocesses the input 3D data block as follows:

[0021] Step 3.1: Input simple shapes with different curvatures into the variational autoencoder model. Use the reflection signal of the flat object as a label. The variational autoencoder model learns and smooths out the difference between the reflection signal of the simple shape and the reflection signal of the flat object to obtain the trained variational autoencoder model.

[0022] Step 3.2: The trained variational autoencoder model normalizes the input arbitrary reflection signal to remove noise interference caused by the material shape.

[0023] Furthermore, the dilation rate of the dilated convolution increases exponentially, doubling with each complex-valued convolutional layer, thus exponentially expanding the receptive field.

[0024] Furthermore, the classifier module includes two complex-valued fully connected layers and four real-valued fully connected layers. The complex-valued fully connected layers and the four real-valued fully connected layers unify the dimensions through modulo operation, thereby achieving mapping to the number of categories for output.

[0025] Compared with the prior art, the present invention has the following technical effects:

[0026] 1) This invention first reconstructs the reflected signal received by the millimeter-wave radar to obtain a three-dimensional data block containing time, space, and frequency features. Then, a variational autoencoder model is used to remove noise interference caused by the material shape in the three-dimensional data block. Then, a designed three-dimensional complex-valued dilated convolutional network is used to extract high-dimensional feature information from the denoised three-dimensional data block, thereby generating a feature map representing material differences. This achieves fine-grained material identification of different shapes. Compared with traditional material identification methods, since RF-MID can directly extract high-dimensional features from the original data, there is no need for manual feature selection, which reduces human error, reduces the complexity of data processing, and improves the efficiency and accuracy of feature extraction.

[0027] 2) The average accuracy of material identification for six common coarse-grained materials, namely metal, cardboard, plastic, ceramic, wood and foam, is 96.4%, and the average accuracy of identification for 24 fine-grained materials of 52 different shapes is 82.6%. It can be seen that the present invention has high material identification accuracy and the identification accuracy is less affected by the shape of the material, and is suitable for identifying fine-grained materials of various shapes.

[0028] 3) In the practical application scenario of classifying cultural relic ceramic fragments, the accuracy rate of the present invention is as high as 78.5%, indicating that it can distinguish very similar materials. Moreover, the material identification performance is robust for materials with shape differences and surface coverings. It can be seen that the identification method of the present invention has potential application value in intelligent sorting tasks. Attached Figure Description

[0029] Figure 1 : A schematic diagram of the structure of the fine-grained material identification system RF-MID of the present invention;

[0030] Figure 2 : A schematic diagram of the structure of the deep learning network model of this invention;

[0031] Figure 3 : A schematic diagram of the structure of the three-dimensional complex-valued convolutional network CDCN of the present invention;

[0032] Figure 4 The present invention includes photographs of 24 different types of materials. Detailed Implementation

[0033] The specific content of the present invention will be further explained in detail below with reference to the embodiments.

[0034] A deep learning-based fine-grained material recognition method based on millimeter-wave radar includes the following steps:

[0035] Step 1: Construct a fine-grained material identification system RF-MID based on a deep learning network model, including a data reconstruction module, a signal preprocessing module, a material feature extraction module, and a classifier module;

[0036] By recombining RF signals and deep learning network models, a system is constructed as follows: Figure 1 The fine-grained material recognition system RF-MID based on a deep learning network model is shown below. The structure of the deep learning network model is as follows: Figure 2 As shown, it is used to extract the differential features of the reconstructed RF signal to achieve accurate material identification, thereby classifying the material;

[0037] Step 2: The data reconstruction module reconstructs the reflected signal received by the millimeter-wave radar into a three-dimensional data block containing time-domain feature variation information, multi-angle information of different antennas, and frequency distribution feature information of multiple samples.

[0038] Considering that radio frequency (RF) data not only contains complex numerical data of amplitude and phase information, but also multiple dimensions of information such as time, space, and frequency, RF data has richer high-dimensional features compared to visual images and model data. Therefore, the original data is reorganized into a three-dimensional data block (i.e., a radar data cube), and a design is made as follows: Figure 2 The deep learning network model shown is suitable for three-dimensional data blocks. Compared with the manual feature extraction methods of traditional signal processing methods, it can directly extract deep features related to materials from the reconstructed original signal, reducing the complexity of data processing and improving the efficiency of feature extraction.

[0039] The process of reassembling the three-dimensional data block is as follows: The number of antennas, sampling points, chirps, and frames in the millimeter-wave radar parameter settings are obtained. The reflected signal data received by the millimeter-wave radar is then assembled into a three-dimensional data block in a frame-by-frame manner. The first dimension contains multi-angle information from different antennas. The second dimension is the sampling time-domain information within a chirp, which is the most important feature component. The third dimension arranges all chirps in each frame side-by-side. For static objects, this expands the multi-sample information; for dynamic objects, it records the changing trend information over a long period. For any reflected signal data received by radar, as long as the above four parameter settings are obtained, the data can be reassembled into a three-dimensional data block containing time, space, and frequency feature information, providing high-dimensional feature information for the input of deep learning network models.

[0040] Step 3: The signal preprocessing module uses a variational autoencoder model to preprocess the input 3D data block, thereby removing noise interference caused by the material shape.

[0041] Although the method of reconstructing the original data and extracting material features can obtain fine-grained material differences, since the reflected signal is related not only to the material of the target material but also to the shape of the target material, the performance of the deep learning network model in identifying the material decreases when the shape of the target material changes. Therefore, it is necessary to eliminate the interference of the shape features of the target material on the material judgment.

[0042] The fundamental principle of material identification based on radar reflection signals is determined by radar equations. Under the same radar settings, the intensity of the reflected signal is related to the radar cross-section (RCS) of the target object. The radar RCS depends on the shape and directionality of the target object, its reflectivity, and the size of its geometric cross-section. In other words, the target's shape and reflection characteristics are related, and changes in directionality are considered changes in the observed target's shape. However, current technologies only use the relationship between amplitude and reflectivity for material identification, ignoring the influence of the target's shape. This is only suitable for identifying materials with significantly different reflectivity. When the reflectivity of two materials differs only slightly, the influence of shape becomes particularly important. Therefore, deep learning network models are more robust to material identification across different shapes, requiring a comprehensive consideration of RCS characteristics, rather than... This embodiment focuses solely on the simple relationship between amplitude and reflectivity. It references the comparative measurement method commonly used in engineering to measure RCS. That is, given the RCS of a standard object, the RCS of the target object is calculated by measuring the reflection signals of the target object and the standard object and then using the ratio. Using a flat-shaped material as a calibration, other simple-shaped objects with different curvatures are input into the variational autoencoder model (VAE). The VAE learns the difference between the reflection signals of other objects and the reflection signals of the standard flat object and smooths it out to obtain a trained variational autoencoder model. The trained variational autoencoder model treats the reflection signals caused by any object shape as noise and performs normalization processing, eliminating the noise caused by the material shape factor in the original signal, thereby reducing the noise interference on the material judgment.

[0043] Step 4: The material feature extraction module uses a three-dimensional complex-valued dilated convolutional network to extract high-dimensional feature information from the preprocessed three-dimensional data blocks, thereby generating feature maps that characterize material differences.

[0044] Considering that radio frequency (RF) signals are complex data containing both amplitude and phase information, possessing richer multidimensional information compared to two-dimensional images, traditional methods only focus on amplitude differences in the frequency distribution, ignoring phase information. Therefore, traditional image feature extraction networks (such as CNNs) are unsuitable for extracting depth features from RF signal data. To effectively process multidimensional raw signal data and obtain distinguishable material features, this embodiment proposes... Figure 3 The illustrated 3D complex-valued dilated convolutional network includes 3D complex-valued convolutional layers, dilated convolutions, and residual blocks, wherein:

[0045] There are four three-dimensional complex-valued convolutional layers. These are complex-valued convolutional layers with a three-dimensional structure. Since the complex-valued data integrates the amplitude representing the reflection intensity of the signal and the phase information that provides key details (such as the shape, edge, and orientation of the object), the amplitude represents the reflection intensity of the signal, and the phase can provide key details. Therefore, the use of complex-valued convolutional layers makes it easier to capture subtle changes and diverse features of the signal. Moreover, the three-dimensional structure helps the complex-valued convolutional layers maintain local features under short-time sampling and extract global information from multiple data segments. It also introduces multi-angle features from multiple antennas. Short-time frequency differences correspond to the material properties of different materials. The global information from multiple data segments allows the features to focus on the target reflection characteristics while ignoring external interference, which improves the overall performance and robustness of the deep learning network model and can tolerate small displacement deviations and shape differences.

[0046] The dilated convolution introduces at least one interval (i.e., dilation rate) inside the convolution kernel to expand the receptive field of the convolution kernel. When the dilation rate is 1, the dilated convolution degenerates into a standard convolution. When the dilation rate is greater than 1, the interval inside the convolution kernel allows it to cover a wider input region, thereby increasing the receptive field. Without increasing parameters and computational cost, it captures a wider range of contextual information and effectively handles long-distance dependencies, making it suitable for processing raw time-domain signals. In this embodiment, the dilation rate is set to grow exponentially. The dilation rate doubles with each complex convolutional layer, and the receptive field expands exponentially. This not only better preserves global features but also avoids excessively deep network layers, thus improving the network's operating efficiency.

[0047] There are two residual blocks, each containing two complex-valued dilated convolutional layers and a max-pooling layer, used for deep feature extraction and dimensionality reduction, respectively. The complex-valued dilated convolutional layers use the ReLU activation function, as well as weight normalization and Dropout techniques. The residual blocks alleviate the gradient vanishing problem in deep networks, accelerate training convergence, and reduce parameters and computational cost. The layer structure of the residual blocks can be adjusted according to the input data with different sampling rates.

[0048] Step 5: The classifier module compresses the extracted feature map to obtain a one-dimensional material category feature for outputting the material category score, thereby identifying fine-grained materials;

[0049] The classifier module includes two complex-valued fully connected layers and four real-valued fully connected layers. The complex-valued fully connected layers and the four real-valued fully connected layers unify the dimensions through modulo operation, thereby achieving mapping to the number of categories for output.

[0050] This embodiment trains a deep learning network model from the input of 3D data blocks to the output of the final material category, enabling a seamless information flow from raw data to accurate material classification results, thereby improving the accuracy of RF-MID in material identification.

[0051] To verify the ability of the fine-grained material recognition method based on a deep learning network model proposed in this embodiment to recognize fine-grained materials, materials such as... Figure 4 The 24 different types of materials shown are: (a) to (r) are flat materials, (s) to (x) are ceramic materials, and the ceramic materials in (u) to (v) are similar in material and shape, and all have a layer of glaze on the surface.

[0052] To better train the deep learning network model and test its performance, four objects with different shapes were selected for each type of material. Two objects were randomly selected from the four objects to train the deep learning network model, and the remaining two were used to test the performance of the deep learning network model. The results of repeated tests showed that the accuracy of ceramic material recognition in this embodiment was as high as 78.5%. It can be seen that it can identify very similar materials with different shapes, and the recognition results are not greatly affected by the surface covering.

Claims

1. A deep learning-based fine-grained material recognition method based on millimeter-wave radar, characterized in that, Includes the following steps: Step 1: Construct a fine-grained material identification system RF-MID based on a deep learning network model, including a data reconstruction module, a signal preprocessing module, a material feature extraction module, and a classifier module; Step 2: The data reconstruction module reconstructs the reflected signal received by the millimeter-wave radar into a three-dimensional data block containing time-domain feature variation information, multi-angle information of different antennas, and frequency distribution feature information of multiple samples. Step 3: The signal preprocessing module uses a variational autoencoder model to preprocess the input three-dimensional data block, thereby removing noise interference caused by the material shape. Step 4: The material feature extraction module uses a three-dimensional complex-valued dilated convolutional network to extract high-dimensional feature information from the preprocessed three-dimensional data blocks, thereby generating feature maps that characterize material differences. The three-dimensional complex-valued dilated convolutional network includes dilated convolution, four three-dimensional complex-valued convolutional layers, and two residual blocks. The three-dimensional complex-valued convolutional layers are three-dimensional complex-valued convolutional layers used to capture subtle changes and diverse features of signals. They can preserve local features under short-time sampling, extract global information from multiple data segments, and introduce multi-angle features from multiple antennas. The dilated convolution introduces at least one dilation rate inside the convolutional kernel to expand the receptive field of the kernel. The residual blocks include two complex-valued dilated convolutional layers for extracting depth features and a max-pooling layer for dimensionality reduction. Step 5: The classifier module compresses the extracted feature map to obtain a one-dimensional material category feature for outputting material category scores, thereby identifying fine-grained materials.

2. The deep learning-based fine-grained material recognition method based on millimeter-wave radar according to claim 1, characterized in that, The process of reassembling the three-dimensional data blocks is as follows: The number of antennas, sampling points, chirps, and frames in the millimeter-wave radar parameter settings are obtained. The reflected signal data received by the millimeter-wave radar is reassembled into a three-dimensional data block in the form of frames. The first dimension of the three-dimensional data block is the multi-angle information of different antennas, the second dimension is the sampling time-domain information within a chirp, and the third dimension arranges all chirps in each frame side by side, thereby expanding the multi-sample information of static objects or recording the change trend information of dynamic objects over a long period of time.

3. The deep learning-based fine-grained material recognition method based on millimeter-wave radar according to claim 1, characterized in that, The variational autoencoder model preprocesses the input 3D data block as follows: Step 3.1: Input simple shapes with different curvatures into the variational autoencoder model. Use the reflection signal of the flat object as a label. The variational autoencoder model learns and smooths out the difference between the reflection signal of the simple shape and the reflection signal of the flat object to obtain the trained variational autoencoder model. Step 3.2: The trained variational autoencoder model normalizes the input arbitrary reflection signal to remove noise interference caused by the material shape.

4. The deep learning-based fine-grained material recognition method based on millimeter-wave radar according to claim 1, characterized in that, The dilation rate of the dilated convolution increases exponentially, doubling with each complex convolutional layer, thus exponentially expanding the receptive field.

5. The deep learning-based fine-grained material recognition method based on millimeter-wave radar according to claim 1, characterized in that, The classifier module includes two complex-valued fully connected layers and four real-valued fully connected layers. The complex-valued fully connected layers and the four real-valued fully connected layers unify the dimensions through modulo operation, thereby achieving mapping to the number of categories for output.

Images