Through-the-wall radar high-resolution imaging method based on large language model

Abstract

The invention belongs to the technical field of radar signal processing, and particularly relates to a through-the-wall radar high-resolution imaging method based on a large language model. The method specifically comprises the steps of 1, obtaining a BP radar imaging result and a corresponding optical image, and constructing a data set; 2, building and training a high-resolution imaging network based on a large language model; the network comprises an input adapter, a large language model and an output adapter; a low-rank adaptation module is arranged in the large language model; when the data set is used for training, freezing a query projection matrix and a value projection matrix of each Transform module in a pre-trained large language model, only training a low-rank matrix in the low-rank adaptation module, and adding the low-rank matrix to the query projection matrix and the value projection matrix; and step 3, inputting a BP radar image to be processed into the imaging network to obtain a high-resolution imaging result containing shape and contour information.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of radar signal processing, and particularly relates to a through-wall radar high-resolution imaging method based on a large language model. BACKGROUND

[0002] With the continuous advancement of urbanization, the demand for concealed space detection has significantly increased, and plays an important role in the fields of anti-terrorism, security, rescue, medical treatment, etc.

[0003] Through-wall radar (TWR) uses low-frequency electromagnetic waves to penetrate buildings to realize the imaging display of concealed space targets, and is a main technical means for concealed space detection.

[0004] At present, the traditional through-wall radar imaging algorithm is limited by the antenna aperture and array spacing, and no matter what the shape of the target to be imaged is, the imaging result is in the form of a fuzzy spot, which cannot directly reflect the shape information of the target and is difficult to directly identify and use. Therefore, it is an urgent problem in the field to study a high-resolution method for through-wall radar. SUMMARY

[0005] The present application proposes a through-wall radar high-resolution imaging method based on a large language model (LLM), which can obtain high-resolution imaging results from low-resolution radar images to solve the problems of insufficient imaging resolution and difficulty in directly identifying and using the imaging results.

[0006] The technical solutions of the present application are as follows:

[0007] In a first aspect, the present application is a through-wall radar high-resolution imaging method based on a large language model, comprising:

[0008] Step 1: Obtain the BP radar imaging result and the corresponding optical image, and construct a data set;

[0009] Simulation imaging result acquisition: move the BP radar antenna by an interval d1, establish a through-wall echo signal model of different positions of the antenna, and construct echo data imaging results of different targets and different rotation angles of the target based on the model;

[0010] Actual measurement imaging result acquisition: move the BP radar antenna by an interval d1, collect echo data of different positions of the antenna to obtain radar imaging, and repeat the above process to obtain echo data imaging results of different targets and different rotation angles of the target;

[0011] Step 2: Build and train a high-resolution imaging network based on a large language model;

[0012] The network comprises an input adapter, a large language model, and an output adapter; a low-rank adaptation module is arranged in the large language model;

[0013] During training of the data set, the query projection matrix and the value projection matrix of each Transformer module in the pre-trained large language model are frozen, only the low-rank matrix in the low-rank adaptive module is trained, and the low-rank matrix is added to the query projection matrix and the value projection matrix;

[0014] Step 3: input the BP radar image to be processed into the imaging network to obtain a high-resolution imaging result containing shape and contour information.

[0015] Optionally, the wall penetration echo signal model of the different position antenna in the application is superimposed by target reflection signals wall direct reflection signals reflection signals between targets and noise v m,k .

[0016] Optionally, the radar transmission signal in the application is a stepped frequency signal, the initial frequency is f0, the stepped frequency interval is Δf, and the frequency within the bandwidth is:

[0017] f k = f0 + kΔf, k = 0,..., K-1

[0018] The received echo signal model at the mth antenna position and the kth frequency is represented as:

[0019]

[0020] Wherein, P is the number of targets, W is the number of wall direct reflection paths, R is the number of multiple reflection paths on the target, N x ×N y is the size of the imaging scene, and σ pq are the reflection coefficients of the wth wall direct reflection path, the rth reflection path on the pth target, and the pth and qth target-to-target path, respectively.

[0021] Optionally, the application collects echo data of M groups of different position antennas, performs IFFT transformation on the echo data to obtain s(m, k), and calculates the two-way time delay τ mh of the mth antenna position and the hth pixel of the imaging area.

[0022]

[0023] Optionally, the input adapter of the present application encodes the 3D radar image into a series of embedding vectors with the same dimension as the language embedding, and the input adapter is composed of 6 sequentially connected 3D convolution layers and 1 Linear layer, and each 3D convolution layer is followed by a BatchNorm layer, a LeakyReLU layer and a BatchNorm layer.

[0024] Optionally, the query projection matrix W q and the value projection matrix W q in the Transformer module of the large language model of the present application are W q ′ and W v ′ are the query projection matrix and the value projection matrix after adding a low-rank matrix;

[0025] W q ′ = W q + ΔW q = W q + A q B q , W v ′ = W v + ΔW v = W v + A v B v

[0026] wherein A q and A v are low-rank matrices with dimensions d in × r, B q and B v are low-rank matrices with dimensions r × d out , and r is a set parameter.

[0027] Optionally, the low-rank matrices A q and A v of the present application are randomly initialized, and the low-rank matrices B q and B v are initialized to all 0.

[0028] Optionally, the loss function L during training of the present application is:

[0029] L = λ MSE L MSE + λ p L p + λ COS L COS

[0030] wherein L MSE is the MSE loss between the optical image and the generated image, L pis the perceptual loss calculated by the pre-trained VGG network, L COS is the loss calculated based on cosine similarity, λ MSE ,λ p and λ COS To set the weight.

[0031] In a second aspect, the present invention provides a through-wall radar high-resolution imaging device based on a large language model, comprising: an input adapter, a large language model, and an output adapter, obtained by training according to any one of claims 1 to 8;

[0032] Input adapter, used to receive radar images and generate embedding vector X;

[0033] A large language model is used to process the embedding vector X to generate a hidden state vector;

[0034] Output adapter, used to reconstruct the output of the large language model into a high-resolution image containing shape and contour information.

[0035] Beneficial effects:

[0036] The present invention proposes a through-wall radar high-resolution imaging method based on a large language model (LLM), and designs a high-resolution imaging network based on the large language model, which consists of an input adapter, an output adapter, and an LLM including a low-rank adaptation module. The input adapter encodes the radar image into an embedding vector that can be processed by the LLM. The embedding vector is processed by the LLM and then input into the output adapter to obtain the image result. In addition, the pre-trained parameters of the LLM are frozen and a trainable low-rank matrix is ​​introduced in this process to reduce the computational complexity. Finally, the network can achieve high-resolution imaging of stationary targets, reconstruct the shape, contour and other information of the target, so that people can directly identify and use it. The network designed by the present invention can obtain high-resolution imaging results from low-resolution radar images, reconstruct the shape, contour and other information of the target, so that people can directly identify and use it. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0038] Figure 1 is a flow chart of an embodiment of the present invention;

[0039] Figure 2 It is a schematic diagram of the signal scenario in the present invention;

[0040] Figure 3is a schematic diagram of the network structure in the present application;

[0041] Figure 4 is an example of a simulation data set of the present application, where (a) is a simulation model diagram, (b) is a data set label, (c) is a three-dimensional BP radar diagram, (d) is a top view, and (e) is a front view;

[0042] Figure 5 is a schematic diagram of a measured scene of the present application, where (a) is a non-wall-penetrating scene, (b) is a radar behind the wall in a wall-penetrating scene, and (c) is a target placement schematic diagram in a wall-penetrating scene;

[0043] Figure 6 is a part of the test results of the present application on simulation and measured data, where (a) is an imaged target in a simulation and measured scene, (b) is a true value diagram, (c) is a three-dimensional diagram of a radar BP imaging result, (d) is a top view of a radar imaging result, (e) is a front view, (f) is a left view, and (g) is a generated result of the proposed method. DETAILED DESCRIPTION

[0044] The embodiments of the present application will be described in detail below with reference to the accompanying drawings.

[0045] It should be noted that the following embodiments and features in the embodiments can be combined with each other without conflict; and all other embodiments obtained by those of ordinary skill in the art based on the embodiments in the present disclosure without creative labor are within the scope of protection of the present disclosure.

[0046] It should be noted that the various aspects of the embodiments described below are within the scope of the appended claims. It should be apparent that the aspects described herein can be embodied in a wide variety of forms and that any specific structure and / or function described herein is merely illustrative. Based on the teachings herein one skilled in the art should appreciate that an aspect described herein can be implemented independently of any other aspects and that two or more aspects can be combined in various ways. For example, an apparatus can be implemented and / or a method practiced using any number of the aspects set forth herein. In addition, an apparatus can be implemented and / or a method practiced using other structure and / or functionality in addition to or other than one or more of the aspects set forth herein.

[0047] As Figure 1 shown, the present application embodiment is a wall-penetrating radar high-resolution imaging method based on a large language model, and the specific steps include:

[0048] Step 1: Obtain the BP radar imaging result and the corresponding optical image, and construct a data set;

[0049] Acquisition of simulation imaging results: Assume that the BP radar antenna is moved at intervals of d1, establish a model of the echo signal of the antenna passing through the wall at different positions, and construct the imaging results of the echo data of different targets and different rotation angles of the target based on the model;

[0050] Acquisition of measured imaging results: Move the BP radar antenna at intervals of d1 to collect echo data from antennas at different positions to obtain radar imaging. Repeat the above process to obtain imaging results of echo data for different targets and at different rotation angles of the target.

[0051] Step 2: Build and train a high-resolution imaging network based on a large language model;

[0052] The network includes an input adapter, a large language model, and an output adapter; a low-rank adaptation module is set in the large language model;

[0053] When training with the dataset, freezing the query projection matrix and the value projection matrix of each Transformer module in the pre-trained large language model, training only the low-rank matrix in the low-rank adaptation module, and using the low-rank matrix to update the query projection matrix and the value projection matrix;

[0054] Step 3: Input the BP radar image to be processed into the imaging network to obtain a high-resolution imaging result containing shape and contour information.

[0055] The specific implementation of each of the above steps is described in detail below:

[0056] Step 1: Obtain BP radar imaging results and corresponding optical images (i.e., dataset labels) and construct a dataset. The specific process is as follows:

[0057] Consider Figure 2 The size of the image including one reflection is N x ×N y In the imaging scenario, the radar transmission signal s(t) is a stepped frequency signal with a starting frequency of f0, a stepped frequency interval of Δf, and K frequency points. The frequency within the bandwidth is:

[0058] f k =f0+kΔf,k=0,...,K-1.

[0059] Assume the wall thickness is d wall , the dielectric constant is ε r , move the antenna at interval d1, then the position of the mth antenna is (x m ,-d1). Considering the direct reflection and primary reflection signals of the target, the direct reflection signal of the wall and the reflection signal between the targets, the received echo signal at the mth antenna position and the kth frequency can be regarded as the target reflection signal The wall directly reflects the signal Reflection signals between targets and noise v m,k The superposition of the above can be expressed as

[0060]

[0061] Where P is the number of targets, W is the number of wall-direct reflection paths, R is the number of multiple reflection paths on a target, and r represents the reflection path number, and σ pq are the reflection coefficients of the wthwall-direct reflection path, the rthreflection path on the pthtarget, and the pthand qthtarget-to-target path, respectively. τ pqm represents the received signal after the signal transmitted by the mthantenna is reflected by the pthand qthtargets.

[0062] Taking the target reflection path Path-A shown in Figure 2 as an example, τ pm is the two-way time delay from the mthantenna to the pthtarget. is the two-way time delay of the wthwall-direct reflection path, which can be expressed as:

[0063]

[0064] Where l1is the propagation distance of the electromagnetic wave in the wall, l2and l3are the propagation distances of the electromagnetic wave in the air, is the length of the wthwall-direct reflection path, and v is the propagation speed of the electromagnetic wave in the wall. According to Snell's law, v can be expressed as

[0065]

[0066] Where c is the speed of light.

[0067] For three-dimensional BP imaging, the imaging region is divided into a grid along the xyz axes, and echo data of M groups of antennas at different positions are collected. The s(m, k) is obtained by IFFT transformation of the echo data, and the two-way time delay τ mh between the mthantenna position and the hthpixel in the imaging region is calculated.

[0068]

[0069] The imaging results of all pixels on the entire region are calculated.

[0070] As shown in Figures 4-5As shown, the above steps are repeated to collect echo data of different targets and different rotation angles of the target and perform imaging to construct a complete simulation data set. When the simulation data set is constructed, s(m, k) in the above formula is obtained through the above model. The actual measurement experiment is performed to construct a complete measured data set. When the measured data set is constructed, s(m, k) in the above formula is the echo data actually obtained after IFFT transformation.

[0071] Step 2: Building and training a high-resolution imaging network based on a large language model

[0072] The present application uses a deep learning model based on a large language model for high-resolution imaging, which is composed of an input adapter, an output adapter and an LLM containing a low-rank adaptation module, and the overall processing flow is as shown in Figure 3

[0073] During the training process, the 3D radar image is first processed and converted into an embedding representation by the input adapter, and the embedding representation is input into the LLM to obtain an output in the form of a vector, and then processed by the output adapter to obtain a high-resolution 2D image containing reconstructed detailed information. In this process, the pre-training parameters of the LLM are frozen to save computing resources, and a low-rank adaptation is introduced to enable the LLM to maintain learning ability.

[0074] Input adapter: Since the size of the input 3D radar image is 32x32x32, it is very different from the usual text input of the LLM, which causes the LLM to be unable to directly process the radar image. Therefore, the input adapter is designed to encode the 3D radar image into a series of embedding vectors with the same dimension as the language embedding. The input adapter is composed of 6 sequentially connected 3D convolution layers and 1 Linear layer, which can map the 3D radar image into a 4096x1 vector. Each 3D convolution layer is followed by a BatchNorm layer and a LeakyReLU layer. The BatchNorm layer is used to stabilize the training and improve the convergence of the network, and the LeakyReLU layer introduces nonlinearity to prevent gradient vanishing or degradation. After this process, the key features of the input radar image are extracted into the embedding vector X, which can be further processed by the Transformer module in the LLM in a manner similar to text processing.

[0075] ​Low-rank adaptation: Large language models usually have a parameter quantity of billions, if all parameters are updated during training, it will cause a large amount of training time and computational resource consumption. In order to improve the efficiency of the training process and save computing resources, the present invention introduces a low-rank matrix, which limits the training to a small number of parameters, namely low-rank adaptation (LoRA). LoRA injects trainable low-rank matrices into each layer of the original model and only optimizes these additional low-rank matrices. The present invention uses Llama2 as the base large language model, and adds additional low-rank matrices to the query projection matrix W q and the value projection matrix W q in each Transformer module of LLM, then the process of self-attention mechanism in Transformer can be represented as:

[0076]

[0077] where Q = W q 'X and V = W v 'X are the query matrix and value matrix respectively. W q ' and W v ' are the query projection matrix and value projection matrix after adding low-rank matrices. During the training process, only ΔW q and ΔW v parameters are updated, the pre-training parameters of LLM are frozen, which can be represented as:

[0078] W q ' = W q + ΔW q = W q + A q B q , W v ' = W v + ΔW v = W v + A v B v

[0079] where A q and A v are low-rank matrices with dimension d in × r, B q and B v are low-rank matrices with dimension r × d out . A q and A v are randomly initialized, B q and B v are initialized to all 0. Since r << min{d in , d out},only 2.09% of the parameters are updated, thereby saving computing resources and improving computing efficiency.

[0080] Output adapter: After the embedded vector obtained through the input adapter is processed by the LLM, the output still has the same dimension as the input embedding and is not the required two-dimensional image. In order to obtain effective output results, the present application introduces an output adapter to bridge the gap between the output of the LLM and the two-dimensional image pixel space. That is, the output adapter is to reconstruct the output of the LLM into a meaningful and interpretable image, which consists of 5 layers of transposed convolutional layers and normalization layers and ReLU activation functions, for upsampling the output of the LLM back to a 128x128 image, thereby ensuring the reliability of the LLM in image generation.

[0081] Loss function: the loss function in the training process contains 3 parts, which can be expressed as:

[0082] L=λ MSE L MSE +λ p L p +λ COS L COS

[0083] Wherein, L MSE is the MSE loss between the true label and the generated image to ensure the accuracy and consistency of the generated image and the label at the pixel level. L p is the perceptual loss calculated by the pre-trained VGG network (Visual Geometry Group Network, perceptual feature extraction network, input is the generated image and the label) to make the generated image more consistent with human eye perception. L COS is the loss based on cosine similarity calculation to ensure that the generated image has overall structural consistency with the label image. λ MSE , λ p and λ COS are constants, so that the generated image is correct in detail, consistent in overall structure and at the same time easier to be directly recognized by the human eye.

[0084] Step 3: input the BP radar image into the network to obtain a high-resolution imaging result containing shape and contour information.

[0085] Thus, a through-wall radar high-resolution imaging method based on a large language model is completed.

[0086] Embodiment:

[0087] According to step 1, the simulated transmitted signal is a step frequency signal, the antenna step is 5 cm, the imaging area is 3m x 3m x 4m, and a table (Tablesim ), Human sim ), Human sim ). The transmit signal is set as stepped frequency signal with frequency interval of 2MHz and frequency range of 1.7GHz-2.2GHz. The antenna array is 10Tx10Rx with size of 40cmx40cm. The distance between radar and target is 2m in the non-wall scenario. In the wall-penetrating scenario, the wall thickness is 0.2m, the radar is close to one side of the wall, and the target is 2m away from the other side of the wall. The datasets containing Chair-1 exp ), Chair-2 exp ), Human exp ) with different rotation angles are collected.

[0088] According to step 2, the AdamW optimizer with batch size of 4 and initial learning rate of 2e-4 is set to train the high-resolution imaging network to enable it to recover the shape of the target.

[0089] According to step 3, the simulated and measured radar imaging data are input into the trained imaging network to obtain high-resolution imaging results, and the cosine similarity (COS Similarity), structural similarity index (SSIM), learning perceptual image block similarity (LPIPS), intersection over union (IoU), and surface loss percentage (Surface Miss) are used as evaluation indicators.

[0090] Figure 6 Examples of imaging results of the network on simulated and measured data are shown, where (a) is the imaged target in the simulated and measured scene, (b) is the ground truth image, (c) is the three-dimensional image of the radar BP imaging result, (d) is the top view of the radar imaging result, (e) is the front view, (f) is the left view, and (g) is the generated result of the proposed method. It can be seen that the resolution of the three-dimensional image and the three-view image of the radar image is low, and the shape information of the target type cannot be directly distinguished. The proposed method can obtain high-resolution imaging results close to the labeled image, and can reconstruct the shape contour and other detailed information of the imaged target, so that the human eye can directly recognize and use it. Table 1 shows the results of cosine similarity, SSIM, LPIPS, IoU and surface loss percentage on the test set, which shows that each indicator has improved, proving the effectiveness of the improved resolution imaging method.

[0091] Table 1 Comparison of result statistics on test set

[0092]

[0093] In summary, the application proposes a through-wall radar high-resolution imaging method based on a large language model. The proposed model consists of an input adapter, an output adapter, and an LLM containing low-rank adaptation. Simulation and real measurement experiments show that the proposed method has the ability to perform high-resolution imaging on static targets and obtain imaging results that can be directly recognized by the human eye.

[0094] The above is the preferred embodiment of the application. It should be noted that for those skilled in the art, without departing from the technical principles of the application, several improvements and modifications can be made, and these improvements and modifications should also be considered as the protection scope of the application.

Claims

1. A through-wall radar high-resolution imaging method based on a large language model, characterized in that: include: Step 1: Obtain BP radar imaging results and corresponding optical images to construct a data set, where the imaging results include simulated imaging and measured imaging; Step 2: Build and train a high-resolution imaging network based on a large language model; The network includes an input adapter, a large language model, and an output adapter; a low-rank adaptation module is set in the large language model; When training with the dataset, freezing the query projection matrix and the value projection matrix of each Transformer module in the pre-trained large language model, training only the low-rank matrix in the low-rank adaptation module, and using the low-rank matrix to add to the query projection matrix and the value projection matrix; Step 3: Input the BP radar image to be processed into the imaging network to obtain a high-resolution imaging result containing shape and contour information.

2. The through-wall radar high-resolution imaging method based on a large language model according to claim 1 is characterized in that: Acquisition of simulation imaging results: Assume that the BP radar antenna is moved at intervals of d1, establish a model of the echo signal of the antenna passing through the wall at different positions, and construct the imaging results of the echo data of different targets and different rotation angles of the target based on the model; Acquisition of measured imaging results: Move the BP radar antenna at intervals of d1 to collect echo data from antennas at different positions to obtain radar imaging. Repeat the above process to obtain echo data imaging results for different targets and different rotation angles of the target.

3. The through-wall radar high-resolution imaging method based on a large language model according to claim 1 is characterized in that: The wall echo signal model of antennas at different positions is composed of target reflected signals The wall directly reflects the signal Reflected signals between targets and noise v m,k of superposition.

4. The through-wall radar high-resolution imaging method based on a large language model according to claim 3 is characterized in that The radar transmission signal is a stepped frequency signal with a starting frequency of f0 and a stepped frequency interval of Δf. It contains K frequency points, so the frequency within the bandwidth is: f k =f0+kΔf,k=0,...,K-1 The received echo signal model at the mth antenna position and the kth frequency is expressed as: Where P is the number of targets, W is the number of direct reflection paths from the wall, R is the number of multiple one-time reflection paths on the target, and N x ×N y is the size of the imaging scene, and σ pq are the reflection coefficients of the wth wall direct reflection path, the rth reflection path on the pth target, and the path between the pth and qth targets, respectively.

5. The through-wall radar high-resolution imaging method based on a large language model according to claim 2, characterized in that: Collect echo data from M groups of antennas at different positions, perform IFFT transformation on the echo data to obtain s(m,k), and calculate the round-trip delay τ between the mth antenna position and the hth pixel in the imaging area. mh , then the imaging result of the h-th pixel is:

6. The through-wall radar high-resolution imaging method based on a large language model according to claim 1, characterized in that: The input adapter encodes the 3D radar image into a series of embedding vectors with the same dimensions as the language embedding. The input adapter consists of six 3D convolutional layers and one Linear layer connected in sequence, and each 3D convolutional layer is followed by a BatchNorm layer, a LeakyReLU layer, and a BatchNorm layer.

7. The through-wall radar high-resolution imaging method based on a large language model according to claim 1, characterized in that: The query projection matrix W in the Transformer module of the large language model q Sum projection matrix W q , W q ′ and W v ′ is the query projection matrix and value projection matrix after adding the low-rank matrix; IN q ′=W q +ΔW q =In q +A q B q ,IN v ′=W v +ΔW v =In v +A v B v Among them, A q and A v The dimension is d in ×r low-rank matrix, B q and B v The dimension is r×d out The low-rank matrix, r sets the parameters.

8. The through-wall radar high-resolution imaging method based on a large language model according to claim 7, characterized in that: The low-rank matrix A q and A v Using random initialization, the low-rank matrix B q and B v Initialized to all 0s.

9. The through-wall radar high-resolution imaging method based on a large language model according to claim 1, characterized in that: The loss function L during training is: L=λ MSE L MSE +λ p L p +λ COS L COS Among them, L MSE is the MSE loss between the optical image and the generated image, L p is the perceptual loss calculated by the pre-trained VGG network, L COS is the loss calculated based on cosine similarity, λ MSE ,λ p and λ COS To set the weight.

10. A high-resolution through-wall radar imaging device based on a large language model, characterized in that: include: The input adapter, the large language model, and the output adapter are obtained by training according to any one of claims 1 to 8; Input adapter, used to receive radar images and generate embedding vector X; A large language model, used to process the embedding vector X to generate a hidden state vector; Output adapter, used to reconstruct the output of the large language model into a high-resolution image containing shape and contour information.

Images