Millimeter wave radar human perception method based on time-frequency domain and deep cnn

Abstract

The application provides a millimeter wave radar personnel sensing method based on a time-frequency domain and a deep CNN, relates to the technical field of radar signal processing, and comprises preprocessing, spectrum conversion and enhancement of a millimeter wave radar echo signal; spatial features are extracted by using a deep separable convolution with a residual structure, time-frequency features are extracted by combining a dual-tree complex wavelet transform and an attention-enhanced hollow convolution, and motion features are extracted by optical flow estimation and three-dimensional convolution, and the three are adaptively fused; a dynamic space-time reasoning network is constructed, key space-time features are obtained by using a recurrent neural tensor network, a non-local neural network and a deformable convolution, multi-scale features are extracted and aligned in an adversarial manner, and a dynamic routing mechanism based on a capsule network is used to complete personnel target classification. The application can effectively suppress clutter interference, fully utilize spatial, time-frequency and motion information, realize accurate sensing of personnel targets, and improve classification accuracy.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of radar signal processing, in particular to a millimeter wave radar personnel perception method based on time-frequency domain and deep CNN. BACKGROUND

[0002] With the development of intelligent perception technology, millimeter wave radar plays an important role in personnel target detection and identification due to its advantages of all-weather working, strong penetration ability and being not limited by light conditions. In the traditional millimeter wave radar personnel perception method, the target features are mainly obtained by time-frequency analysis of the echo signal, and feature extraction and classification are carried out based on deep convolutional neural network; among them, the micro-Doppler feature as a typical feature of personnel target can reflect the time-varying characteristics of human motion, and is an important basis for realizing target identification.

[0003] However, the existing millimeter wave radar personnel perception method still has the following shortcomings: the traditional signal processing method is easily disturbed by noise and clutter in complex environment, which reduces the accuracy of feature extraction; the conventional feature extraction method often only focuses on a single feature dimension, and cannot fully utilize the multi-dimensional feature information of the target; the existing deep learning model is mostly designed for image data, and the time sequence characteristics and frequency domain features of radar signals are not fully considered; there is a lack of effective feature alignment and migration mechanism, which leads to poor generalization performance of the model in different scenes.

[0004] In summary, there is an urgent need for a millimeter wave radar personnel perception method based on time-frequency domain and deep CNN to effectively suppress clutter interference and improve signal quality; to construct a multi-modal feature fusion framework to fully exploit target features; to design a deep learning model suitable for radar signal characteristics to enhance feature expression ability; and to establish a feature alignment mechanism to improve model generalization performance. By solving the above technical problems, the accuracy and robustness of millimeter wave radar personnel perception are improved. SUMMARY

[0005] The embodiment of the present application provides a millimeter wave radar personnel perception method based on time-frequency domain and deep CNN, which can solve the problems in the prior art.

[0006] The first aspect of the embodiment of the present application is,

[0007] A millimeter wave radar personnel perception method based on time-frequency domain and deep CNN is provided, comprising:

[0008] The personnel target echo signal is collected by the millimeter wave radar and preprocessed to obtain a preprocessed signal, a two-dimensional fast Fourier transform is performed on the preprocessed signal to obtain a range-Doppler spectrogram, a recursive threshold segmentation algorithm is used for clutter suppression to obtain a target spectrogram, the target spectrogram is converted into a three-dimensional feature tensor, and compression and enhancement processing is performed to obtain an enhanced feature image;

[0009] Parallel feature extraction is performed on the enhanced feature image to obtain spatial features, time-frequency features and motion features respectively, and an adaptive feature fusion module is used to allocate attention weights to the spatial features, time-frequency features and motion features to obtain a target feature vector;

[0010] A recurrent neural tensor network is used to iteratively decompose the target feature vector to obtain associated features, a non-local neural network is used to construct a feature association graph and a deformable convolutional network is used to extract key spatio-temporal features, the key spatio-temporal features are multi-scale decomposed and an adversarial feature alignment network is used to eliminate distribution differences, and a coupling weight routing based on a capsule network is used to complete personnel target classification.

[0011] In an alternative embodiment, a millimeter wave radar is used to collect personnel target echo signals and pre-process them to obtain pre-processed signals, a two-dimensional fast Fourier transform is performed on the pre-processed signals to obtain a range-Doppler spectrogram, a recursive threshold segmentation algorithm is used for clutter suppression to obtain a target spectrogram, the target spectrogram is converted into a three-dimensional feature tensor, and compression and enhancement processing is performed to obtain an enhanced feature image including:

[0012] A millimeter wave radar is used to collect personnel target echo signals, zero-mean normalization is used to eliminate direct current components, and a self-adaptive window function is used for windowing processing to obtain pre-processed signals; a two-dimensional fast Fourier transform is performed on the pre-processed signals to obtain a complex value matrix, and the complex value matrix is converted into a range-Doppler spectrogram;

[0013] The range-Doppler spectrogram is decomposed into an amplitude spectrum and a phase spectrum, the amplitude spectrum is initially segmented using a bimodal maximum inter-class variance method to obtain an initial mask, the amplitude spectrum mean value and the phase spectrum standard deviation are fused using a fusion weight coefficient based on the initial mask to obtain a new threshold, the iteration is stopped when the difference between the new threshold and the last iteration threshold is less than a preset difference threshold to obtain a final threshold, and the range-Doppler spectrogram is segmented using the final threshold to obtain a target spectrogram;

[0014] A multi-layer wavelet packet decomposition tree is constructed, the node with the minimum information entropy is selected in each layer of the multi-layer wavelet packet decomposition tree for decomposition to obtain an optimal basis function set; the optimal basis function set is used for time-frequency analysis of the target spectrogram to obtain a three-dimensional feature tensor; and after dimension reduction, compression and equalization processing, an enhanced feature image is obtained.

[0015] In an alternative embodiment, the range-Doppler spectrogram is decomposed into an amplitude spectrum and a phase spectrum, and the amplitude spectrum is initially segmented using a bimodal maximum inter-class variance method to obtain an initial mask including:

[0016] The range-Doppler spectrum is complexly decomposed to obtain an amplitude spectrum and a phase spectrum, a gray scale histogram is counted and smoothed to obtain a gray scale distribution curve, positions of a main peak and a secondary peak are determined in the gray scale distribution curve, a valley point position between the main peak and the secondary peak is extracted, and a main peak region and a secondary peak region are determined respectively;

[0017] A main peak gray scale value sum in the main peak region and a secondary peak gray scale value sum in the secondary peak region are calculated, an initial weight is determined according to a ratio of the main peak gray scale value sum to the secondary peak gray scale value sum, local gray scale variances of the main peak region and the secondary peak region are calculated respectively, and the initial weight is updated for the first time to obtain an updated weight;

[0018] A number of gray scale values belonging to a corresponding region in a neighborhood of each sampling point in the main peak region and the secondary peak region is counted, a product of the number of gray scale values and a preset neighborhood influence factor is taken as a neighborhood constraint term, the neighborhood constraint term is used to update the updated weight for the second time to obtain a final weight;

[0019] A candidate threshold value is selected in a gray scale value range of the valley point position, a weighted inter-class variance of the main peak region and the secondary peak region is calculated, the candidate threshold value that makes the weighted inter-class variance maximum is taken as an optimal segmentation threshold value, and the amplitude spectrum is binarized to obtain an initial mask through morphological processing.

[0020] In an optional embodiment, parallel feature extraction is performed on the enhanced feature image to obtain spatial features, time-frequency features and motion features respectively, and an adaptive feature fusion module is used to perform attention weight distribution on the spatial features, the time-frequency features and the motion features to obtain a target feature vector.

[0021] The enhanced feature image is input into a residual depth separable convolutional network to perform spatial feature extraction to obtain a depth spatial feature map, and residual connection operation is performed on the enhanced feature image to obtain a spatial feature map;

[0022] The enhanced feature image is subjected to double-tree complex wavelet transform decomposition to obtain a low-frequency feature subband and a high-frequency feature subband, the low-frequency feature subband is input into a dilated convolutional network to perform context feature extraction to obtain a context feature map, enhanced low-frequency feature map is obtained through channel attention calculation, the high-frequency feature subband is input into a direction-sensitive convolutional network to perform texture feature extraction to obtain a direction feature map, and the enhanced low-frequency feature map and the direction feature map are subjected to wavelet reconstruction to obtain a time-frequency feature map;

[0023] Optical flow calculation is performed on adjacent frames of the enhanced feature image to obtain an optical flow feature map, the enhanced feature image is spliced in a channel dimension to obtain a space-time feature sequence, and the space-time feature sequence is input into a three-dimensional convolutional network to perform time sequence feature extraction to obtain a motion feature map;

[0024] The normalized cross-correlation between each two of the spatial feature map, the time-frequency feature map and the motion feature map is calculated to obtain a feature correlation degree, and the feature correlation degrees are combined to form a feature correlation matrix; a nonlinear transformation is performed on the feature correlation matrix to obtain attention weight coefficients, and the spatial feature map, the time-frequency feature map and the motion feature map are weighted and summed to obtain a target feature vector.

[0025] In an optional embodiment, the enhanced feature image is subjected to a dual-tree complex wavelet transform decomposition to obtain a low-frequency feature subband and a high-frequency feature subband, including:

[0026] A tree filter structure including a real tree filter bank and a complex tree filter bank is constructed, the real tree filter bank including a real tree low-pass filter and a real tree high-pass filter, the complex tree filter bank including a complex tree low-pass filter and a complex tree high-pass filter, and the complex tree filter bank being a Hilbert transform pair of the real tree filter bank;

[0027] The enhanced feature image is subjected to a row direction filtering through the real tree filter bank to obtain real tree row direction coefficients, and the real tree row direction coefficients are subjected to a column direction filtering through the complex tree filter bank to obtain real tree first-level coefficients; the enhanced feature image is subjected to a row direction filtering through the complex tree filter bank to obtain complex tree row direction coefficients, and the complex tree row direction coefficients are subjected to a column direction filtering through the real tree filter bank to obtain complex tree first-level coefficients;

[0028] The real tree first-level coefficients and the complex tree first-level coefficients are respectively taken as real parts and imaginary parts to construct complex transform coefficients; the amplitudes and phases of the complex transform coefficients are calculated, a local direction angle value is calculated according to the phases, an angle distribution histogram is counted, and a peak value corresponding angle is selected as a dominant decomposition angle; the filter bank response is subjected to a rotation transformation according to the dominant decomposition angle to obtain a directional filter bank;

[0029] The enhanced feature image is subjected to a filtering decomposition through the directional filter bank to obtain second-level coefficients; the second-level coefficients are subjected to a frequency domain separation, and a low-frequency feature subband and a high-frequency feature subband are extracted according to a preset frequency threshold.

[0030] In an optional embodiment, a recurrent neural tensor network is adopted to iteratively decompose the target feature vector to obtain associated features, a feature association graph is constructed through a non-local neural network, and key spatiotemporal features are extracted by combining a deformable convolutional network; the key spatiotemporal features are subjected to a multi-scale decomposition, and a distribution difference is eliminated by using an adversarial feature alignment network; and a personnel target classification is completed based on a coupling weight routing of a capsule network, including:

[0031] A recurrent neural tensor network is constructed, the target feature vector is mapped to a tensor space through a tensor and a bias vector, and iteratively and recurrently decomposed through a nonlinear activation function; when a feature change amount is less than a preset change amount threshold, the iteration is terminated to obtain associated features;

[0032] The associated features are input into a non-local neural network to construct a non-local feature association graph; the associated features are weighted and aggregated based on the edge weights in the non-local feature association graph to obtain weighted features, the weighted features are input into a deformable convolution network, the offset of the sampling position of the convolution kernel is calculated, the convolution kernel position is adjusted according to the offset to perform convolution operation, and key spatio-temporal features are obtained;

[0033] The key spatio-temporal features are subjected to multi-scale feature decomposition, an adversarial feature alignment network including a feature extractor and a feature discriminator is constructed, scene classification of the source domain feature map and the target domain feature map is performed by the feature discriminator to obtain a discrimination loss, the feature extractor is subjected to iterative adversarial training based on the discrimination loss, and aligned features with consistent feature distribution are obtained;

[0034] The aligned features are input into a low-level capsule of a capsule network, the coupling coefficient with a high-level capsule is calculated and normalized to obtain a routing weight, the output features of the low-level capsule are weighted and summed according to the routing weight, the result of the weighted sum is mapped through a nonlinear compression function, the output features of the high-level capsule are obtained, and a target classification is determined.

[0035] In an alternative embodiment, the associated features are input into a non-local neural network to construct a non-local feature association graph, which includes:

[0036] The associated features are subjected to discrete cosine transformation through a feature transformation layer of the non-local neural network to obtain frequency domain features, a multi-band power spectral density matrix is calculated and subjected to eigenvalue decomposition to obtain feature spectral bases and spectral coefficients, an orthogonal projection matrix is constructed based on the feature spectral bases, and the associated features are projected into a spectral space to obtain spectral projection features.

[0037] In a similarity calculation layer of the non-local neural network, a spectral similarity matrix is calculated using the spectral projection features, an initial spectral graph is constructed and a Laplacian matrix is calculated, the optimal subgraph number is determined through eigenvalue decomposition, and the initial spectral graph is divided into multiple subgraphs using a spectral clustering algorithm based on the optimal subgraph number.

[0038] In a graph structure generation layer of the non-local neural network, the degree centrality and feature vector centrality of each subgraph node are calculated, a local centrality index is determined, a subgraph key node is selected by selecting a subgraph node with a local centrality index greater than a preset local centrality threshold, and a hierarchical feature association graph is constructed by taking the similarity of the subgraph key nodes as the edge weight between different subgraphs.

[0039] In the dynamic update layer of the non-local neural network, the feature vectors of the subgraph nodes in multiple continuous historical update periods are extracted, the Euclidean distance sequence and the coefficient of variation between the feature vectors of adjacent periods are calculated to obtain a stability index; when the stability index is less than a preset stability threshold, node splitting is performed; when the similarity between adjacent subgraph nodes is greater than a preset similarity threshold, the feature vectors are weighted and averaged to construct a fusion node; based on the node splitting and the fusion node, the edge weight is recalculated to obtain a non-local feature association graph.

[0040] A second aspect of the embodiment of the application,

[0041] A millimeter wave radar personnel perception system based on time-frequency domain and deep CNN is provided, comprising:

[0042] A first unit is configured to collect personnel target echo signals by using a millimeter wave radar and obtain preprocessed signals by preprocessing, perform two-dimensional fast Fourier transform on the preprocessed signals to obtain a range-Doppler spectrogram, perform clutter suppression on the spectrogram by using a recursive threshold segmentation algorithm to obtain a target spectrogram, convert the target spectrogram into a three-dimensional feature tensor, and obtain an enhanced feature image by compression and enhancement processing;

[0043] A second unit is configured to perform parallel feature extraction on the enhanced feature image to obtain spatial features, time-frequency features and motion features, respectively, and perform attention weight distribution on the spatial features, time-frequency features and motion features by using an adaptive feature fusion module to obtain a target feature vector;

[0044] A third unit is configured to perform iterative decomposition on the target feature vector by using a recurrent neural tensor network to obtain associated features, construct a feature association graph by using a non-local neural network and extract key spatiotemporal features by combining a deformable convolutional network, perform multi-scale decomposition on the key spatiotemporal features and eliminate distribution differences by using an adversarial feature alignment network, and complete personnel target classification based on the coupling weight routing of a capsule network.

[0045] A third aspect of the embodiment of the application,

[0046] An electronic device is provided, comprising:

[0047] A processor;

[0048] A memory for storing processor-executable instructions;

[0049] The processor is configured to invoke the instructions stored in the memory to perform the method described above.

[0050] A fourth aspect of the embodiment of the application,

[0051] A computer-readable storage medium is provided, which stores computer program instructions, and the computer program instructions are executed by a processor to implement the method described above.

[0052] In the embodiment of the present application, the multi-modal spectral domain fusion recursive threshold segmentation algorithm effectively suppresses clutter interference, and combines adaptive multi-resolution wavelet packet decomposition and time series reconstruction algorithm to extract more fine target features, thereby improving the accuracy of personnel target detection and classification; the fusion of residual structure deep separable convolution, dual-tree complex wavelet transform, attention enhanced hollow convolution, optical flow estimation and three-dimensional convolution and other feature extraction methods further improves the feature expression ability, so that the personnel perception is more accurate; the multi-modal spectral domain fusion recursive threshold segmentation algorithm is used for clutter suppression, which effectively reduces the influence of environmental noise and clutter on personnel target detection, and improves the robustness of the system. The adaptive feature fusion module can allocate weights according to the importance of different features, further enhancing the adaptability of the system to complex environment; the dynamic space-time reasoning network is used, combined with recursive neural tensor network, non-local neural network and deformable convolution to extract key space-time features, and through the alignment of the adversarial features, the distribution difference is eliminated, and finally the personnel target classification is completed based on the dynamic routing mechanism of the capsule network; this multi-level feature extraction and classification strategy can realize the fine differentiation of different personnel targets, thereby meeting the more complex personnel perception application requirements. BRIEF DESCRIPTION OF DRAWINGS

[0053] Figure 1 The flowchart of the personnel perception method based on time-frequency domain and deep CNN of the embodiment of the present application is shown in the figure.

[0054] Figure 2 The edge effect diagram of morphological processing is shown in the figure.

[0055] Figure 3 The energy distribution comparison diagram of high-frequency subbands in different directions is shown in the figure.

[0056] Figure 4 The calculation efficiency comparison diagram of feature association modeling task is shown in the figure.

[0057] Figure 5 The structure diagram of the personnel perception system based on time-frequency domain and deep CNN of the embodiment of the present application is shown in the figure. DETAILED DESCRIPTION

[0058] In order to make the purpose, technical scheme and advantages of the embodiment of the present application clearer, the technical scheme in the embodiment of the present application will be described clearly and completely below in combination with the drawings in the embodiment of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0059] The technical solutions of the present application will be described in detail below with specific examples. The following specific examples can be combined with each other, and the same or similar concepts or processes may not be described in detail in some examples.

[0060] Figure 1 The flowchart of the personnel perception method based on time-frequency domain and deep CNN of the embodiment of the present application is shown in Figure 1 The method comprises:

[0061] The personnel target echo signal is collected by the millimeter wave radar and preprocessed to obtain a preprocessed signal. A two-dimensional fast Fourier transform is performed on the preprocessed signal to obtain a range-Doppler spectrogram. A recursive threshold segmentation algorithm is used for clutter suppression to obtain a target spectrogram. The target spectrogram is converted into a three-dimensional feature tensor. Compression and enhancement processing are performed to obtain an enhanced feature image.

[0062] Parallel feature extraction is performed on the enhanced feature image to obtain spatial features, time-frequency features, and motion features. An adaptive feature fusion module is used to assign attention weights to the spatial features, time-frequency features, and motion features to obtain a target feature vector.

[0063] A recurrent neural tensor network is used to iteratively decompose the target feature vector to obtain associated features. A non-local neural network is used to construct a feature association graph and extract key spatiotemporal features in combination with a deformable convolutional network. The key spatiotemporal features are subjected to multi-scale decomposition and an adversarial feature alignment network is used to eliminate distribution differences. A coupling weight routing based on a capsule network is used to complete personnel target classification.

[0064] In an alternative embodiment, the personnel target echo signal is collected by the millimeter wave radar and preprocessed to obtain a preprocessed signal. A two-dimensional fast Fourier transform is performed on the preprocessed signal to obtain a range-Doppler spectrogram. A recursive threshold segmentation algorithm is used for clutter suppression to obtain a target spectrogram. The target spectrogram is converted into a three-dimensional feature tensor. Compression and enhancement processing are performed to obtain an enhanced feature image, which comprises:

[0065] The personnel target echo signal is collected by the millimeter wave radar. A zero-mean normalization is used to eliminate the direct current component, and a self-adaptive window function is used for windowing processing to obtain a preprocessed signal. A two-dimensional fast Fourier transform is performed on the preprocessed signal to obtain a complex value matrix. The complex value matrix is converted into a range-Doppler spectrogram.

[0066] The range-Doppler spectrogram is decomposed into an amplitude spectrum and a phase spectrum. An initial segmentation of the amplitude spectrum is performed by a bimodal maximum interclass variance method to obtain an initial mask. Based on the initial mask, the amplitude spectrum mean and the phase spectrum standard deviation are weighted and fused by a fusion weight coefficient to obtain a new threshold. When the difference between the new threshold and the last iteration threshold is less than a preset difference threshold, the iteration is stopped to obtain a final threshold. The range-Doppler spectrogram is segmented using the final threshold to obtain a target spectrogram.

[0067] A multi-layer wavelet packet decomposition tree is constructed, a node with the minimum information entropy in each layer of the multi-layer wavelet packet decomposition tree is selected for decomposition, and an optimal basis function set is obtained; the optimal basis function set is used for time-frequency analysis of a target spectrum, and a three-dimensional feature tensor is obtained; after dimension reduction, compression and equalization processing, an enhanced feature image is obtained.

[0068] In a specific embodiment, personnel target echo signal acquisition and preprocessing: millimeter wave radar is used to collect the echo signal of the personnel target. Since the echo signal usually contains a direct current component, which will affect the subsequent signal processing, zero-mean normalization processing is needed to eliminate the influence of the direct current component. Specifically, the average value of the echo signal is calculated, and then the average value is subtracted from each sampling point. Next, in order to reduce spectral leakage, the signal is windowed using an adaptive window function. The selection of the adaptive window function can be adjusted according to the specific application scenario and radar parameters, for example, a Hanning window, a Hamming window or a Gaussian window can be selected. Assuming that the collected echo signal is a sequence [s1, s2,..., s1024] containing 1024 sampling points, and the average value is mean(s). Then the zero-mean normalized signal is [s1-mean(s), s2-mean(s),..., s1024-mean(s)]. Then, multiply the signal with the selected adaptive window function (such as Hanning window) point by point to get the preprocessed signal.

[0069] Range-Doppler spectrum generation: perform two-dimensional fast Fourier transform on the preprocessed signal to get a complex value matrix. This matrix contains information about the signal at different ranges and Doppler frequencies. Calculate the modulus of each element of the complex value matrix to get the range-Doppler spectrum, also known as the amplitude spectrum. At the same time, the phase angle of each element can also be calculated to get the phase spectrum. For example, assuming an element of the complex value matrix is a+bi, then its corresponding amplitude is sqrt(a 2 +b 2 ), and the phase angle is arctan(b / a).

[0070] Recursive threshold segmentation algorithm based on multi-modal spectral domain fusion for clutter suppression: To suppress the clutter in the range-Doppler spectrum, a recursive threshold segmentation algorithm based on multi-modal spectral domain fusion is adopted. The bimodal Otsu method is used to perform initial segmentation on the amplitude spectrum, obtaining an initial mask. This mask divides the spectrum into two parts: foreground (target) and background (clutter). According to the initial mask, the mean of the amplitude spectrum and the standard deviation of the phase spectrum of the foreground region are calculated. These two values are fused by a fusion weight coefficient to obtain a new threshold. The fusion weight coefficient can be adjusted according to the actual situation, for example, it can be set to 0.5, indicating that the contributions of the amplitude spectrum and the phase spectrum are the same. Compare the difference between the new threshold and the last iteration threshold. If the difference is less than a preset difference threshold (e.g., 0.01), stop iteration and use the current threshold as the final threshold. Otherwise, update the mask using the new threshold and repeat the above steps until the stopping condition is met. The final threshold is used to segment the range-Doppler spectrum to obtain the target spectrum.

[0071] Adaptive multi-resolution wavelet packet decomposition and time series reconstruction: To extract features from the target spectrum, adaptive multi-resolution wavelet packet decomposition is adopted. A multi-layer wavelet packet decomposition tree is constructed. In each layer, the node with the smallest information entropy is selected for decomposition until a preset decomposition layer number is reached. The smaller the information entropy, the more concentrated the information contained in the node, which is more conducive to feature extraction. Thus, a set of optimal basis functions is obtained. The target spectrum is analyzed using this set of basis functions to obtain a three-dimensional feature tensor. The three dimensions of the tensor represent time, frequency, and decomposition layer number, respectively. For example, assuming the size of the target spectrum is 64x64 and the decomposition layer number is 3, the size of the obtained three-dimensional feature tensor is 64x64x3.

[0072] Compression and enhancement processing: To reduce computational complexity and improve the robustness of features, compression and enhancement processing are performed on the three-dimensional feature tensor. Principal component analysis (PCA) is used to reduce the dimensionality of the three-dimensional feature tensor, for example, from 3 to 2. Max pooling is used to compress the obtained two-dimensional feature image, for example, using a 2x2 pooling window. Adaptive histogram equalization is performed on the compressed image to enhance the contrast and detail information of the image.

[0073] In this embodiment, through multi-modal spectral domain fusion and adaptive wavelet packet decomposition, clutter can be effectively suppressed and target features can be extracted, thereby improving the accuracy of target detection; the adaptive window function and adaptive histogram equalization processing adopted can enhance the robustness of the algorithm to different environments and noises; through principal component analysis and max pooling, the features are compressed, reducing the computational complexity and being conducive to real-time processing.

[0074] In an alternative embodiment, the range-Doppler map is decomposed into an amplitude spectrum and a phase spectrum, and an initial mask is obtained by performing an initial Otsu segmentation on the amplitude spectrum, including:

[0075] The range-Doppler map is complex-decomposed to obtain an amplitude spectrum and a phase spectrum, a gray level histogram is counted and smoothed to obtain a gray level distribution curve, the positions of the main peak and the secondary peak are determined in the gray level distribution curve, the valley point position between the main peak and the secondary peak is extracted, and the main peak region and the secondary peak region are determined respectively;

[0076] The total sum of the main peak gray level values in the main peak region and the total sum of the secondary peak gray level values in the secondary peak region are calculated, and the initial weight is determined according to the ratio of the total sum of the main peak gray level values to the total sum of the secondary peak gray level values; the local gray level variance of the main peak region and the secondary peak region is calculated respectively, and the initial weight is updated for the first time to obtain an updated weight;

[0077] The number of gray level values belonging to the corresponding region in the neighborhood of each sampling point in the main peak region and the secondary peak region is counted, and the product of the number of gray level values and a preset neighborhood influence factor is taken as a neighborhood constraint term, and the neighborhood constraint term is used to update the updated weight for the second time to obtain a final weight;

[0078] A candidate threshold value is selected within the gray level value range of the valley point position, and the weighted inter-class variance of the main peak region and the secondary peak region is calculated; the candidate threshold value that makes the weighted inter-class variance maximum is selected as the optimal segmentation threshold value, and the amplitude spectrum is binarized, and an initial mask is obtained after morphological processing.

[0079] In a specific embodiment, the range-Doppler map is converted into a complex form and decomposed into an amplitude spectrum and a phase spectrum. For example, a range-Doppler map can be represented as a two-dimensional matrix, and each element in the matrix is a complex number. By decomposing each complex number into amplitude and phase, the amplitude spectrum and the phase spectrum are obtained.

[0080] The gray level histogram of the amplitude spectrum is counted and smoothed to obtain a continuous gray level distribution curve. For example, the number of occurrences of each gray level value in the amplitude spectrum can be counted, and then a Gaussian filter is used to smooth the histogram.

[0081] On the gray level distribution curve, the positions of the main peak and the secondary peak are identified. The main peak usually corresponds to background noise, while the secondary peak corresponds to target signals. For example, the positions of the main peak and the secondary peak can be determined by finding the local maximum values in the gray level distribution curve. The main peak usually has a larger amplitude and a wider width, while the secondary peak has a smaller amplitude and a narrower width.

[0082] The valley position between the main peak and the secondary peak is extracted, which usually corresponds to the boundary between the target signal and the background noise. For example, the valley position can be determined by finding the local minimum value between the main peak and the secondary peak.

[0083] According to the positions of the main peak and the secondary peak, the main peak region and the secondary peak region are determined respectively. For example, the region left to the main peak can be defined as the main peak region, and the region right to the secondary peak can be defined as the secondary peak region.

[0084] The sum of the main peak gray values in the main peak region and the sum of the secondary peak gray values in the secondary peak region are calculated. Then, the initial weight is determined according to the ratio of the sum of the main peak gray values to the sum of the secondary peak gray values. For example, if the sum of the main peak gray values is 1000 and the sum of the secondary peak gray values is 100, the initial weight can be set to 10.

[0085] The local gray variance of the main peak region and the secondary peak region is calculated respectively, which reflects the dispersion degree of the gray values in the region. Then, the initial weight is updated for the first time by using the local gray variance to obtain the updated weight. For example, the updated weight can be set to the initial weight multiplied by the inverse of the local gray variance.

[0086] The number of gray values belonging to the corresponding region in the neighborhood of each sampling point in the main peak region and the secondary peak region is counted. The product of the number of gray values and the preset neighborhood influence factor is taken as the neighborhood constraint term. For example, if there are 8 pixels belonging to the main peak region in the neighborhood of a sampling point, and the neighborhood influence factor is set to 0.1, then the neighborhood constraint term is 0.8. Then, the updated weight is updated for the second time by using the neighborhood constraint term to obtain the final weight. For example, the final weight can be set to the updated weight multiplied by the neighborhood constraint term.

[0087] A candidate threshold value is selected within the gray value range of the valley position. For example, if the gray value of the valley position is 50, the gray values between 40 and 60 can be selected as the candidate threshold value.

[0088] The weighted inter-class variance of the main peak region and the secondary peak region is calculated. The weighted inter-class variance reflects the difference degree of the gray values between different regions. For example, the inter-class variance can be weighted by using the final weight.

[0089] The candidate threshold value that makes the weighted inter-class variance maximum is selected as the optimal segmentation threshold value. For example, if the weighted inter-class variance is maximum when the candidate threshold value is 45, then the optimal segmentation threshold value is 45.

[0090] The amplitude spectrum is binarized by using the optimal segmentation threshold value, and morphological processing such as erosion and dilation operation is performed to obtain an initial mask. The initial mask is used to distinguish the target signal and the background noise.

[0091] AsFigure 2 The edge retention effect before and after morphological processing in the technical solution is shown. From the processing flow, it can be clearly seen that the original range-Doppler amplitude spectrum is decomposed into amplitude spectrum and phase spectrum, and then the initial binary result is obtained by threshold segmentation. Subsequently, morphological processing is performed on the binary result, including erosion operation using a 3x3 structure element and dilation operation using a 5x5 structure element, effectively removing noise and smoothing the target edge, and finally obtaining a high-quality target mask. From the edge retention index comparison table, the edge integrity, edge accuracy and edge retention rate of the technical solution before morphological processing are 87.5%, 91.2% and 89.3% respectively, and after morphological processing, they are improved to 92.8%, 95.7% and 94.2%, with an improvement rate of 5.5%. In contrast, although the traditional OTSU method has a higher improvement rate of 14.1% after morphological processing, its final edge retention rate is only 77.5%, which is much lower than that of the technical solution. In the comparison curve under different noise levels, it can be seen that the technical solution maintains the highest edge retention rate under all noise conditions, especially under-5dB noise conditions, it can still achieve an edge retention rate of 94.2%, while the traditional OTSU method is only 77.5%. As the noise level decreases, the performance gap between the technical solution and other methods becomes more obvious, proving that the method has stronger edge retention ability in complex noise environment.

[0092] In this embodiment, by adaptively determining the optimal segmentation threshold, the weak target signal can be more effectively separated from the strong background noise, thereby improving the sensitivity of target detection; by introducing a two-level weight updating mechanism and a neighborhood constraint term, the influence of noise can be effectively suppressed, and the noise resistance of the algorithm is enhanced; the optimal segmentation threshold can be automatically determined without human intervention, realizing the automation of target detection and reducing the burden of the operator.

[0093] In an optional implementation, parallel feature extraction is performed on the enhanced feature image to obtain spatial features, time-frequency features and motion features, respectively, and an adaptive feature fusion module is used to assign attention weights to the spatial features, time-frequency features and motion features to obtain a target feature vector, including:

[0094] The enhanced feature image is input into a residual depth separable convolutional network for spatial feature extraction to obtain a depth spatial feature map, and the residual connection operation is performed on the depth spatial feature map and the enhanced feature image to obtain a spatial feature map;

[0095] The enhanced feature image is decomposed by a dual-tree complex wavelet transform to obtain a low-frequency feature subband and a high-frequency feature subband; the low-frequency feature subband is input into a dilated convolution network to extract context features to obtain a context feature map, and an enhanced low-frequency feature map is obtained through channel attention calculation; the high-frequency feature subband is input into a direction-sensitive convolution network to extract texture features to obtain a direction feature map, and the enhanced low-frequency feature map and the direction feature map are reconstructed by a wavelet to obtain a time-frequency feature map;

[0096] Optical flow calculation is performed on adjacent frames of the enhanced feature image to obtain an optical flow feature map, and the enhanced feature image is spliced in the channel dimension to obtain a space-time feature sequence, which is input into a three-dimensional convolution network to extract time sequence features to obtain a motion feature map;

[0097] Based on the spatial feature map, the time-frequency feature map and the motion feature map, normalized cross-correlation is calculated between each other to obtain a feature correlation degree, which is combined to form a feature correlation matrix; nonlinear transformation is performed on the feature correlation matrix to obtain an attention weight coefficient, which is weighted and summed with the spatial feature map, the time-frequency feature map and the motion feature map to obtain a target feature vector.

[0098] In a specific embodiment, the input video image sequence is preprocessed, such as denoising, enhancing contrast, etc., to obtain an enhanced feature image. Taking an RGB image with a resolution of 256x256 as an example, the pixel value range of the preprocessed image is normalized to 0 to 1.

[0099] The enhanced feature image is input into a residual depth separable convolution network to extract spatial features. Depth separable convolution decomposes standard convolution into depth convolution and pointwise convolution, which can reduce computation and improve efficiency. Residual connection helps to alleviate the problem of gradient disappearance and allows training of deeper networks. It is assumed that the residual depth separable convolution network contains 3 residual blocks, each containing 2 depth separable convolution layers, and the output depth spatial feature map has a dimension of 128x128x64. The depth spatial feature map is upsampled and spliced with the original 256x256x3 enhanced feature image, and then convolution operation is performed, and finally the spatial feature map is obtained through residual connection, with a dimension of 128x128x64.

[0100] The enhanced feature image is decomposed by dual-tree complex wavelet transform to obtain a low-frequency feature subband and a high-frequency feature subband. Assuming that two-layer decomposition is adopted, the dimensions of the low-frequency subband and the high-frequency feature subband are both 64×64×3. The low-frequency subband is input into a dilated convolution network to extract context features. Dilated convolution can expand the receptive field without increasing the amount of calculation, and is helpful to capture global context information. Assuming that the dilated convolution network contains 2 layers of dilated convolution, the dilated rates are 2 and 4 respectively, and the dimension of the output context feature map is 64×64×32. Channel attention calculation is performed on the context feature map to obtain an enhanced low-frequency feature map, and the dimension is still 64×64×32. The high-frequency subband is input into a direction-sensitive convolution network to extract texture features to obtain a direction feature map, and the dimension is 64×64×32. The enhanced low-frequency feature map and the direction feature map are subjected to wavelet reconstruction to obtain a time-frequency feature map, and the dimension is 128×128×64.

[0101] Optical flow calculation is performed on adjacent frames of the enhanced feature image to obtain an optical flow feature map, and the dimension is 256×256×2. The optical flow feature map and the enhanced feature image are spliced in the channel dimension to obtain a spatio-temporal feature sequence, and the dimension is 256×256×5. The spatio-temporal feature sequence is input into a three-dimensional convolution network to extract time sequence features. Three-dimensional convolution can capture time dynamic information in a video sequence. Assuming that the three-dimensional convolution network contains 2 layers of three-dimensional convolution, the dimension of the output motion feature map is 128×128×64.

[0102] Based on the spatial feature map, the time-frequency feature map and the motion feature map, normalized cross-correlation is calculated between each two to obtain feature correlation degrees. The feature correlation degrees are combined to form a feature correlation matrix, and the dimension is 3×3. Nonlinear transformation is performed on the feature correlation matrix, for example, a Softmax function, to obtain attention weight coefficients, and the dimension is also 3×3. The spatial feature map, the time-frequency feature map and the motion feature map are respectively weighted and summed with the corresponding attention weight coefficients to finally obtain a target feature vector, and the dimension is 128×128×64.

[0103] In the embodiment, by fusing spatial, time-frequency and motion features and adaptively allocating weights by using an attention mechanism, a more representative target feature vector can be extracted, and the expression ability of the features is effectively improved. The introduction of residual connection and the attention mechanism makes the model more robust to interference factors such as noise and illumination changes, and improves the generalization performance of the model. The use of depth separable convolution reduces the computational complexity of the model, and makes the model easier to train and deploy.

[0104] In an optional implementation, parallel feature extraction is performed on the enhanced feature image to respectively obtain spatial features, time-frequency features and motion features, and an adaptive feature fusion module is used to allocate attention weights to the spatial features, the time-frequency features and the motion features to obtain a target feature vector.

[0105] inputting the enhanced feature image into a residual depth separable convolutional network to obtain a deep spatial feature map, and performing residual connection operation on the deep spatial feature map and the enhanced feature image to obtain a spatial feature map;

[0106] performing double-tree complex wavelet transform decomposition on the enhanced feature image to obtain a low-frequency feature subband and a high-frequency feature subband; inputting the low-frequency feature subband into a dilated convolutional network to obtain a context feature map, and performing channel attention calculation to obtain an enhanced low-frequency feature map; inputting the high-frequency feature subband into a direction-sensitive convolutional network to obtain a direction feature map, and performing wavelet reconstruction on the enhanced low-frequency feature map and the direction feature map to obtain a time-frequency feature map;

[0107] performing optical flow calculation on adjacent frames of the enhanced feature image to obtain an optical flow feature map, splicing the optical flow feature map and the enhanced feature image in a channel dimension to obtain a space-time feature sequence, and inputting the space-time feature sequence into a three-dimensional convolutional network to obtain a motion feature map;

[0108] calculating normalized cross-correlations between the spatial feature map, the time-frequency feature map and the motion feature map to obtain feature correlation degrees, and combining the feature correlation degrees to form a feature correlation matrix; performing nonlinear transformation on the feature correlation matrix to obtain attention weight coefficients, and performing weighted summation on the spatial feature map, the time-frequency feature map and the motion feature map based on the attention weight coefficients to obtain a target feature vector.

[0109] In one specific implementation, a tree filter structure is constructed. This structure contains two groups of filters: a real tree filter group and a complex tree filter group. The real tree filter group contains a real tree low-pass filter and a real tree high-pass filter. The complex tree filter group contains a complex tree low-pass filter and a complex tree high-pass filter, which are a Hilbert transform pair corresponding to the real tree filters. For example, a "db4" wavelet can be selected as the real tree filter, and its corresponding Hilbert transform pair as the complex tree filter.

[0110] First-level decomposition of the image is performed. The enhanced feature image is convolved with the real tree filter group and the complex tree filter group, respectively. Specifically, the enhanced feature image is row-filtered with the real tree filter group to obtain real tree row coefficients. The real tree row coefficients are column-filtered with the complex tree filter group to obtain real tree first-level coefficients. Similarly, the enhanced feature image is row-filtered with the complex tree filter group to obtain complex tree row coefficients. The complex tree row coefficients are column-filtered with the real tree filter group to obtain complex tree first-level coefficients. Assuming that the enhanced feature image is a 256x256 grayscale image, after the first-level decomposition, the real tree and complex tree first-level coefficients are both 128x128 in size.

[0111] Construct complex transform coefficients. Take the real tree first level coefficients as the real part, and the complex tree first level coefficients as the imaginary part, to construct the complex transform coefficients. For example, the value of the real tree first level coefficient at position (10, 10) is 50, and the value of the complex tree first level coefficient at position (10, 10) is 30, then the complex transform coefficient at this position is 50 + 30i.

[0112] Calculate the local direction distribution feature. Calculate the amplitude and phase of each complex transform coefficient. Calculate the local direction angle value according to the phase, which ranges from 0° to 180°. Statistically calculate the distribution histogram of these angle values. For example, divide 0° to 180° into 18 intervals, and count the number of angle values in each interval. Select the angle value corresponding to the peak value in the distribution histogram as the dominant decomposition angle. Suppose that after counting, it is found that the number of angle values in the interval from 45° to 55° is the largest, then select 50° as the dominant decomposition angle.

[0113] Perform directional optimization on the filter. According to the obtained dominant decomposition angle, perform a rotation transformation on the filter responses of the real tree filter bank and the complex tree filter bank to obtain a directional real tree filter bank and a directional complex tree filter bank. The operation of rotation transformation can be realized by performing a specific linear combination on the filter coefficients.

[0114] Perform second level decomposition of the image. Filter and decompose the enhanced feature image using the directional real tree filter bank and the directional complex tree filter bank, similar to the process of the first level decomposition, to obtain directional real tree second level coefficients and directional complex tree second level coefficients.

[0115] Extract low frequency and high frequency feature subbands. Perform frequency domain separation on the directional real tree second level coefficients and the directional complex tree second level coefficients. According to a pre-set frequency threshold, extract the low frequency components to obtain low frequency feature subbands, and extract the high frequency components to obtain high frequency feature subbands. For example, the threshold can be set to half of the frequency range of the image, and the components higher than the threshold are high frequency, and the components lower than the threshold are low frequency.

[0116] In Figure 3The energy distribution of the high-frequency subband in different directions is shown. The energy distribution of the technical solution in each direction is more uniform and has stronger selectivity, especially in the 45° direction, the energy reaches 0.187, and in the 135° direction, the energy reaches 0.179, which is very close to the theoretical value 0.183, with an error of only 2.2% and 2.4%. In contrast, the energy of the traditional dual-tree complex wavelet in the 45° and 135° directions is 0.153 and 0.157, respectively, with a deviation of 16.4% and 14.2% from the theoretical value. In the 30°, 60°, 120° and 150° directions, the energy values of the technical solution are 0.123, 0.125, 0.124 and 0.122, respectively, with a standard deviation of only 0.0013, indicating that it has consistent decomposition ability in these directions; while the energy values of the traditional dual-tree complex wavelet in these directions are 0.093, 0.112, 0.109 and 0.097, respectively, with a standard deviation of 0.0091, which is 7 times that of the technical solution, indicating that its direction selectivity is poor. The technical solution successfully improves the uniform response ability of the transform in each direction through the calculation of the advantage decomposition angle and the directional optimization of the filter, so that the high-frequency subband can more accurately represent the texture and edge features in different directions of the image.

[0117] In this embodiment, by optimizing the filter, the local direction information of the image can be better captured, thereby improving the expression ability of the features and being more conducive to subsequent image processing tasks; the dual-tree complex wavelet transform has good direction selectivity and phase information, which can effectively suppress the influence of noise on feature extraction and improve the robustness of the algorithm; by separating in the frequency domain through the preset frequency threshold, the data amount can be effectively reduced, the computational complexity can be reduced, and the processing efficiency can be improved.

[0118] In an optional implementation, a recurrent neural tensor network is used to iteratively decompose the target feature vector to obtain associated features, a non-local neural network is used to construct a feature association graph and combine a deformable convolutional network to extract key spatiotemporal features, the key spatiotemporal features are multi-scale decomposed and an adversarial feature alignment network is used to eliminate distribution differences, and a coupling weight routing based on a capsule network is used to complete personnel target classification, including:

[0119] A recurrent neural tensor network is constructed, the target feature vector is mapped to a tensor space through a tensor and a bias vector, and iteratively recursively decomposed through a nonlinear activation function, and the iteration is terminated when the feature change amount is less than a preset change amount threshold to obtain associated features;

[0120] The associated features are input into a non-local neural network to construct a non-local feature association graph; the associated features are weighted and aggregated based on the edge weights in the non-local feature association graph to obtain weighted features, the weighted features are input into a deformable convolution network, the offset of the sampling position of the convolution kernel is calculated, the convolution kernel position is adjusted according to the offset to perform convolution operation, and key spatio-temporal features are obtained;

[0121] The key spatio-temporal features are subjected to multi-scale feature decomposition, an adversarial feature alignment network comprising a feature extractor and a feature discriminator is constructed, the scene classification of the source domain feature map and the target domain feature map is performed by the feature discriminator to obtain a discrimination loss, the feature extractor is subjected to iterative adversarial training based on the discrimination loss, and aligned features with consistent feature distribution are obtained.

[0122] The aligned features are input into a low-level capsule of a capsule network, the coupling coefficient with a high-level capsule is calculated and normalized to obtain a routing weight, the output features of the low-level capsule are weighted and summed according to the routing weight, the result of the weighted sum is mapped through a nonlinear compression function, the output features of the high-level capsule are obtained, and the target classification is determined.

[0123] In a specific embodiment, target video data is obtained, and video frames are preprocessed, such as image scaling, normalization, etc. Taking a surveillance video containing multiple personnel targets as an example, the video frames are scaled to 256x256 pixels in size, and pixel value normalization processing is performed. A pre-trained convolutional neural network (such as ResNet-50) is used to extract target feature vectors of each frame of image, and it is assumed that the dimension of the target feature vector is 2048.

[0124] A recurrent neural tensor network is constructed to decompose the target feature vector into associated features. Specifically, the 2048-dimensional target feature vector is mapped to a 1024-dimensional tensor space through a learnable tensor and bias vector. In the tensor space, the features are iteratively and recursively decomposed using a ReLU nonlinear activation function. In each iteration, the current obtained feature is combined with the feature of the last iteration, such as adding them together. By comparing the difference (such as Euclidean distance) between the current iteration feature and the last iteration feature, it is determined whether the feature converges. If the difference is less than a preset threshold (such as 0.001), the iteration is terminated, and the feature obtained in the last iteration is taken as the associated feature, and the dimension is still 1024.

[0125] The associated features are input into a non-local neural network to construct a non-local feature association graph. The nodes of the graph represent the associated features, and the edges represent the relationships between the features. The similarity (e.g., cosine similarity) between the features of the nodes is calculated as the weight of the edge. Assuming there are two associated feature vectors a and b, and their cosine similarity is 0.8, then 0.8 is taken as the weight of the edge between the two nodes. The associated features are weighted and aggregated based on the edge weights in the non-local feature association graph to obtain weighted features. For example, the weights of all edges connected to a certain node are normalized, and then the features of the neighbor nodes are weighted and summed according to the normalized weights to obtain the weighted features of the node, still with a dimension of 1024.

[0126] The weighted features are input into a deformable convolution network. Taking a 3x3 convolution kernel as an example, the offset of each convolution kernel sampling position is calculated. For example, the offset of each position is learned through a small convolution network, and the offset range is assumed to be between -1 and 1. According to the calculated offset, the position of the convolution kernel is adjusted to obtain an offset convolution kernel. Then, the offset convolution kernel is used to perform convolution operation on the weighted features to obtain key spatio-temporal features, with a dimension of 512.

[0127] The key spatio-temporal features are decomposed into different scales, for example, using convolution operations with different size of convolution kernels to obtain multi-scale feature maps. For example, 3x3, 5x5 and 7x7 convolution kernels are used for convolution to obtain three feature maps of different scales. An adversarial feature alignment network is constructed, which includes a feature extractor and a feature discriminator. The feature extractor takes the multi-scale feature maps as input and outputs aligned features. The feature discriminator takes the source domain feature map and the target domain feature map as input to determine which domain the feature comes from and outputs a discrimination loss. The feature extractor and the feature discriminator are iteratively trained in an adversarial manner. The feature extractor tries to generate features that can confuse the discriminator, while the discriminator tries to distinguish which domain the feature comes from. When the discrimination loss is less than a pre-set threshold (e.g., 0.1), the iteration is terminated, and the aligned features with consistent feature distribution are obtained, still with a dimension of 512.

[0128] The aligned features are input into the low-level capsules in the capsule network. Assuming there are 10 low-level capsules, each outputting an 8-dimensional feature vector. The inner product of the low-level capsule output feature and the high-level capsule feature is calculated to obtain the coupling coefficient. Assuming there are 2 high-level capsules, corresponding to 2 personnel categories. The coupling coefficients are normalized to obtain routing weights. The low-level capsule output features are weighted and summed according to the routing weights, and the result of the weighted sum is mapped through a non-linear compression function (e.g., squash function) to obtain the high-level capsule output feature, with a dimension of 16 (2 capsules, each 8-dimensional). The length of the high-level capsule output feature is used to determine the target classification. For example, the category corresponding to the capsule with the maximum length is the final classification result.

[0129] In the embodiment, more discriminative correlation features are extracted by decomposing the target feature vector through the recurrent neural tensor network; key spatio-temporal information is captured through the non-local neural network and the deformable convolution; the distribution difference in different scenes is eliminated through the alignment of the adversarial features; finally, more accurate personnel target classification is realized through the capsule network; through the alignment of the adversarial features, the model has stronger adaptability to data in different scenes, and the generalization ability and robustness of the model are improved; through the recurrent neural tensor network and the non-local neural network, key features are effectively extracted, the computational complexity is reduced, and the running efficiency of the model is improved.

[0130] In an optional implementation, the correlation features are input into the non-local neural network to construct a non-local feature correlation graph.

[0131] Through the feature transformation layer of the non-local neural network, the correlation features are subjected to discrete cosine transformation to obtain frequency domain features, a multi-band power spectrum density matrix is calculated and subjected to eigenvalue decomposition to obtain feature frequency spectrum bases and spectrum coefficients, and an orthogonal projection matrix is constructed based on the feature frequency spectrum bases to project the correlation features to a frequency spectrum space to obtain frequency spectrum projection features.

[0132] In the similarity calculation layer of the non-local neural network, a frequency spectrum similarity matrix is calculated based on the frequency spectrum projection features, an initial frequency spectrum graph is constructed and a Laplacian matrix is calculated, the optimal subgraph number is determined through eigenvalue decomposition, and the initial frequency spectrum graph is divided into multiple subgraphs based on the spectral clustering algorithm and the optimal subgraph number.

[0133] In the graph structure generation layer of the non-local neural network, the degree centrality and the feature vector centrality of each subgraph node are calculated, the local centrality index is determined, the subgraph key nodes are selected by taking the local centrality index greater than a preset local centrality threshold value, the hierarchical feature correlation graph is constructed by taking the similarity of the subgraph key nodes as the edge weight between different subgraphs.

[0134] In the dynamic update layer of the non-local neural network, the feature vectors of the subgraph nodes in multiple continuous historical update periods are extracted, the Euclidean distance sequence and the coefficient of variation between the feature vectors of adjacent periods are calculated to obtain a stability index; when the stability index is less than a preset stability threshold value, node splitting is performed; when the similarity of adjacent subgraph nodes is greater than a preset similarity threshold value, a fusion node is constructed by weighted averaging the feature vectors; based on the node splitting and the fusion node, the edge weight is recalculated to obtain the non-local feature correlation graph.

[0135] In a specific implementation, the correlation features are subjected to frequency domain conversion and spectrum analysis to construct a spectrum graph. Specifically, the correlation features are subjected to discrete cosine transform by a feature transformation layer of the non-local neural network to obtain frequency domain features. The frequency domain features are divided into multiple frequency bands (for example, the frequency domain range is evenly divided into 16 frequency bands), and the root mean square value of each frequency band is calculated to form a power spectral density matrix. For example, a sample containing 32 correlation features is subjected to discrete cosine transform to obtain 32 frequency domain features, and after being divided into 16 frequency bands, a 16x16 power spectral density matrix is obtained. The power spectral density matrix is subjected to eigenvalue decomposition to obtain eigen-spectrum bases and spectrum coefficients. An orthogonal projection matrix is constructed based on the eigen-spectrum bases. The original correlation features are projected into the spectrum space to obtain spectrum projection features.

[0136] The spectrum graph is subjected to subgraph division based on spectral clustering to determine subgraph nodes. In the similarity calculation layer of the non-local neural network, the spectrum projection features are taken as query features and key features to calculate cosine similarity and obtain a spectrum similarity matrix. The matrix is taken as an adjacency matrix to construct an initial spectrum graph and calculate a Laplacian matrix. The Laplacian matrix is subjected to eigenvalue decomposition to obtain an eigenvalue sequence. Assuming that the eigenvalue sequence is [0, 0.1, 0.2, 0.5, 1.2, 1.3, 2.5,...], the eigenvalue sequence is observed, and it is found that there is a large gap from 0.5 to 1.2, so the optimal subgraph number is determined to be 4. The initial spectrum graph is divided into four subgraphs based on the optimal subgraph number by using the spectral clustering algorithm.

[0137] A hierarchical feature correlation graph is constructed using the centrality indicators of the subgraph nodes. In the graph structure generation layer of the non-local neural network, the degree centrality and feature vector centrality of each subgraph node representing a subgraph are calculated to determine a local centrality indicator. Assuming that the preset local centrality threshold is 0.8, the degree centrality of a subgraph node is 0.9, and the feature vector centrality is 0.85, then the local centrality indicator of the node is (0.9+0.85) / 2=0.875, which is greater than the preset threshold, so the node is selected as a key node of the subgraph. A hierarchical feature correlation graph is constructed by taking the spectrum similarity as the edge weight within the subgraph and taking the key node similarity as the edge weight between different subgraphs. For example, the key node of the subgraph Figure 1 has a key node similarity of 0.7 with the key node of the subgraph Figure 2 , so the edge weight between the two subgraphs is 0.7.

[0138] The node stability index is introduced for dynamic updating to obtain a non-local feature correlation graph. In the dynamic updating layer of the non-local neural network, after the hierarchical feature correlation graph is completed in a preset iteration period (for example, 10 iterations per period), the feature vectors of the target subgraph nodes in multiple consecutive historical updating periods (for example, the last 5 periods) are extracted. The Euclidean distance between the feature vectors of adjacent periods is calculated to obtain a distance sequence. The coefficient of variation of the distance sequence is calculated. The reciprocal of the coefficient of variation is taken as the stability index of the target subgraph node. Assuming that the preset stability threshold is 0.5, and the stability index of a certain node is 0.3, which is less than the threshold, the node is split into multiple split nodes. Assuming that the preset similarity threshold is 0.9, if the similarity between two adjacent target subgraph nodes is greater than 0.9, the feature vectors corresponding to the two nodes are weighted and averaged to obtain a new feature vector, and a fusion node is constructed. The edge weight is recalculated based on the split nodes and the fusion node to obtain the final non-local feature correlation graph.

[0139] Figure 4 The four methods are shown to have varying trends in computational efficiency as the size of the data set increases. As can be clearly seen from the chart, when the data volume increases, the inference time of the traditional graph convolution network, the self-attention mechanism and the standard non-local neural network increases super-linearly, while the present technical solution shows approximately linear growth, and the computational efficiency advantage is obvious. When the size of the data set reaches 50K, the average inference time of the present technical solution is only 38.5ms, while the traditional graph convolution network is 157.6ms, the self-attention mechanism is 258.2ms, and the standard non-local neural network is 275.3ms. This means that the speed of the present technical solution on a large-scale data set is more than 7 times that of the standard non-local neural network. This advantage mainly comes from the spectral analysis and subgraph division strategy in the present technical solution, which converts the high-dimensional feature space into a more compact spectral space, and then divides the subgraph through spectral clustering, greatly reducing the computational complexity, especially when dealing with large-scale data.

[0140] In the present embodiment, through frequency domain conversion and spectral analysis, global information and local details in the correlation features can be better captured, thereby improving the feature expression capability; the dynamic updating mechanism can be adjusted according to the stability of the nodes, so that the generated feature correlation graph is more stable, thereby enhancing the robustness of the model; the hierarchical graph structure and the spectral clustering algorithm can effectively reduce the computational complexity and improve the efficiency of the model.

[0141] Figure 5 The structure schematic diagram of the millimeter wave radar personnel perception system based on time-frequency domain and deep CNN in the embodiment of the present application is shown in Figure 5 The system comprises:

[0142] The first unit is configured to collect a personnel target echo signal by using a millimeter wave radar, and preprocess the personnel target echo signal to obtain a preprocessed signal; perform two-dimensional fast Fourier transform on the preprocessed signal to obtain a range-Doppler spectrogram; perform clutter suppression on the range-Doppler spectrogram by using a recursive threshold segmentation algorithm to obtain a target spectrogram; convert the target spectrogram into a three-dimensional feature tensor; and perform compression and enhancement processing on the three-dimensional feature tensor to obtain an enhanced feature image.

[0143] The second unit is configured to perform parallel feature extraction on the enhanced feature image to obtain spatial features, time-frequency features and motion features, respectively; and perform attention weight distribution on the spatial features, the time-frequency features and the motion features by using an adaptive feature fusion module to obtain a target feature vector.

[0144] The third unit is configured to perform iterative decomposition on the target feature vector by using a recurrent neural tensor network to obtain associated features; construct a feature association graph by using a non-local neural network, and extract key spatiotemporal features by using a deformable convolutional network; perform multi-scale decomposition on the key spatiotemporal features, and eliminate distribution differences by using an adversarial feature alignment network; and complete personnel target classification based on a coupling weight routing of a capsule network.

[0145] A third aspect of the embodiment of the present application,

[0146] An electronic device is provided.

[0147] A processor;

[0148] A memory for storing processor-executable instructions;

[0149] The processor is configured to invoke the instructions stored in the memory to perform the method described above.

[0150] A fourth aspect of the embodiment of the present application,

[0151] A computer-readable storage medium is provided, which stores computer program instructions. The computer program instructions are executed by a processor to implement the method described above.

[0152] The present application can be a method, device, system and / or computer program product. The computer program product can include a computer readable storage medium having computer readable program instructions stored therein, which are used to perform various aspects of the present application.

[0153] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement for part or all of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for personnel perception using millimeter-wave radar based on time-frequency domain and deep CNN, characterized in that, include: The echo signal of personnel target is collected by millimeter-wave radar and preprocessed to obtain a preprocessed signal. A two-dimensional fast Fourier transform is performed on the preprocessed signal to obtain a range-Doppler spectrum. A recursive threshold segmentation algorithm is used to suppress clutter and obtain a target spectrum. The target spectrum is converted into a three-dimensional feature tensor and then compressed and enhanced to obtain an enhanced feature image. Parallel feature extraction is performed on the enhanced feature image to obtain spatial features, time-frequency features and motion features respectively. An adaptive feature fusion module is used to assign attention weights to the spatial features, time-frequency features and motion features to obtain the target feature vector. A recurrent neural tensor network is used to iteratively decompose the target feature vector to obtain associated features. A feature association graph is constructed through a nonlocal neural network and combined with a deformable convolutional network to extract key spatiotemporal features. The key spatiotemporal features are decomposed at multiple scales, and an adversarial feature alignment network is used to eliminate distribution differences. Personnel target classification is completed based on coupled weight routing of capsule networks, including: A recursive neural tensor network is constructed to map the target feature vector to the tensor space through tensors and bias vectors, and then iteratively decomposes it using a nonlinear activation function. The iteration terminates when the feature change is less than a preset change threshold to obtain the associated features. The associated features are input into a nonlocal neural network to construct a nonlocal feature association graph; the associated features are weighted and aggregated based on the edge weights in the nonlocal feature association graph to obtain weighted features; the weighted features are input into a deformable convolutional network, the offset of the sampling position of the convolution kernel is calculated, the position of the convolution kernel is adjusted according to the offset, and convolution operation is performed to obtain key spatiotemporal features; Multi-scale feature decomposition is performed on key spatiotemporal features to construct an adversarial feature alignment network containing a feature extractor and a feature discriminator. The feature discriminator performs scene classification on the source domain feature map and the target domain feature map to obtain the discriminant loss. The feature extractor performs iterative adversarial training based on the discriminant loss to obtain aligned features with consistent feature distribution. The alignment features are input into the lower-level capsules of the capsule network. The coupling coefficients with the higher-level capsules are calculated and normalized to obtain the routing weights. The output features of the lower-level capsules are weighted and summed according to the routing weights. The weighted summation result is mapped through a nonlinear compression function to obtain the output features of the higher-level capsules and determine the target classification.

2. The method according to claim 1, characterized in that, Millimeter-wave radar is used to collect echo signals from personnel targets, which are then preprocessed to obtain a preprocessed signal. A two-dimensional fast Fourier transform is performed on the preprocessed signal to obtain a range-Doppler spectrum. A recursive threshold segmentation algorithm is used to suppress clutter and obtain a target spectrum. The target spectrum is converted into a three-dimensional feature tensor, which is then compressed and enhanced to obtain an enhanced feature image, including: Millimeter-wave radar is used to collect echo signals of personnel targets. The DC component is eliminated by zero-mean normalization, and the preprocessed signal is obtained by windowing using an adaptive window function. The preprocessed signal is then subjected to a two-dimensional fast Fourier transform to obtain a complex-valued matrix, which is then converted into a range-Doppler spectrum. The range-Doppler spectrum is decomposed into an amplitude spectrum and a phase spectrum. The amplitude spectrum is initially segmented using the bimodal maximum inter-class variance method to obtain an initial mask. Based on the initial mask, the mean of the amplitude spectrum and the standard deviation of the phase spectrum are weighted and fused using a fusion weighting coefficient to obtain a new threshold. When the difference between the new threshold and the threshold of the previous iteration is less than a preset difference threshold, the iteration stops to obtain the final threshold. The target spectrum is obtained by segmenting the range-Doppler spectrum using the final threshold. A multi-layer wavelet packet decomposition tree is constructed. In each layer of the multi-layer wavelet packet decomposition tree, the node with the minimum information entropy is selected for decomposition to obtain the optimal basis function set. The optimal basis function set is used to perform time-frequency analysis on the target spectrum to obtain the three-dimensional feature tensor. After dimensionality reduction, compression and equalization processing, the enhanced feature image is obtained.

3. The method according to claim 2, characterized in that, The range-Doppler spectrum is decomposed into an amplitude spectrum and a phase spectrum. The initial mask is obtained by performing a bimodal maximum inter-class variance method on the amplitude spectrum, including: The range-Doppler spectrum is decomposed into a complex number to obtain the amplitude spectrum and phase spectrum. The gray-level histogram of the amplitude spectrum is statistically analyzed and smoothed to obtain the gray-level distribution curve. The positions of the main peak and the secondary peak are determined in the gray-level distribution curve. The positions of the valley points between the main peak and the secondary peak are extracted to determine the main peak region and the secondary peak region respectively. Calculate the sum of gray values ​​of the main peak within the main peak region and the sum of gray values ​​of the secondary peak within the secondary peak region. Determine the initial weights based on the ratio of the sum of gray values ​​of the main peak to the sum of gray values ​​of the secondary peaks. Calculate the local gray variances of the main peak region and the secondary peak region respectively, and update the initial weights to obtain the updated weights. The number of gray values ​​belonging to the corresponding region in the neighborhood of each sampling point in the main peak region and the secondary peak region is counted. The product of the number of gray values ​​and the preset neighborhood influence factor is used as the neighborhood constraint term. The neighborhood constraint term is used to update the update weight a second time to obtain the final weight. Candidate thresholds are selected within the gray value range of the valley point, and the weighted inter-class variance of the main peak region and the secondary peak region is calculated. The candidate threshold that maximizes the weighted inter-class variance is selected as the optimal segmentation threshold, and the amplitude spectrum is binarized. The initial mask is obtained after morphological processing.

4. The method according to claim 1, characterized in that, Parallel feature extraction is performed on the enhanced feature image to obtain spatial features, time-frequency features, and motion features. An adaptive feature fusion module is then used to assign attention weights to the spatial features, time-frequency features, and motion features to obtain the target feature vector, which includes: The enhanced feature image is input into the residual depthwise separable convolutional network to extract spatial features and obtain a depth spatial feature map. The enhanced feature image is then subjected to residual connection operation to obtain a spatial feature map. The enhanced feature image is decomposed by dual-tree complex wavelet transform to obtain low-frequency feature sub-bands and high-frequency feature sub-bands; the low-frequency feature sub-bands are input into a dilated convolutional network for context feature extraction to obtain a context feature map, and then channel attention is used to calculate the enhanced low-frequency feature map; the high-frequency feature sub-bands are input into a direction-sensitive convolutional network for texture feature extraction to obtain a direction feature map, and the enhanced low-frequency feature map and the direction feature map are reconstructed by wavelet to obtain a time-frequency feature map; Optical flow is calculated on adjacent frames of the enhanced feature image to obtain optical flow feature maps, which are then concatenated with the enhanced feature image in the channel dimension to obtain a spatiotemporal feature sequence. This sequence is then input into a three-dimensional convolutional network for temporal feature extraction to obtain a motion feature map. Based on the spatial feature map, time-frequency feature map, and motion feature map, the normalized cross-correlation between each pair is calculated to obtain the feature correlation degree, which is then combined to form the feature correlation matrix. The feature correlation matrix is ​​then subjected to a nonlinear transformation to obtain the attention weight coefficient, which is then weighted and summed with the spatial feature map, time-frequency feature map, and motion feature map to obtain the target feature vector.

5. The method according to claim 4, characterized in that, The enhanced feature image is decomposed using a dual-tree complex wavelet transform to obtain low-frequency and high-frequency feature subbands, including: Construct a tree-like filter structure that includes a real tree filter bank and a complex tree filter bank. The real tree filter bank includes a real tree low-pass filter and a real tree high-pass filter, and the complex tree filter bank includes a complex tree low-pass filter and a complex tree high-pass filter. The complex tree filter bank is a Hilbert transform pair of the real tree filter bank. The enhanced feature image is filtered in the row direction through a real tree filter bank to obtain the real tree row coefficients. The real tree row coefficients are then filtered in the column direction through a complex tree filter bank to obtain the real tree first-level coefficients. The enhanced feature image is filtered in the row direction through a complex tree filter bank to obtain the complex tree row coefficients. The complex tree row coefficients are then filtered in the column direction through a real tree filter bank to obtain the complex tree first-level coefficients. The first-order coefficients of the real tree and the first-order coefficients of the complex tree are used as the real and imaginary parts to construct complex transformation coefficients, respectively. The magnitude and phase of the complex transformation coefficients are calculated, and the local direction angle value is calculated based on the phase. The angle distribution histogram is statistically analyzed, and the angle corresponding to the peak value is selected as the dominant decomposition angle. The filter bank response is rotated and transformed based on the dominant decomposition angle to obtain the directional filter bank. The enhanced feature image is filtered and decomposed using a directional filter bank to obtain the second-level coefficients; the second-level coefficients are then separated in the frequency domain, and low-frequency feature sub-bands and high-frequency feature sub-bands are extracted according to a preset frequency threshold.

6. The method according to claim 1, characterized in that, The associated features are input into a nonlocal neural network to construct a nonlocal feature association graph, including: By using the feature transformation layer of a nonlocal neural network, discrete cosine transform is performed on the associated features to obtain frequency domain features. The multi-band power spectral density matrix is ​​calculated and eigenvalue decomposition is performed to obtain the feature spectral basis and spectral coefficients. Based on the feature spectral basis, an orthogonal projection matrix is ​​constructed to project the associated features onto the spectral space to obtain spectral projection features. In the similarity calculation layer of the nonlocal neural network, the spectral similarity matrix is ​​calculated using spectral projection features, an initial spectrogram is constructed and the Laplacian matrix is ​​calculated, the optimal number of subgraphs is determined by eigenvalue decomposition, and the initial spectrogram is divided into multiple subgraphs based on the optimal number of subgraphs using a spectral clustering algorithm. In the graph structure generation layer of the nonlocal neural network, the degree centrality and eigenvector centrality of each subgraph node are calculated, the local centrality index is determined, and the subgraph nodes with local centrality indices greater than the preset local centrality threshold are selected as key nodes of the subgraph. The similarity of the key nodes of the subgraphs is used as the edge weight between different subgraphs to construct a hierarchical feature association graph. In the dynamic update layer of the nonlocal neural network, feature vectors of subgraph nodes are extracted in multiple consecutive historical update cycles. The Euclidean distance sequence and coefficient of variation between feature vectors in adjacent cycles are calculated to obtain a stability index. When the stability index is less than a preset stability threshold, node splitting is performed. When the similarity between adjacent subgraph nodes is greater than a preset similarity threshold, the feature vectors are weighted and averaged to construct a fused node. Based on node splitting and fused nodes, the edge weights are recalculated to obtain a nonlocal feature association graph.

7. A millimeter-wave radar personnel perception system based on time-frequency domain and deep CNN, used to implement the method of any one of claims 1-6, characterized in that, include: The first unit is used to collect personnel target echo signals using millimeter-wave radar and preprocess them to obtain a preprocessed signal. A two-dimensional fast Fourier transform is performed on the preprocessed signal to obtain a range-Doppler spectrum. A recursive threshold segmentation algorithm is used to suppress clutter and obtain a target spectrum. The target spectrum is converted into a three-dimensional feature tensor and then compressed and enhanced to obtain an enhanced feature image. The second unit is used to perform parallel feature extraction on the enhanced feature image to obtain spatial features, time-frequency features and motion features respectively. An adaptive feature fusion module is used to assign attention weights to the spatial features, time-frequency features and motion features to obtain the target feature vector. The third unit is used to iteratively decompose the target feature vector using a recurrent neural tensor network to obtain associated features. It constructs a feature association graph through a nonlocal neural network and extracts key spatiotemporal features by combining a deformable convolutional network. It performs multi-scale decomposition on the key spatiotemporal features and uses an adversarial feature alignment network to eliminate distribution differences. It completes personnel target classification based on the coupled weight routing of the capsule network.

8. An electronic device, characterized in that, include: processor; Memory used to store processor-executable instructions; The processor is configured to invoke instructions stored in the memory to execute the method according to any one of claims 1 to 6.

9. A computer-readable storage medium having computer program instructions stored thereon, characterized in that, When the computer program instructions are executed by the processor, they implement the method described in any one of claims 1 to 6.

Images