Domain-adaptive neural architecture search method for cross-room non-line-of-sight acoustic signal recognition

Abstract

The application discloses a domain adaptive neural architecture search method for cross-room non-line-of-sight (NLOS) acoustic signal recognition, which comprises the processes of acoustic signal processing, feature extraction, adversarial domain adaptation, neural architecture search and the like, wherein the acoustic signal processing part modulates a transmitting signal into a linear sweep signal, converts a received signal into a frequency spectrum matrix and carries out denoising, segmentation and the like; the feature extraction part extracts structural features in the frequency spectrum matrix by using a deep convolutional neural network model; the adversarial domain adaptation improves the performance of the model on unknown data sets by means of a gradient flip layer and adversarial training; and finally, the neural architecture search technology is used to adjust and optimize the specific network structure. The application can realize high-precision NLOS / LOS signal recognition, effectively improves the recognition rate and recognition accuracy, and performs well in an unmarked unfamiliar environment, which is favorable for reducing the cost of a deep learning model and improving the generalization of the model.

Description

Technical Field

[0001] This invention relates to the field of indoor positioning technology, and in particular to a domain-adaptive neural architecture search method for cross-room non-line-of-sight acoustic signal recognition. Background Technology

[0002] To meet the growing demand for high-precision positioning of people and assets, location-based services (LBS) have flourished with the advancement of Internet of Things (IoT) technology, finding widespread applications in disaster relief, object tracking, and vehicle and pedestrian navigation. Outdoors, Global Navigation Satellite Systems (GNSS) provide reliable navigation solutions. However, due to the inability of GNSS signals to penetrate walls and shielding effects, indoor positioning typically requires additional signal media. Accurate positioning in indoor and underground scenarios without satellite signals has become a pressing issue. Various wireless indoor positioning services, such as Bluetooth, Wi-Fi, ZigBee, Near Field Communication (NFC), and UWB, are emerging. Compared to other indoor positioning technologies, acoustic positioning based on Time of Arrival (ToA) / Time Difference of Arrival (TDoA) can typically achieve positioning accuracy from decimeters to centimeters. More importantly, acoustic signals have inherently low synchronization costs and are highly compatible with current edge devices such as smartphones. Users can utilize accurate and fast indoor positioning services without installing additional hardware. Compared to other indoor positioning technologies, acoustic signals offer strong compatibility and lower deployment costs, making them considered to have significant potential for building accurate indoor positioning systems (IPS).

[0003] Indoor acoustic positioning faces many challenges, among which non-line-of-sight (NLOS) is generally considered the most significant factor affecting the degradation of acoustic signal ranging performance. NLOS occurs when the line of sight between two communicating points is obstructed, preventing them from seeing each other. Conversely, line-of-sight (LOS) occurs when the wireless signal propagates unobstructed between the transmitter and receiver. Detecting, filtering, or correcting NLOS signals has become crucial for IPS (Instrument Probe). The accuracy of the detection algorithm directly impacts every step of the process. NLOS refers to the situation where obstacles are placed between the transmitter and receiver. Acoustic signals may bounce off walls multiple times before reaching the surrounding area, leading to delays and potential loss and degradation. Acoustic signals typically have frequencies of approximately 20Hz-20kHz and wavelengths of approximately 17mm-17m. Significant attenuation occurs when blocked by obstacles, undoubtedly increasing the difficulty of identifying LOS / NLOS signals. Currently, there are four methods for identifying LOS / NLOS signals:

[0004] The first approach is based on statistics. It constructs a Gaussian distribution with zero mean by recording previous LOS measurements as priors, compares the current data to this distribution, and discards or corrects for LOS signals if unreasonable discrepancies exist. This prior measurement can provide the target's velocity, direction, range, etc. Measurement results can be obtained from the signal itself or through external sensors such as inertial measurement units (IMUs), which measure the object's three-axis attitude angles and accelerations. Hypothesis testing can use statistical parameters such as variance and mean. However, this method struggles to define thresholds for LOS and NLOS signals, and the recognition accuracy is significantly affected by hyperparameters.

[0005] The second approach is based on indoor maps and context. It estimates the approximate indoor location based on positioning and matches it with an indoor map to infer whether signal obstruction exists. However, this method requires additional precise indoor maps and pre-positioning, which wastes computing power and storage to some extent.

[0006] The third approach is a data-driven traditional machine learning method. Features such as channel impulse response (CIR), channel state information (CSI), propagation delay, energy intensity, kurtosis, and interpeak delay are frequently used in nonparametric machine learning, such as support vector machines (SVM), multilayer sensing (MLP), and decision trees, to classify NLOS / LOS signals. However, this method requires manual feature extraction. Since signal propagation path loss models are affected by many factors, manually selected features may not be sufficient to accurately distinguish between line-of-sight (LOS) and non-line-of-sight (LOS) signals.

[0007] The fourth approach is based on data-driven deep learning. In recent years, much work has focused on introducing deep neural networks to identify NLOS signals. Jiang et al. (see IEEE Communications Letters, 24(10):2226–2230, 2020) proposed a serial network for NLOS / LOS classification in UWB CIR measurements. Wang et al. (see 22nd international conference on indoor positioning and indoornavigation (IPIN), Beijing. 2022: 5-7) used CNN to automatically extract features of the spectral matrix of two-dimensional class images and used Bi-LSTM for classification. Wei et al. (see Measurement, 195: 111191, 2022) proposed a parallel network composed of CNN and LSTM to process spectrograms and audio respectively, and used the time-frequency information of UBW to identify NLOS signals. Zhao et al. (see Applied Sciences, 12(13): 6484, 2022) proposed using a generative adversarial network (GAN) to enhance LOS / NLOS data. However, this method mostly uses only the temporal information of the signal as input to the deep learning model, and the model structure is relatively simple, resulting in low recognition accuracy. Its generalization ability is poor, it has high requirements for the dataset, and it is highly sensitive to environmental characteristics. This method requires a large amount of manually labeled data to train the network model, and the performance of the trained model will significantly decrease on unfamiliar datasets (such as data collected from a room with a different structure). The design and adjustment of the network model structure in this method rely on engineers' deep learning knowledge and experience, and require a significant amount of time for trial and error. Summary of the Invention

[0008] The purpose of this invention is to address the shortcomings of existing technologies by providing a domain-adaptive neural architecture search method for cross-room non-line-of-sight (NLOS) acoustic signal recognition. This invention achieves high-precision NLOS / LOS signal recognition and performs well in unlabeled, unfamiliar environments.

[0009] The objective of this invention is achieved through the following technical solution: A first aspect of this invention provides a domain-adaptive neural architecture search method for cross-room non-line-of-sight acoustic signal recognition, comprising the following steps:

[0010] (1) Design an audio signal. The speakers in the room periodically send the designed audio signal to the microphone, and the microphone collects the audio signal in the room.

[0011] (2) The acquired audio signals are preprocessed to obtain the preprocessed spectrum matrix;

[0012] (3) Construct an adversarial domain adaptation network model, which includes a feature extractor, a label predictor and a domain classifier. Input the preprocessed spectrum matrix into the feature extractor to obtain the feature vector; input the feature vector into the label predictor and the domain classifier respectively to obtain the non-line-of-sight signal recognition result and its corresponding domain recognition result.

[0013] (4) Freeze the structural parameters of the feature extractor and train only its weight parameters; use the neural architecture search method to perform a structure search in the given search space, periodically train the label classifier and the domain classifier, input the preprocessed spectrum matrix into the adversarial domain adaptation network model, obtain the predicted non-line-of-sight signal recognition results and their corresponding domain recognition results, calculate the loss function of the adversarial domain adaptation network model based on the predicted non-line-of-sight signal recognition results and their corresponding domain recognition results and the signal labels, and adjust the parameters of the label classifier and the domain classifier based on the loss function of the adversarial domain adaptation network model to obtain the optimal network structure and parameters of the label classifier and the domain classifier.

[0014] Furthermore, the audio signal designed in step (1) is a linear frequency sweep signal, expressed as:

[0015]

[0016]

[0017] Where s(t) is a single linear sweep signal, j represents the imaginary unit, f0 and u0 are the initial frequency and modulation rate, respectively, the length of the linear sweep signal is t, t(τ) represents the periodic overall linear sweep signal, τ represents the propagation delay from different paths, T is the transmission period of the linear sweep signal, the time interval (Tt) is the silence time, ε(·) is the step function, and i represents the i-th signal.

[0018] Furthermore, the audio signal captured by the microphone is represented as follows:

[0019]

[0020] Where res(τ) represents the audio signal in the room captured by the microphone, n r and n d n represents the number of reflected and diffracted signals, respectively. l =0,1 indicates whether it is a line-of-sight signal, n l =1 indicates that the audio signal is a line-of-sight signal, n l=0 indicates that the audio signal is a non-line-of-sight signal, β represents the fading coefficient of different paths, the subscripts l, r, and d represent the direct path, reflection path, and diffraction path, respectively, the Black Man window w· is used to eliminate slight multidirectional fluctuations, and n· represents noise.

[0021] Furthermore, step (2) includes the following sub-steps:

[0022] (2.1) A bandpass filter is used to perform bandpass filtering on the acquired audio signal to remove noise from the audio signal and quantize the passband of the audio signal into M units;

[0023] (2.2) The peak detection method is used to find the maximum peak in the audio signal, and the audio signal before and after the maximum peak point is retained for N / 2 time units to obtain a spectrum matrix of shape N×M, and the spectrum matrix is ​​stored in tensor form;

[0024] (2.3) Normalize all elements in the spectrum matrix to [0,1] to obtain the preprocessed spectrum matrix.

[0025] Furthermore, the bandpass filter includes a Butterworth filter, a Chebyshev filter, and a Bessel filter;

[0026] The normalization methods include min-max normalization, scaling normalization, standardization normalization, and mean-variance normalization.

[0027] Furthermore, the feature extractor is constructed using the backbone of an existing neural network model, and the feature extractor includes multiple convolutional layers and pooling layers;

[0028] The non-line-of-sight signal recognition result output by the tag predictor is expressed as follows:

[0029]

[0030] in, G represents the non-line-of-sight signal recognition result predicted by the tag predictor. y G represents the label predictor. f (x;θ f ) represents the feature vector output by the feature extractor, which is also the label predictor G. y The input is x, which represents the preprocessed spectrum matrix, and θ. f Let θ represent the trainable parameters of the feature extractor f. y G represents the feature extractor y Trainable parameters;

[0031] The domain identification result output by the domain classifier is represented as follows:

[0032]

[0033] in, G represents the domain recognition result predicted by the domain classifier. d Representation domain classifier, G f (x;θ f ) represents the feature vector output by the feature extractor, and is also the domain classifier G. d The input, θ d Representation domain classifier G d Trainable parameters.

[0034] Furthermore, in step (4), when training the domain classifier, a gradient inversion layer is added between the feature extractor and the domain classifier, as shown below:

[0035] R λ (x f )=x f

[0036]

[0037] Where, x f Let R represent the feature vector, λ be a parameter that varies with the training cycle, and R be the feature vector. λ This represents the gradient inversion layer, where I represents the identity matrix.

[0038] In step (4), the loss function of the adversarial domain adaptation network model is expressed as:

[0039]

[0040] Where, E(θ) f ,θ y ,θ d ) represents the loss of the adversarial domain adaptive network model, L y x represents the loss of the label predictor. i Let y represent the i-th spectrum matrix. i L represents the signal label corresponding to the i-th spectrum matrix. d Represents the domain classification loss.

[0041] Furthermore, the neural architecture search method includes:

[0042] ① Random search methods, specifically including:

[0043] The candidate model set is determined based on the candidate operator set. The structural hyperparameters of the label classifier and the domain classifier are randomly set, and all candidate models in the candidate model set are trained for several cycles.

[0044] When periodically training the candidate model, the structural parameters of the feature extractor are frozen, the spectrum matrix is ​​used as the input of the candidate model, and the output is the predicted non-line-of-sight signal recognition result and its corresponding domain recognition result. At the same time, the loss function of the adversarial domain adaptation network model is calculated based on the predicted non-line-of-sight signal recognition result, its corresponding domain recognition result and the signal label corresponding to the spectrum matrix, and the structural hyperparameters of the label classifier and the domain classifier are adjusted based on the loss function of the adversarial domain adaptation network model.

[0045] Based on the predicted non-line-of-sight signal recognition results, their corresponding domain recognition results, and the signal labels corresponding to the spectrum matrix, the recognition accuracy of all candidate models is determined, and the candidate model with the highest accuracy is selected as the optimal candidate model.

[0046] Train the optimal candidate model until the loss function of the adversarial domain adaptation network model converges, and obtain the optimal network structure and parameters of the label classifier and the domain classifier.

[0047] ② Differentiable architecture search methods, specifically including:

[0048] The network structures of the label classifier and the domain classifier are classification variables. Relaxation of the classification options is achieved by calculating the SoftMax of all candidate operators in the network structure.

[0049]

[0050] Where O represents the set of candidate operators, and x′ refers to the input of the current network layer in the neural network. It is a weight vector with the same dimension as O, representing the probability of different operators being selected. Let (i,j) represent the weighted mixture of all operators at (i,j), where (i,j) represents the position index in the vector network structure;

[0051] Searching for discrete structure vectors is equivalent to learning continuous vectors α = {α (i,j) After the search is complete, select the operator with the highest probability:

[0052]

[0053] Among them, o (i,j) This represents the operator with the highest probability, and argmax represents taking the maximum element in the vector.

[0054] The task of searching for network structure is transformed into a two-layer optimization problem, where the structure vector α is the external variable and the weight vector w is the internal variable:

[0055] minL α (w * (α),α)

[0056] stw * (α)=argmin w L train (w,α)

[0057] Among them, w * L represents the optimal weight vector. train Represents the loss function on the training set;

[0058] A two-layer optimization method is used to alternately update the network structure and weights of the label classifier and the domain classifier to obtain the optimal network structure and parameters of the label classifier and the domain classifier.

[0059] A second aspect of the present invention provides a domain-adaptive neural architecture search device for cross-room non-line-of-sight acoustic signal recognition, comprising one or more processors and a memory, wherein the memory is coupled to the processor; wherein the memory is used to store program data, and the processor is used to execute the program data to implement the above-described domain-adaptive neural architecture search method for cross-room non-line-of-sight acoustic signal recognition.

[0060] A third aspect of the present invention provides a computer-readable storage medium having a program stored thereon, which, when executed by a processor, is used to implement the above-described domain-adaptive neural architecture search method for cross-room non-line-of-sight acoustic signal recognition.

[0061] The beneficial effects of this invention are as follows: It uses a spectrum matrix as input to the network model, increasing frequency domain information; it uses a two-dimensional convolutional neural network to extract features; and it removes redundant information through filtering and cropping, further improving the recognition rate and effectively enhancing recognition accuracy. This invention introduces a transfer learning method, employing a deep domain adaptation approach. A domain classifier parallel to the label classifier is added after the feature extractor in the model. Only a small amount of unlabeled target domain data and source domain data are needed for joint training, significantly improving the model's recognition accuracy in the target domain and enhancing generalization. For a trained model, when facing an unknown environment, only a small amount of unlabeled data needs to be added for retraining, without needing to collect and label a large amount of new data and retrain the model. This invention uses NAS technology to automatically adjust model parameters, reducing reliance on deep learning knowledge and experience, thus reducing the cost of deep learning models. This invention can achieve a 99% recognition accuracy for NLOS / LOS sound signals within the same room and a 95% recognition accuracy across rooms. Attached Figure Description

[0062] Figure 1 This is a flowchart illustrating the domain-adaptive neural architecture search method for cross-room non-line-of-sight acoustic signal recognition according to the present invention.

[0063] Figure 2 This is a schematic diagram of the network architecture of the domain-adaptive neural architecture search method for cross-room non-line-of-sight acoustic signal recognition according to the present invention.

[0064] Figure 3 This is a schematic diagram illustrating the linear sweep frequency signal used in this invention, which changes periodically and linearly with time.

[0065] Figure 4 This is a schematic diagram of the NLOS / LOS signal in this invention;

[0066] Figure 5 This is a comparative schematic diagram of the spectrum matrix and spectrum diagram in this invention; wherein, Figure 5 (a) in the diagram is the spectrum of the line-of-sight signal; Figure 5 (b) in the diagram is the spectrum of the non-line-of-sight signal; Figure 5 In the figure, (c) is the spectrum matrix of the line-of-sight signal in (a), and the boxes in the figure indicate the clipping of the spectrum matrix; Figure 5 In the figure, (d) is the spectrum matrix of the non-line-of-sight signal in (b), and the boxes in the figure indicate the clipping of the spectrum matrix;

[0067] Figure 6 This is a diagram of the Resnet34 backbone network structure in this invention;

[0068] Figure 7 This is a schematic diagram of a domain-adaptive neural architecture search device for cross-room non-line-of-sight acoustic signal recognition according to the present invention. Detailed Implementation

[0069] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present invention. Rather, they are merely examples of apparatuses and methods consistent with some aspects of the invention as detailed in the appended claims.

[0070] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. The singular forms “a,” “the,” and “the” used in this invention and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any or all possible combinations of one or more of the associated listed items.

[0071] It should be understood that although the terms first, second, third, etc., may be used in this invention to describe various information, this information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, first information may also be referred to as second information without departing from the scope of this invention, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to a determination."

[0072] The present invention will now be described in detail with reference to the accompanying drawings. Unless otherwise specified, the features of the following embodiments and implementations can be combined with each other.

[0073] Combination Figure 1 and Figure 2 The principle and process of the domain-adaptive neural architecture search method for cross-room non-line-of-sight acoustic signal recognition described in this invention are explained, specifically including the following steps:

[0074] (1) Design audio signals. The speakers in the room periodically send the designed audio signals to the microphone, and the microphone collects the audio signals in the room.

[0075] It should be understood that audio signals need to be actively transmitted before being received; receiving is the process of acquiring audio signals, so the audio signal design must be done first. Other optional audio signals include sine waves and triangle waves, but linear sweep signals are preferred. Linear sweep signals are chosen because they have stronger anti-interference capabilities and are more suitable for acoustic environments with low signal-to-noise ratios.

[0076] Furthermore, the audio signal is as follows Figure 3 The linear chirp signal shown is represented as:

[0077]

[0078]

[0079] Where s(t) is a single linear sweep signal, j represents the imaginary unit, f0 and u0 are the initial frequency and modulation rate, respectively, the length of the linear sweep signal is t, t(τ) represents the periodic overall linear sweep signal, τ represents the propagation delay from different paths, T is the transmission period of the linear sweep signal, the time interval (Tt) is the silence time, ε(·) is the step function, and i represents the i-th signal.

[0080] Furthermore, the audio signal captured by the microphone is represented as:

[0081]

[0082] Where res(τ) represents the audio signal in the room captured by the microphone, n r and n d n represents the number of reflected and diffracted signals, respectively. l =0,1 indicates whether it is a line-of-sight (LOS) signal, n l =1 indicates that the audio signal is a LOS signal, n l =0 indicates that the audio signal is a non-line-of-sight (NLOS) signal, β represents the fading coefficient for different paths, and the subscripts l, r, and d represent the direct path, reflection path, and diffraction path, respectively. Figure 4 As shown, the Blackman window w· is used to eliminate slight multidirectional fluctuations, and n· represents noise, including electromagnetic vibration noise, etc.

[0083] It should be understood that the audio signal corresponding to the direct path is a line-of-sight signal, while the signals corresponding to the reflection path and the diffraction path are non-line-of-sight signals.

[0084] (2) The collected audio signals are preprocessed to obtain the preprocessed spectrum matrix.

[0085] It should be understood that when training a non-line-of-sight (NLOS) audio signal recognition network, using spectrograms or time-domain signals as network input will generate many redundant and useless features, leading to severe overfitting of the network parameters. Therefore, a spectral matrix is ​​chosen instead of spectrograms or time-domain signals as network input. Key information from the spectrogram is mapped to the gray levels of the spectral matrix, while secondary information is discarded. Thus, the acquired audio signal is preprocessed to obtain the spectral matrix.

[0086] Furthermore, the acquired audio signal is preprocessed to obtain a preprocessed spectrum matrix, specifically including:

[0087] (2.1) Bandpass filter is used to perform bandpass filtering on the acquired audio signal to remove noise in the audio signal and quantize the passband of the audio signal into M units.

[0088] Furthermore, bandpass filters include, but are not limited to, Butterworth filters, Chebyshev filters, and Bessel filters.

[0089] It should be understood that, since the audio signal designed in step (1) has a maximum and minimum frequency, a Butterworth filter is used to perform bandpass filtering on the received original audio signal to remove noise outside the transmission frequency, thereby improving the signal-to-noise ratio and quantizing the passband of the audio signal to M units, such as 128 units. Of course, bandpass filters such as Chebyshev filters and Bessel filters can also be used for bandpass filtering. The advantage of the Butterworth filter is that the frequency response curve within the passband is as flat as possible without fluctuations, while it gradually decreases to zero in the stopband.

[0090] Furthermore, the Butterworth filter is expressed using the following formula for the square of the amplitude versus the frequency:

[0091]

[0092] Where H(ω) represents the amplitude of the audio signal, ω represents the frequency of the audio signal, and n ′ ω represents the order of the filter. c ω represents the cutoff frequency of the audio signal. p Indicates the passband edge frequency of an audio signal. For |Hω)| 2 Values ​​at the edge of the passband.

[0093] (2.2) The peak detection method is used to find the maximum peak value in the audio signal. The audio signal before and after the maximum peak value is retained for N / 2 time units to obtain a spectrum matrix with shape N×M, and the spectrum matrix is ​​stored in tensor form.

[0094] Specifically, the peak detection method in the paper "An Efficient Algorithm for Automatic Peak Detection in Noisy Periodic and Quasi-Periodic Signals" is used to find the maximum peak value in the audio signal. The audio signal before and after the maximum peak value is retained for N / 2 time units, e.g., N=64, for a total of 64 time units. Then, the spectrum matrix is ​​cropped to a 64×128 shape and stored as a tensor, as shown below. Figure 5 As shown.

[0095] (2.3) Normalize all elements in the spectrum matrix to [0,1] to obtain the preprocessed spectrum matrix.

[0096] Furthermore, normalization methods include min-max normalization, scaling normalization, standardization normalization, and mean-variance normalization.

[0097] It should be understood that the spectrum matrix needs to undergo preprocessing such as bandpass filtering and peak detection. The size after clipping is not fixed, but is preferably 64×128.

[0098] (3) Construct an adversarial domain adaptation network model, which includes a feature extractor, a label predictor, and a domain classifier. The preprocessed spectrum matrix is ​​input into the feature extractor to obtain feature vectors; the feature vectors are then input into the label predictor and the domain classifier to obtain the non-line-of-sight signal recognition results and their corresponding domain recognition results. The adversarial domain adaptation network model can improve the accuracy of cross-room NLOS / LOS signal recognition.

[0099] In this embodiment, the feature extractor is constructed using the backbone of an existing neural network model, which includes multiple convolutional layers and pooling layers. The preprocessed spectral matrix is ​​input into the feature extractor to obtain the feature vector G. f (x;θ f ), where f represents the feature extractor, x represents the preprocessed spectrum matrix, and θ f This represents the trainable parameters of the feature extractor f.

[0100] Furthermore, existing neural network models include ResNet18, ResNet34, ResNet50, MobileNet, and VGG19. The backbone of these models is responsible for feature extraction. Therefore, the backbone of the neural network model is used as a feature extractor for the latent features of the audio signal. The preprocessed spectrum matrix is ​​input into the feature extractor to obtain the corresponding feature vectors. These feature vectors are then input into a task header with domain adaptation capabilities. The main difference between ResNet series network structures lies in the number of layers; for example, ResNet18 has 18 layers, and ResNet34 has 34 layers. Taking ResNet34 as an example, its backbone network structure is as follows: Figure 6 As shown, it specifically includes multiple convolutional layers and pooling layers; MobileNet and VGG19 have different structures, specifically consisting of stacked convolutional layers and pooling layers.

[0101] It should be understood that when selecting an existing neural network model, ResNet34 is preferred, taking into account both the accuracy of the task results and the computational cost required to run the model.

[0102] In this embodiment, in the cross-room LOS / NLOS classification task, the source domain is defined as labeled audio signals collected in a known room, and the target domain is defined as unlabeled audio signals collected in an unknown room. Due to differences in room layout and environment, the distribution of audio signals received by the microphones also differs. A domain adaptation method is used to reduce the data distribution differences between the source and target domains.

[0103] Specifically, assume that the audio signals in the source and target domains follow distributions S(x,y) and T(x,y), respectively, where x∈X is the input spectrum matrix and y∈Y is the signal label. Signal labels include line-of-sight (LAS) and non-LAS signals. The signal labels of the audio signals in the source domain are known, while the signal labels of the audio signals in the target domain are unknown. The network is then divided into two branches: a label predictor and a domain classifier. The label predictor starts from the high-level feature vector C... f (x;θ fPredict the non-line-of-sight signal recognition result, that is, whether the audio signal corresponding to the current feature vector is a non-line-of-sight signal; the domain classifier is used to determine whether the audio signal corresponding to the current feature vector comes from the source domain or the destination domain.

[0104] Furthermore, the non-line-of-sight signal recognition result output by the tag predictor is expressed as:

[0105]

[0106] in, G represents the non-line-of-sight signal recognition result predicted by the tag predictor. y G represents the label predictor. f (x;θ f ) represents the feature vector output by the feature extractor, which is also the label predictor G. y The input, θ y G represents the feature extractor y Trainable parameters.

[0107] Furthermore, the domain identification result output by the domain classifier is represented as:

[0108]

[0109] in, G represents the domain recognition result predicted by the domain classifier. d Representation domain classifier, G f (x;θ f ) represents the feature vector output by the feature extractor, and is also the domain classifier G. d The input, θ d Representation domain classifier G d Trainable parameters.

[0110] (4) Freeze the structural parameters of the feature extractor and train only its weight parameters; use the neural architecture search method to perform a structure search in the given search space, periodically train the label classifier and the domain classifier, input the preprocessed spectrum matrix into the adversarial domain adaptation network model, obtain the predicted non-line-of-sight signal recognition results and their corresponding domain recognition results, calculate the loss function of the adversarial domain adaptation network model based on the predicted non-line-of-sight signal recognition results and their corresponding domain recognition results and the signal labels, and adjust the parameters of the label classifier and the domain classifier based on the loss function of the adversarial domain adaptation network model to obtain the optimal network structure and parameters of the label classifier and the domain classifier.

[0111] It should be understood that since the feature extractor uses the backbone of an existing neural network model, the structural parameters of which are pre-trained, the structural parameters of the feature extractor are frozen during the structure search process, and only the weight parameters are trained.

[0112] It should be noted that, in order to study the neural network architecture best suited for cross-room LOS / NLOS signal recognition, a neural architecture search method is used to automatically find better-performing models within a given search space. To compress the search space and reduce computational costs, only the architectures of domain classifiers and label predictors are searched, and their architectures are obtained by combining candidate operator sets.

[0113] Furthermore, during the training of the domain classifier, adding a gradient inversion layer between the feature extractor and the domain classifier enables adversarial training. This gradient inversion layer is represented as follows:

[0114] R λ (x f )=x f

[0115]

[0116] Where, x f Let R represent the feature vector, λ be a parameter that varies with the training cycle, and R be the feature vector. λ This represents the gradient inversion layer, and I represents the identity matrix.

[0117] Furthermore, the loss function of the adversarial domain adaptation network model is expressed as:

[0118]

[0119] Where, E(θ) f ,θ y ,θ d ) represents the loss of the adversarial domain adaptive network model, L y x represents the loss of the label predictor. i Let y represent the i-th spectrum matrix. i L represents the signal label corresponding to the i-th spectrum matrix. d Represents the domain classification loss.

[0120] In this embodiment, the neural architecture search method includes:

[0121] ① Random search methods, specifically including:

[0122] A candidate model set is determined based on the candidate operator set. The structural hyperparameters of the label classifier and domain classifier are randomly set, and all candidate models in the set are trained for several epochs. During periodic training of the candidate models, the structural parameters of the feature extractor are frozen. The spectrum matrix is ​​used as the input to the candidate model, and the output is the predicted non-line-of-sight (NLS) signal recognition result and its corresponding domain recognition result. Simultaneously, the loss function of the adversarial domain adaptation network model is calculated based on the predicted NLS signal recognition result, its corresponding domain recognition result, and the signal label corresponding to the spectrum matrix. The structural hyperparameters of the label classifier and domain classifier are adjusted based on the loss function of the adversarial domain adaptation network model. The recognition accuracy of all candidate models is determined based on the predicted NLS signal recognition result, its corresponding domain recognition result, and the signal label corresponding to the spectrum matrix. The candidate model with the highest accuracy is selected as the optimal candidate model. This optimal candidate model is trained until the loss function of the adversarial domain adaptation network model converges, obtaining the optimal network structure and parameters of the label classifier and domain classifier.

[0123] For example, when the backbones of RestNet18 and ResNet34 are selected as feature extractors, since their outputs are one-dimensional tensors, the label classifier and domain classifier are composed of one-dimensional operators. Correspondingly, the candidate operator set O is defined as {fully connected layer, convolutional layer with kernel 1, convolutional layer with kernel 3, identity mapping layer}. When ResNet50, VGG, and MobileNet are selected as feature extractors, since their outputs are two-dimensional tensors, the label classifier and domain classifier are composed of two-dimensional operators. Correspondingly, the candidate operator set O is defined as {fully connected layer, 1×1 convolutional layer, 3×3 convolutional layer, 5×5 convolutional layer, identity mapping layer}.

[0124] It should be understood that the candidate model set is all permutations and combinations of operators in the candidate operator set O; that is, each layer of the candidate model corresponds to a random operator in the operator set. The number and complexity of candidate models increase exponentially with the number of model layers. Both domain classifiers and label predictors have their own candidate model sets.

[0125] ② Differentiable Architecture Search (DARTS) method, specifically including:

[0126] The network structures of the label classifier and the domain classifier are classification variables. Relaxation of the classification options is achieved by calculating the SoftMax of all candidate operators in the network structure.

[0127]

[0128] Where O represents the set of candidate operators, and x′ refers to the input of the current network layer in the neural network. It is a weight vector with the same dimension as O, representing the probability of different operators being selected. Let (i,j) represent the weighted mixture of all operators on (i,j), where (i,j) represents the position index in the vector network structure.

[0129] It should be understood that SoftMax can convert multiple values ​​into probabilities, assigning a probability value to each output classification result to represent the likelihood of belonging to each category; for example, with n values, the larger the value, the higher the probability, and after SoftMax conversion, the sum of these n values ​​is 1. After being converted into probabilities by SoftMax, these probabilities are multiplied by the output of the corresponding operator, and all operators are multiplied in this way, then summed, finally yielding a weighted mixed output of all operators. Relaxation, in mathematical terms, is the relaxation method, which obtains the optimal solution to a related problem through a stepwise approach. In this embodiment, the categorical variable is converted into a probability distribution for solution.

[0130] Searching for discrete structure vectors is equivalent to learning continuous vectors α = {α (i,j) After the search is complete, select the operator with the highest probability:

[0131]

[0132] Among them, o (i,j) This represents the operator with the highest probability, and argmax represents taking the maximum element in the vector.

[0133] The task of searching for network structure is transformed into a two-layer optimization problem, where the structure vector α is the external variable and the weight vector w is the internal variable:

[0134] minL α (w * (α),α)

[0135] stw * (α)=argmin w L train (w,α)

[0136] Among them, w * L represents the optimal weight vector. train This represents the loss function on the training set.

[0137] A two-layer optimization method is used to alternately update the network structure and weights of the label classifier and the domain classifier to obtain the optimal network structure and parameters of the label classifier and the domain classifier, so that the optimized model has good expressiveness and trainability.

[0138] It should be noted that the first random search method first performs a small amount of training on all candidate models, compares the recognition accuracy of all candidate models after training, and then selects the best-performing candidate model to train until convergence. This method is faster and suitable for situations with fewer candidate models. The second differentiable architecture search method is suitable for situations with more candidate models. Therefore, the appropriate neural architecture search method can be chosen based on the specific circumstances.

[0139] It should be understood that the structure of the label classifier and domain classifier of the network model can be automatically obtained through NAS search. The label classifier and domain classifier can be composed of one-dimensional or two-dimensional convolution and pooling operators, preferably one-dimensional operators.

[0140] In summary, the linear sweep frequency acoustic signal used in this invention and the adversarial domain adaptation network model can improve the accuracy of cross-room NLOS / LOS signal recognition. The input of the model is a preprocessed spectrum matrix, and the structure of the model's label classifier and domain classifier is automatically obtained through NAS search.

[0141] Corresponding to the aforementioned embodiments of the domain-adaptive neural architecture search method for cross-room non-line-of-sight acoustic signal recognition, the present invention also provides embodiments of a domain-adaptive neural architecture search device for cross-room non-line-of-sight acoustic signal recognition.

[0142] See Figure 7 The present invention provides a domain-adaptive neural architecture search device for cross-room non-line-of-sight acoustic signal recognition, comprising one or more processors and a memory, wherein the memory is coupled to the processor; wherein the memory is used to store program data, and the processor is used to execute the program data to implement the domain-adaptive neural architecture search method for cross-room non-line-of-sight acoustic signal recognition in the above embodiment.

[0143] The embodiments of the domain-adaptive neural architecture search device for cross-room non-line-of-sight acoustic signal recognition of the present invention can be applied to any device with data processing capabilities, such as a computer. The device embodiments can be implemented in software, hardware, or a combination of both. Taking software implementation as an example, as a logical device, it is formed by the processor of any data-processing device loading the corresponding computer program instructions from non-volatile memory into memory for execution. From a hardware perspective, such as... Figure 7 The diagram shown is a hardware structure diagram of any data processing-capable device in which the domain-adaptive neural architecture search device for cross-room non-line-of-sight acoustic signal recognition of the present invention is located, except... Figure 7In addition to the processor, memory, network interface, and non-volatile memory shown, any data processing device in the embodiment may also include other hardware depending on the actual function of the data processing device, which will not be described in detail here.

[0144] The specific implementation process of the functions and roles of each unit in the above device can be found in the implementation process of the corresponding steps in the above method, and will not be repeated here.

[0145] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of the present invention according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0146] This invention also provides a computer-readable storage medium storing a program that, when executed by a processor, implements the domain-adaptive neural architecture search method for cross-room non-line-of-sight acoustic signal recognition described in the above embodiments.

[0147] The computer-readable storage medium can be an internal storage unit of any data processing device described in any of the foregoing embodiments, such as a hard disk or memory. The computer-readable storage medium can also be any data processing device, such as a plug-in hard disk, smart media card (SMC), SD card, flash card, etc., equipped on the device. Furthermore, the computer-readable storage medium can include both internal storage units of any data processing device and external storage devices. The computer-readable storage medium is used to store the computer program and other programs and data required by the data processing device, and can also be used to temporarily store data that has been output or will be output.

[0148] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A domain-adaptive neural architecture search method for cross-room non-line-of-sight acoustic signal recognition, characterized in that, Includes the following steps: (1) Design audio signals. The speakers in the room periodically send the designed audio signals to the microphone, and the microphone collects the audio signals in the room; (2) Preprocess the acquired audio signals to obtain the preprocessed spectrum matrix; (3) Construct an adversarial domain adaptation network model, which includes a feature extractor, a label predictor and a domain classifier. Input the preprocessed spectrum matrix into the feature extractor to obtain the feature vector. Will The feature vectors are input into the label predictor and the domain classifier respectively to obtain the non-line-of-sight signal recognition results and their corresponding domain recognition results; (4) Freeze the structural parameters of the feature extractor and train only its weight parameters; A neural architecture search method is employed to perform a structure search within a given search space. The label classifier and domain classifier are periodically trained. The preprocessed spectral matrix is ​​input into an adversarial domain adaptation network model to obtain the predicted non-line-of-sight signal recognition results and their corresponding domain recognition results. Based on the predicted non-line-of-sight signal recognition results, their corresponding domain recognition results, and the signal labels, the loss function of the adversarial domain adaptation network model is calculated. The parameters of the label classifier and domain classifier are then adjusted according to the loss function of the adversarial domain adaptation network model to obtain the optimal network structure and parameters for the label classifier and domain classifier.

2. The domain-adaptive neural architecture search method for cross-room non-line-of-sight acoustic signal recognition according to claim 1, characterized in that, The audio signal designed in step (1) is a linear frequency sweep signal, represented as: in, For a single linear sweep frequency signal, j represents the imaginary unit. and The initial frequency and modulation rate are respectively, and the length of the linear sweep signal is . , This represents a periodic overall linear sweep frequency signal. This indicates the propagation delay from different paths. The transmission period of the linear sweep frequency signal, the time interval For the silence period, It is a step function. Indicates the first One signal.

3. The domain-adaptive neural architecture search method for cross-room non-line-of-sight acoustic signal recognition according to claim 1, characterized in that, The audio signal collected by the microphone is represented as follows: in, This indicates the audio signal in the room captured by the microphone. and These represent the quantities of reflected and diffracted signals, respectively. Indicates whether it is a line-of-sight signal. This indicates that the audio signal is a line-of-sight signal. This indicates that the audio signal is a non-line-of-sight signal. Indicates the fading coefficients of different paths, subscript , , These represent the direct path, reflection path, and diffraction path, respectively, and the Black Mann window. Used to eliminate slight multi-directional fluctuations Indicates noise.

4. The domain-adaptive neural architecture search method for cross-room non-line-of-sight acoustic signal recognition according to claim 1, characterized in that, Step (2) includes the following sub-steps: (2.1) A bandpass filter is used to perform bandpass filtering on the acquired audio signal to remove noise from the audio signal and quantize the passband of the audio signal into M units; (2.2) The peak detection method is used to find the maximum peak value in the audio signal. The audio signal before and after the maximum peak value is retained for N / 2 time units to obtain a spectrum matrix with shape N×M, and the spectrum matrix is ​​stored in tensor form. (2.3) Normalize all elements in the spectrum matrix to [0,1] to obtain the preprocessed spectrum matrix.

5. The domain-adaptive neural architecture search method for cross-room non-line-of-sight acoustic signal recognition according to claim 4, characterized in that, The bandpass filters include Butterworth filters, Chebyshev filters, and Bessel filters; The normalization methods include min-max normalization, scaling normalization, standardization normalization, and mean-variance normalization.

6. The domain-adaptive neural architecture search method for cross-room non-line-of-sight acoustic signal recognition according to claim 1, characterized in that, The feature extractor is constructed using the backbone of an existing neural network model, and the feature extractor includes multiple convolutional layers and pooling layers. The non-line-of-sight signal recognition result output by the tag predictor is expressed as follows: in, The non-line-of-sight signal recognition result predicted by the tag predictor. Represents the label predictor. This represents the feature vector output by the feature extractor, and is also the label predictor. Input, This represents the preprocessed spectrum matrix. Feature extractor Trainable parameters, Feature extractor Trainable parameters; The domain identification result output by the domain classifier is represented as follows: in, The domain identification result predicted by the domain classifier. Representation domain classifier, This represents the feature vector output by the feature extractor, and is also the domain classifier. Input, Representation domain classifier Trainable parameters.

7. The domain-adaptive neural architecture search method for cross-room non-line-of-sight acoustic signal recognition according to claim 1, characterized in that, In step (4), when training the domain classifier, a gradient inversion layer is added between the feature extractor and the domain classifier, as shown below: in, Represents the eigenvector. These are parameters that change over the training cycle. This indicates a gradient inversion layer. Represents the identity matrix; In step (4), the loss function of the adversarial domain adaptation network model is expressed as: in, This represents the loss of the adversarial domain adaptive network model. This represents the loss of the label predictor. Represents the i-th spectrum matrix. This represents the signal label corresponding to the i-th spectrum matrix. Represents the domain classification loss.

8. The domain-adaptive neural architecture search method for cross-room non-line-of-sight acoustic signal recognition according to claim 1, characterized in that, The neural architecture search method includes: ① Random search methods, specifically including: The candidate model set is determined based on the candidate operator set. The structural hyperparameters of the label classifier and the domain classifier are randomly set, and all candidate models in the candidate model set are trained for several cycles. When periodically training the candidate model, the structural parameters of the feature extractor are frozen, the spectrum matrix is ​​used as the input of the candidate model, and the output is the predicted non-line-of-sight signal recognition result and its corresponding domain recognition result. At the same time, the loss function of the adversarial domain adaptation network model is calculated based on the predicted non-line-of-sight signal recognition result, its corresponding domain recognition result and the signal label corresponding to the spectrum matrix, and the structural hyperparameters of the label classifier and the domain classifier are adjusted based on the loss function of the adversarial domain adaptation network model. Based on the predicted non-line-of-sight signal recognition results, their corresponding domain recognition results, and the signal labels corresponding to the spectrum matrix, the recognition accuracy of all candidate models is determined, and the candidate model with the highest accuracy is selected as the optimal candidate model. Train the optimal candidate model until the loss function of the adversarial domain adaptation network model converges, and obtain the optimal network structure and parameters of the label classifier and the domain classifier. ② Differentiable architecture search methods, specifically including: The network structures of the label classifier and the domain classifier are classification variables. Relaxation of the classification options is achieved by calculating the SoftMax of all candidate operators in the network structure. in, Denotes the set of candidate operators. The input to the current network layer in a neural network. Is with A weight vector of the same dimension represents the probability of different operators being selected. express A weighted mixture of all operators on, Represents the position index in a vector network structure; Searching for discrete structure vectors is equivalent to learning continuous vectors. After the search is complete, select the operator with the highest probability: in, This represents the operator with the highest probability, and argmax represents taking the maximum element in the vector. The task of searching network structure is transformed into a two-layer optimization problem, where the structure vector... It is an external variable, a weight vector. It is an internal variable: in, Represents the optimal weight vector. Represents the loss function on the training set; A two-layer optimization method is used to alternately update the network structure and weights of the label classifier and the domain classifier to obtain the optimal network structure and parameters of the label classifier and the domain classifier.

9. A domain-adaptive neural architecture search device for cross-room non-line-of-sight acoustic signal recognition, comprising one or more processors and a memory, characterized in that, The memory is coupled to the processor; wherein the memory is used to store program data, and the processor is used to execute the program data to implement the domain-adaptive neural architecture search method for cross-room non-line-of-sight acoustic signal recognition as described in any one of claims 1-8.

10. A computer-readable storage medium, characterized in that, It stores a program that, when executed by a processor, implements the domain-adaptive neural architecture search method for cross-room non-line-of-sight acoustic signal recognition as described in any one of claims 1-8.

Images