Method and system for audio identity clustering of physically guided agent neural architecture search

Abstract

The application discloses a physical guiding agent neural architecture search method and system for audio identity clustering, relates to the technical fields of audio signal processing, unsupervised clustering, neural architecture search and agent decision-making, and calculates a Log-Mel time-frequency matrix, acoustic physical features, view coding, an unlabeled data set fingerprint and acoustic credible weights by constructing a full-fragment and two sub-fragment audios; according to search stagnation and clustering diagnosis, search uncertainty is calculated and evidence is searched on demand, a candidate clustering scheme and a verifiable prediction are generated by an agent; audio embedding and clustering results are obtained by executing the candidate scheme, an unlabeled reward is calculated based on clustering clarity, stability, time consistency, view invariance, cluster balance and acoustic consistency, and according to the reward and the prediction verification result, acceptance, rollback, re-diagnosis or experience saving are performed, and an optimal clustering scheme is output; the real identity label does not participate in the search process.

Description

Technical Field

[0001] This invention relates to the fields of audio signal processing, unsupervised clustering, neural architecture search, and agent decision-making, specifically to a physically guided agent neural architecture search method and system for audio identity clustering. Background Technology

[0002] Audio identity clustering refers to automatically grouping audio segments with the same source identity, speaker identity, music identity, device fingerprint, or acoustic content source into the same category when manual annotation is lacking or the true category is not visible. This technology can be applied to scenarios such as audio fingerprint retrieval, speaker merging, music track management, audio content deduplication, and identity recognition in complex recording environments. Because the actual acquisition process often includes reverberation, noise, differences in segmentation positions, differences in recording equipment, and differences in data partitioning, audio samples exhibit significant perturbations in temporal structure, spectral envelope, and local content, leading to increased differences between samples with the same identity, and potentially high similarity between samples with different identities.

[0003] Traditional audio clustering methods typically rely on manually designed acoustic features such as Mel-frequency cepstral coefficients, spectral centroids, chroma, rhythm, and energy statistics, combined with algorithms like KMeans, spectral clustering, or hierarchical clustering. These methods are highly sensitive to the combination of feature sets, normalization methods, dimensionality reduction methods, distance metrics, and clustering algorithms. The optimal pipeline often varies significantly across different audio datasets, making it difficult for human experience to adapt in a timely manner. This leads to problems in clustering results such as cluster collapse, cluster number deviation, cluster size imbalance, and excessive susceptibility to view perturbations.

[0004] Deep neural networks and self-supervised learning can be used for audio representation learning. Audio embeddings are obtained by contrastive learning of different augmented views of the same audio, and a feature space suitable for clustering is learned under unsupervised conditions through cluster-level contrastive loss. However, the network structure, augmentation methods, loss term weights, training epochs, and subsequent clustering strategies of deep encoders also require a lot of experimentation. Directly using random search or fixed grid search will lead to high training costs, low search efficiency, and difficulty in interpreting the basis for structure selection.

[0005] While existing agent-based automated machine learning methods can call upon large language models or experience bases to propose candidate configurations, the candidate design is prone to leaking real labels or post-evaluation results into the search process. When selecting models based solely on single internal metrics such as silhouette coefficients, constraints in areas such as audio segment temporal consistency, acquisition view invariance, and acoustic-physical neighborhood preservation are easily overlooked. Furthermore, the reasons given by the agent are mostly natural language explanations, lacking verifiable predictions bound to observable unlabeled metrics, making it difficult to determine the credibility of the reasoning after the candidate scheme is executed, and also difficult to transfer credible experience across datasets.

[0006] Therefore, there is a need for an intelligent architecture search method for audio identity clustering that combines acoustic physical features, unlabeled internal diagnostics, dynamic neural architecture training, verifiable reasoning, and cross-dataset memory, without requiring real identity labels to participate in the search. Summary of the Invention

[0007] To address the problems in existing audio clustering pipelines, such as reliance on human experience, high search costs, easy leakage of real labels, insufficient evaluation of unlabeled data, and difficulty in verifying agent reasoning, this invention proposes a physically guided agent neural architecture search method and system for audio identity clustering.

[0008] To achieve the above objectives, the present invention provides the following technical solution:

[0009] The unlabeled original audio samples are used to construct segments, resulting in full segment audio, first sub-segment audio, and second sub-segment audio. Acoustic physical features are extracted and view codes are recorded.

[0010] Calculate the unlabeled dataset fingerprint and acoustic confidence weight based on the acoustic physical characteristics;

[0011] Search uncertainty is calculated based on the degree of search stagnation and the degree of cluster diagnosis anomaly, and evidence is retrieved on demand based on the search uncertainty;

[0012] The intelligent agent generates candidate audio clustering schemes and verifiable predictions based on the fingerprint of the unlabeled dataset, acoustic confidence weights, and the current clustering diagnosis results, and generates the candidate audio clustering schemes in conjunction with the evidence when it is retrieved.

[0013] The candidate audio clustering scheme is executed to obtain audio embeddings and clustering results, and an unlabeled reward is calculated based on the audio embeddings, clustering results, view encoding, and acoustic physical features.

[0014] Compare the unlabeled rewards of the current candidate audio clustering scheme with those of the previous scheme, verify the verifiable predictions, update the search results based on the reward comparison results, use the verifiable predictions that have been determined to be verified for memory updates, and output the optimal audio clustering scheme obtained from the current search.

[0015] Among them, real identity tags do not participate in candidate audio clustering scheme generation, unlabeled reward calculation, agent retrieval, and memory update.

[0016] Furthermore, the fragment construction and extraction of acoustic physical features include:

[0017] The original audio samples are unified into mono audio with a fixed sampling rate. A fixed-length main audio segment is extracted as the full segment audio, and the full segment audio is divided into a first sub-segment audio and a second sub-segment audio on an even basis.

[0018] Calculate the Log-Mel time-frequency matrix of the full audio segment, the first sub-segment audio segment, and the second sub-segment audio segment, respectively;

[0019] The handcrafted acoustic feature set is extracted from the full audio segment and then spliced ​​and normalized to form the acoustic physical features. The handcrafted acoustic feature set includes Mel-frequency cepstral coefficients, chromaticity features, and spectral centroid features.

[0020] Furthermore, the view encoding is used to record the source conditions or perturbation conditions of the audio samples, including clean recording, noisy recording, reverberant recording, different recording devices, different data division sources, or different segmentation positions;

[0021] The unlabeled dataset fingerprint includes the number of samples, the expected number of clusters, the acoustic physical feature dimension, the proportion of each group of acoustic features in all acoustic features, the silhouette coefficient, the Calinski-Harabasz index, and the Davies-Bouldin index.

[0022] Furthermore, the calculation of the acoustic credibility weight includes:

[0023] The acoustic physical features are used to perform initial clustering of all audio samples and calculate the silhouette coefficients.

[0024] Based on the lower and upper reference values ​​of the profile coefficient, the profile coefficient is normalized, and the normalization result is restricted to the range of 0 to 1 to obtain the acoustic confidence weight.

[0025] Furthermore, the degree of search stagnation is calculated as the proportion of the number of rounds in which the reward for no tag is not increased within a preset number of rounds to the preset number of rounds;

[0026] The degree of abnormality in cluster diagnosis is determined based on whether the silhouette coefficient of the current candidate clustering result is lower than the silhouette coefficient threshold, whether the actual number of non-empty clusters is equal to the expected number of clusters, and whether the inter-cluster separation ratio is lower than the inter-cluster separation ratio threshold.

[0027] Based on the comparison results between the search uncertainty and the corresponding retrieval threshold, the search can be performed using historical memory, domain knowledge base, or tag-independent visualization results.

[0028] When neither historical memory nor domain knowledge base returns evidence and the search uncertainty reaches the external technical data retrieval threshold, external technical data is retrieved.

[0029] Assign evidence numbers to the retrieved evidence, and record the source of the evidence, the content of the evidence, and the search stall or cluster diagnosis results that triggered the retrieval of the evidence.

[0030] Furthermore, the candidate audio clustering scheme includes a preprocessing method, a feature selection method, an encoding method, a loss function, and a clustering algorithm. The preprocessing method includes normalization, L2 normalization, or whitening. The encoding method includes a pre-trained audio encoder, a dynamic neural network, or a handcrafted feature path. The clustering algorithm includes KMeans, Gaussian mixture model, spectral clustering, or hierarchical clustering.

[0031] When evidence is retrieved, the candidate audio clustering scheme records the evidence number cited, the search stall situation that triggered the corresponding evidence retrieval or the clustering diagnosis result, and the configuration modifications supported by the evidence.

[0032] The verifiable predictions include the expected improvement in unlabeled metrics, the direction of improvement, and the minimum amount of improvement.

[0033] Further, the candidate audio clustering scheme is executed, including:

[0034] When the encoding method selects a pre-trained audio encoder, the audio embedding generated by the pre-trained audio encoder is read.

[0035] When the encoding method selects a dynamic neural network, a model is constructed according to the network layer type given by the agent, and unsupervised loss is used to train on the Log-Mel time-frequency matrix to obtain audio embeddings.

[0036] When the encoding method selects a handcrafted feature path, the selected handcrafted acoustic features are standardized and dimensionality reduced to obtain the audio embedding.

[0037] Clustering is performed on the full-segment audio embeddings, and the embeddings of the first and second sub-segments of audio are assigned to the nearest full-segment cluster centers, respectively, to obtain the full-segment clustering results, the first sub-segment clustering results, and the second sub-segment clustering results.

[0038] Furthermore, the unlabeled rewards include cluster clarity, cluster stability, temporal consistency, view invariance, cluster balance, and acoustic consistency.

[0039] The clustering clarity is determined based on the silhouette coefficient and the Davies-Bouldin index; the clustering stability is determined by repeatedly clustering the same audio embedding with different random seeds and calculating the adjusted Rand index between different clustering results; the temporal consistency is determined based on the consistency between the full segment clustering result and the first sub-segment clustering result and the second sub-segment clustering result; the view invariance is determined based on the distribution of view encoding within the same cluster; the cluster balance is determined based on the distribution of the number of samples in each cluster; and the acoustic consistency is determined based on the overlap ratio of nearest neighbor samples in the acoustic physical feature space and the audio embedding space.

[0040] The unlabeled reward is a weighted combination of various indicators.

[0041] Furthermore, the previous round of schemes is used as the parent trial and the current candidate audio clustering scheme is used as the child trial, and the unlabeled rewards of the parent trial and the child trial are compared.

[0042] The sub-trial is accepted when its unlabeled reward is higher than that of the parent trial; the sub-trial is rolled back to the parent trial when its unlabeled reward does not increase.

[0043] The improvement of the unlabeled index corresponding to the verifiable prediction in the sub-experiment relative to the parent experiment is compared with the minimum improvement of the verifiable prediction. When the improvement reaches the minimum improvement, the verifiable prediction is determined to be verified. When the improvement does not reach the minimum improvement, it is determined to be contradictory. When the parent experiment or the corresponding unlabeled index is missing, it is determined to be unverifiable.

[0044] Verifiable predictions identified as verified will be used for memory updates.

[0045] Furthermore, a physical-guided agent neural architecture search system for audio identity clustering is proposed to implement the physical-guided agent neural architecture search method for audio identity clustering as described above, including:

[0046] The data construction module is used to construct segments from unlabeled raw audio samples, generating full segment audio, first sub-segment audio, and second sub-segment audio, and recording the view encoding;

[0047] The feature calculation module is used to calculate the Log-Mel time-frequency matrix of the full audio segment, the first sub-segment audio segment, and the second sub-segment audio segment, extract the handcrafted acoustic features, and then splice and standardize the handcrafted acoustic features to form acoustic physical features;

[0048] The dataset fingerprint calculation module is used to calculate the unlabeled dataset fingerprint and acoustic confidence weight based on the acoustic physical characteristics.

[0049] The intelligent agent design module is used to calculate search uncertainty based on the degree of search stagnation and the degree of cluster diagnosis anomaly, retrieve evidence on demand based on the search uncertainty, and generate candidate audio clustering schemes and verifiable predictions based on the fingerprint of the unlabeled dataset, acoustic credibility weights and current cluster diagnosis results. When evidence is retrieved, the candidate audio clustering scheme is generated in combination with the evidence.

[0050] The clustering module is used to execute the candidate audio clustering scheme to obtain audio embeddings and clustering results;

[0051] The unlabeled evaluation module is used to calculate the unlabeled reward based on the audio embedding, clustering results, view encoding, and acoustic physical features.

[0052] The verification update module is used to compare the unlabeled rewards of the current candidate audio clustering scheme with those of the previous scheme, verify the verifiable prediction, and update the search results based on the reward comparison results.

[0053] The memory management module is used to save historical clustering schemes, corresponding unlabeled rewards, and verification results of verifiable predictions, and uses verifiable predictions that have been identified as verified for memory updates.

[0054] The output module is used to output the optimal audio clustering scheme obtained from the current search, the corresponding configuration, and the unlabeled evaluation results;

[0055] Among them, real identity tags do not participate in candidate audio clustering scheme generation, unlabeled reward calculation, agent retrieval, and memory update.

[0056] Compared with existing technologies, it has the following advantages:

[0057] This invention enables the automatic selection of a clustering pipeline suitable for the current audio dataset without using real identity labels in the search. Real labels are used only for the final evaluation and do not participate in agent prompting, reward calculation, candidate generation, or memory update processes, thus maintaining the independence and reliability of the unsupervised clustering search process.

[0058] The present invention achieved experimental results on three audio identity clustering datasets: ACC, NMI, ARI, Recall@1, and mAP were 99.90%, 99.96%, 99.80%, 99.75%, and 95.25% on the NAFP-500 dataset; 95.40%, 98.68%, 92.12%, 99.25%, and 97.81% on the NAFP-500+RIRNoise dataset; and 66.00%, 71.20%, 47.50%, 73.30%, and 54.20% on the VoxCeleb2-15 dataset.

[0059] Compared to the best or strongest contrasting methods on the corresponding datasets, on the NAFP-500 dataset, ACC is improved by 2.15 percentage points and ARI by 7.81 percentage points; on the NAFP-500+RIRNoise dataset, ACC, NMI, ARI, and mAP are improved by 0.45, 0.70, 0.24, and 0.87 percentage points, respectively; and on the VoxCeleb2-15 dataset, ACC, NMI, ARI, Recall@1, and mAP are improved by 3.67, 2.24, 6.05, 5.30, and 2.96 percentage points, respectively.

[0060] When analyzing 200 candidate solutions from 20 complete searches, the Spearman correlation coefficients between the unlabeled reward ranking and the final ACC, NMI, and ARI rankings were 0.966, 0.969, and 0.960, respectively, with an average selection gap of 0.00 percentage points for all three and a Top-3 hit rate of 100%, indicating that the unlabeled reward can be used for candidate solution screening during the search process.

[0061] Through a verifiable prediction and memory filtering mechanism, in cross-dataset searches from NAFP-500 to NAFP-500+RIRNoise, using memory with filtered results improves audio clustering ACC by 5.93 percentage points compared to not using memory, and enables faster attainment of the target reward level. Attached Figure Description

[0062] Figure 1 This is a flowchart illustrating the overall method of the present invention;

[0063] Figure 2 This is a schematic diagram of encoder selection and dynamic topology training according to the present invention;

[0064] Figure 3 This invention provides enhanced memory for retrieval and verification of the gated closed-loop diagram. Detailed Implementation

[0065] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0066] This embodiment uses audio identity clustering as an example for illustration. The identity can be audio fingerprint identity, speaker identity, music track identity, device voiceprint identity, sound source object identity, or other clustered objects stably represented by audio features.

[0067] Example 1:

[0068] This embodiment uses the publicly available NAFP-500 dataset for validation. The NAFP-500 dataset includes 500 different recording identities, each containing 4 audio views, totaling 2,000 audio samples. The "recording identity" refers to the same recording object that should be clustered into the same class; the "audio view" refers to audio samples of the same recording object obtained under different processing conditions, perturbation conditions, or fragment formats. The system only uses the audio samples themselves, view information, and unlabeled features calculated from the audio; it does not use real recording identity labels in candidate scheme generation, unlabeled reward calculation, agent retrieval, and memory update. Real recording identity labels are only used after the search is completed to calculate evaluation metrics such as ACC, NMI, ARI, Recall@1, and mAP to verify whether the clustering results are consistent with the real recording identities.

[0069] Step 1: Construct audio segments and extract basic acoustic features.

[0070] For each original audio sample, it is first standardized to a fixed sampling rate mono audio, and then a fixed-length main audio segment is extracted as the full segment audio. Taking a full segment length of 6 seconds as an example, if the original audio is longer than 6 seconds, a continuous 6 seconds is extracted as the full segment audio; if the original audio is shorter than 6 seconds, zeros are padded to the end to bring it to 6 seconds. Then, this 6-second full segment audio is divided into two 3-second audio segments: the first 3 seconds is the first sub-segment audio, and the last 3 seconds is the second sub-segment audio. The full segment audio is used to obtain the overall representation of the audio sample, while the first and second sub-segments audio are used to check whether the same audio can still be classified into the same or similar clustering results at different time positions.

[0071] Calculate the Log-Mel time-frequency matrix for the entire audio segment, the first sub-segment audio segment, and the second sub-segment audio segment. First, perform a short-time Fourier transform on the audio signal to obtain the power spectrum. Then through the Mel filter Converting to the Mel band and finally taking the logarithm, the Log-Mel time-frequency matrix can be expressed as:

[0072]

[0073] in, Represents a time frame. Indicates frequency point, Indicates the first One Mel frequency band, To prevent a very small constant from having a zero value when taking the logarithm.

[0074] In addition to the Log-Mel time-frequency matrix, handcrafted acoustic feature sets are extracted from the entire audio segment, including Mel-spectral coefficients, chromaticity features, and spectral centroid features. Mel-spectral coefficients are used to describe the timbre characteristics of the audio, and their calculation can be expressed as:

[0075]

[0076] in, Indicates the first cepstral coefficients, This indicates the total number of Mel bands. Indicates the first The average Log-Mel energy of each Mel band.

[0077] Chromaticity features are used to describe the energy distribution of the twelve tone levels, dividing the frequency points... Mapped to corresponding pitch , No. The chromaticity characteristics of each pitch level can be expressed as:

[0078]

[0079] in, Indicates the first The chromatic characteristics of each pitch level Indicates belonging to the first A set of frequencies for each pitch level.

[0080] The centroid of the spectrum is used to represent the concentrated location of audio energy on the frequency axis, and its calculation can be expressed as:

[0081]

[0082] in, Indicates the first The spectral centroid of the frame. The above-mentioned handcrafted acoustic features are spliced ​​and standardized to form acoustic physical features, which are used to characterize the objective acoustic properties of audio in terms of timbre, spectral shape, rhythm, and energy changes.

[0083] Step 2: Label the view encoding and calculate the dataset fingerprint.

[0084] When constructing audio segments, a view code is recorded for each audio sample. The view code is not an identity label or a clustering result, but rather a digital label used to record the source conditions or perturbation conditions of the audio sample. Clean recordings, noisy recordings, reverberant recordings, recordings from different devices, data partitioning sources, or different segmentation locations can all serve as different views; when three views exist, they can be used respectively. , and This indicates clean audio, noisy audio, and reverb audio.

[0085] Calculate the fingerprint of the unlabeled dataset based on the acoustic physical characteristics of all audio samples. Assume the dataset has a total of... Given samples, the expected number of clusters is... Acoustic physical feature dimension is Then the fingerprint of the dataset can be represented as:

[0086]

[0087] in, This represents the current audio dataset. Indicates the number of acoustic feature groups. Indicates the first The proportion of a group of acoustic features in all acoustic features. Represents the profile coefficient. This represents the Calinski-Harabasz index. This represents the Davies-Bouldin index. The percentage of each group of acoustic features. The calculation method is as follows: Let the first... Group acoustic features in the first The standardized feature vectors on each audio sample are: Then the first The average feature intensity of the acoustic features across all samples is ,in Let represent the L2 norm. The proportion of group acoustic features is Where h is the summation index of the acoustic feature set, with a value from 1 to G. This represents the average feature intensity of the h-th acoustic feature across all audio samples, and its calculation method is similar to... same, Indicates the number of acoustic feature groups. To prevent extremely small constants with a denominator of zero. The larger the value, the higher the value. The higher the proportion of group acoustic features in the acoustic physical features of the current dataset; The smaller the value, the lower the contribution of the acoustic features to the overall acoustic representation of the current dataset. The expected number of clusters, K, is derived from the pre-defined number of identities in the current clustering task. If the dataset task description specifies the number of audio identities to be clustered, this number is used as K; if the user specifies the target number of clusters in actual application, the user-specified value is used as K; if neither is specified, initial clustering can be performed within a preset candidate range, and K is selected based on the principle of a larger silhouette coefficient, a larger Calinski-Harabasz exponent, and a smaller Davies-Bouldin exponent. The clustering results used to calculate the silhouette coefficient, Calinski-Harabasz exponent, and Davies-Bouldin exponent are obtained by initial clustering of all audio samples based on acoustic physical features; this initial clustering result is the same as the initial clustering result used to calculate the acoustic confidence weight in step three. Step three only reuses the silhouette coefficient from this initial clustering result for normalization calculation, without defining a new clustering source.

[0088] Suppose that after clustering we get The cluster, the first The feature vector of each sample is , No. Each cluster is , No. The center of each cluster is The global center of all samples is . No. The average distance from each sample to other samples within the same cluster is:

[0089]

[0090] No. The minimum of the average distances from each sample to all other clusters is:

[0091]

[0092] No. The contour values ​​of each sample and the contour coefficients of the entire dataset are as follows:

[0093]

[0094]

[0095] in, Indicates the first The cluster number to which each sample belongs. This represents the distance between the feature vectors of two samples. To prevent extremely small constants with a denominator of zero.

[0096] In the calculation of the Calinski-Harabasz index, the intra-cluster dispersion... and inter-cluster dispersion They are respectively:

[0097]

[0098]

[0099] Therefore, the Calinski-Harabasz index is:

[0100]

[0101] In the calculation of the Davies-Bouldin index, the first The average intra-cluster distance of the first cluster and the first cluster Cluster and the first The similarities between the clusters are as follows:

[0102]

[0103]

[0104] The Davies-Bouldin index is:

[0105]

[0106] A larger silhouette coefficient indicates that samples within the same cluster are closer and samples between different clusters are more separated; a larger Calinski-Harabasz index indicates that the distance between clusters is greater than the distance within a cluster; a smaller Davies-Bouldin index indicates that the clusters are more compact and samples between clusters are more separated.

[0107] Step 3: Calculate the acoustic confidence weights.

[0108] First, perform an initial clustering of all samples using acoustic physical features, and then calculate the silhouette coefficient. and will Mapping to a value between 0 and 1 yields acoustic confidence weights. :

[0109]

[0110] in, This indicates the lower limit reference value of the profile coefficient. This indicates the upper limit reference value for the profile coefficient. To prevent extremely small constants with a denominator of zero, This indicates that the result will be limited to between 0 and 1. In this embodiment, Take 0, Take 0.5. Its determination principle is: contour coefficient. When the average distance between a sample and its own cluster is not less than the average distance between it and other clusters, it indicates that the initial clustering result obtained based on acoustic physical features does not have reliable separation, therefore the acoustic confidence weight is set to 0; silhouette coefficient When the cluster is relatively compact and there is a clear separation between different clusters, the acoustic confidence weight is set to 1; when At that time, the acoustic credibility weight is based on The position changes linearly between 0 and 0.5.

[0111] Step 4: Perform on-demand evidence retrieval based on search uncertainty.

[0112] Before the agent generates new candidate audio clustering schemes, the system first assesses search uncertainty based on factors such as whether the current search has been unlabeled for a long time without increasing rewards, and whether the current clustering results exhibit issues like intra-cluster discontinuity, inter-cluster non-separation, or discrepancies between the actual number of non-empty clusters and the expected number of clusters. In the... In rounds of search, search uncertainty for:

[0113]

[0114] in, Indicates the degree of search stall. Indicates the degree of abnormality in cluster diagnosis. and The weight is denoted as L. The preset number of search rounds is 5 in this embodiment, meaning the system calculates the proportion of rounds in the most recent 5 searches where the reward for no label did not increase as the degree of search stagnation. Profile coefficient threshold. The value is set to 0.25, which is derived from the lower limit reference value of the profile coefficient in the acoustic confidence weight. and upper limit reference value The intermediate value is determined, that is Inter-cluster separation ratio threshold A value of 1 indicates that the minimum cluster center distance should be at least twice the average intra-cluster distance of non-empty clusters. Historical memory retrieval threshold. Set the threshold to 0.2 for domain knowledge base retrieval. Set the threshold to 0.4, the threshold for triggering label-independent visualization. Set the threshold to 0.6 for external technical data retrieval. The threshold is set to 0.8. The above thresholds are increased sequentially from low to high search cost, so that the system first calls the historical memory and domain knowledge with lower cost, and then generates visualization results or retrieves external technical information when the uncertainty is high.

[0115] The degree of search stagnation can be based on the most recent Calculation of the proportion of rounds in which the unlabeled reward did not increase during the search rounds; the degree of cluster diagnosis anomaly can be expressed as:

[0116]

[0117] in, The silhouette coefficient represents the current candidate clustering result. Indicates the contour coefficient threshold. This indicates the number of actual non-empty clusters in the current clustering result. Indicates the desired number of clusters. Indicates the inter-cluster separation ratio. Indicates the threshold for inter-cluster separation ratio. This is an indicator function. When no empty clusters exist, the inter-cluster separation ratio can be expressed as:

[0118]

[0119] in, and They represent the first The cluster and the first The center of each cluster, Indicates the first The average distance from a sample within a cluster to the cluster center. Let the nth cluster be... In the round clustering results, the first Each cluster contains a set of samples. The sample size is .when When the cluster is non-empty, the set of indices for all non-empty clusters is denoted as . The actual number of non-empty clusters is For non-empty clusters The cluster center and the average distance within the cluster are respectively: and .in, Indicates the first Embedding vectors of individual audio samples. Inter-cluster separation ratio. Computation based solely on non-empty clusters: When an empty cluster exists, i.e. When empty clusters are empty, the cluster center and average intra-cluster distance are not calculated, nor are they included in the calculation. Calculate the numerator and denominator, but This fact is still counted as an abnormal item in cluster diagnosis. When the number of non-empty clusters is less than 2, i.e. When there are no two comparable non-empty clusters, let , to ensure It can be actually calculated for all clustering results.

[0120] get Then, the search source is selected based on different thresholds: when At that time, retrieve previously executed clustering schemes, unlabeled rewards, validation results, and proven effective configurations; when When, the applicable conditions for retrieving audio features, encoders, loss functions, and clustering algorithms; when And the current contour coefficient When label-independent 2D feature maps are generated, colored only according to the predicted cluster numbers, to observe cluster overlap, cluster collapse, or obvious outliers; when neither historical memory nor the domain knowledge base returns valid evidence and Then, external technical information will be searched.

[0121] The system assigns an evidence number to each piece of evidence retrieved and records the evidence source, content, and the search stall or clustering diagnosis result that triggered the evidence retrieval. The corresponding evidence number is then used by the agent to generate candidate audio clustering schemes. If no valid evidence is found, the agent generates a scheme based solely on the current dataset fingerprint, acoustic confidence weights, and unlabeled diagnostic results. Figure 3 It illustrates the closed-loop relationship between unlabeled fingerprints of datasets, working memory and long-term memory, uncertainty computation, source-triggered retrieval, evidence rearrangement and annotation, verifiable prediction, and acceptance, rollback, solidification, or re-diagnosis.

[0122] Step 5: Generate candidate audio clustering schemes using the intelligent agent.

[0123] The intelligent agent is a program or model used to automatically select processing schemes. It generates candidate audio clustering schemes based on the current dataset fingerprint, acoustic confidence weights, current clustering diagnostic results, and historical memory, domain knowledge, label-independent visualization results, or external technical materials obtained in step four. If step four returns an evidence number, the candidate scheme specifies the cited evidence number, the search stall condition that triggered the corresponding evidence retrieval or the clustering diagnostic result, and the configuration modifications supported by that evidence; if no evidence is returned, the candidate scheme is generated only based on the current unlabeled diagnostic results.

[0124] Candidate audio clustering schemes include preprocessing methods, feature selection methods, encoding methods, loss functions, and clustering algorithms. Preprocessing methods include normalization, L2 normalization, or whitening; feature selection methods determine the acoustic features to use; encoding methods determine whether to use a pre-trained audio encoder, a dynamic neural network, or handcrafted features; loss functions are used to train the neural network; and clustering algorithms include KMeans, Gaussian mixture models, spectral clustering, or hierarchical clustering. The agent also provides verifiable predictions, clearly defining the unlabeled metrics expected to improve and the direction of improvement for each candidate scheme. The minimum improvement amount is pre-set by the system according to the metric type and is not generated by the agent based on the current search state. Specifically, the minimum improvement amount of the unlabeled reward R... The minimum improvement for silhouette coefficient S, cluster stability, temporal consistency, view invariance, cluster balance, and acoustic consistency is set to 0.01; the minimum improvement for inter-cluster separation ratio is set to 0.05; the Calinski-Harabasz index and Davies-Bouldin index are used with relative improvement ratios, and the minimum relative improvement is set to 0.01. For example, the agent can predict that the silhouette coefficient S will increase by at least 0.01, or predict that the Davies-Bouldin index will decrease by at least 1% relative to the parent experiment.

[0125] Step 6: Execute the candidate scheme and obtain the audio embedding and clustering results.

[0126] The actuator performs calculations according to the candidate audio clustering scheme generated by the agent, converting each audio sample into a fixed-length numerical vector. The audio embedding can be represented as:

[0127]

[0128] in, Indicates the first One audio sample, This represents the vector dimension. This vector is used to represent the identity-related features of the audio, and subsequent clustering algorithms group these vectors.

[0129] like Figure 2 As shown, candidate schemes can choose either a pre-trained audio encoder path or a dynamic neural network path. When a pre-trained audio encoder is selected, the actuator reads the full and sub-segment audio embeddings already generated by the encoder; sample labels must not be included in the agent's prompts, reward calculations, or search processes, and are only used to check whether the cached embeddings are consistent with the sample order. Figure 2The training shown includes InfoNCE, clustering, acoustic, and reconstruction losses. When selecting a handcrafted feature path, the chosen handcrafted acoustic features are used directly, and the audio embeddings are obtained after standardization and dimensionality reduction. When selecting a dynamic neural network, the actuator instantiates the encoder based on the network structure information output by the agent. This network structure information is not a natural language description, but rather an executable structured configuration, including input feature type, list of layer primitives, input number for each layer, number of output channels or hidden dimension, kernel size, stride, normalization method, activation function, connection method, pooling method, embedding dimension, and training parameters. The input feature type is a Log-Mel time-frequency matrix of the entire audio segment or sub-segment audio. Optional layer primitives include one-dimensional temporal convolution, two-dimensional time-frequency convolution, depthwise separable convolution, residual blocks, Transformer coding blocks, fully connected projection layers, batch normalization, layer normalization, ReLU activation, GELU activation, Dropout, average pooling, and attention pooling. The dynamic topology must satisfy the following constraints: the network is a directed acyclic structure; the number of feature extraction blocks is 2 to 6; each layer can only connect to the output of the previous layer or the two layers above it; residual connections require that the input and output dimensions be consistent; if the dimensions are inconsistent, they are aligned using a 1×1 convolution or linear projection layer; after concatenation, the network must be compressed to the specified dimension using a 1×1 convolution or linear layer; the downsampling times for the time and frequency dimensions do not exceed 2; finally, a fixed-length audio embedding is obtained through global average pooling or attention pooling, with the embedding dimension being 128 or 256. If the network structure output by the agent does not satisfy the above constraints, the actuator will not execute the structure and will require the agent to re-output a valid structure.

[0130] After obtaining the audio embedding, clustering is performed on the full-segment audio embedding to obtain the full-segment clustering result for each sample; then the embeddings of the first and second sub-segment audio are assigned to the nearest full-segment clustering centers to obtain two sub-segment clustering results, and a unified clustering and sub-segment consistency evaluation is performed.

[0131] Step 7: Calculate the unlabeled reward.

[0132] Label-free rewards evaluate candidate clustering schemes without real-name labels, and include cluster clarity, cluster stability, temporal consistency, view invariance, cluster balance, and acoustic consistency. Cluster clarity can be measured by the silhouette coefficient. and Davies-Bouldin index calculate:

[0133]

[0134] Temporal consistency is used to determine whether the entire segment, the first sub-segment, and the second sub-segment of the same audio are assigned to the same cluster:

[0135]

[0136] in, Indicates the first Full-fragment clustering labels for each sample. and These represent the clustering labels for the first and second sub-segments, respectively. View invariance is used to determine whether the clustering results are overly dependent on view encoding factors such as noise, reverberation, and recording equipment; cluster balance is used to prevent all samples from being assigned to one cluster or a few clusters; acoustic consistency is used to determine whether the audio embedding still retains the proximity relationships in the acoustic physical features, which can be calculated in the acoustic physical feature space and the audio embedding space. The overlap ratio of nearest neighbor samples. The final unlabeled reward can be expressed as:

[0137]

[0138] in, , , , , and These correspond to cluster clarity, cluster stability, temporal consistency, view invariance, cluster balance, and acoustic consistency, respectively. to Each value represents a weight, and the sum of all weights is 1.

[0139] Cluster stability This is used to determine whether the clustering results obtained by embedding the same audio data under different random initializations are consistent. This embodiment uses... Clustering was repeatedly performed with different random seeds to obtain 5 cluster labels. The average adjusted RAND index (ARI) between any two cluster labels was calculated as the average value. If the average value is less than 0, it is counted as 0; if it is greater than 1, it is counted as 1.

[0140] View Invariance This method is used to determine whether clustering results are overly reliant on view coding due to noise, reverberation, recording equipment, or data augmentation techniques. Specifically, the view coding distribution of all samples is first calculated, then the view coding distribution within each non-empty cluster is calculated, and the Shannon divergence between the two is calculated. The smaller the divergence, the closer the view distribution within the cluster is to the overall view distribution. The larger the value, the better. After normalizing this divergence by dividing it by log2, subtracting the normalized divergence from 1 yields the result. .

[0141] Cluster balance This is used to prevent samples from clustering into one or a few clusters. Specifically, the actual number of non-empty clusters is first calculated. With the expected number of clusters The proportion is then used to calculate the information entropy of the sample quantity distribution of each non-empty cluster, and then... Normalization; multiplying the two yields The more non-empty clusters there are, the closer the sample size in each cluster is to uniformity. The closer it is to 1.

[0142] Acoustic consistency Used to determine whether audio embedding preserves nearest neighbor relationships in the acoustic physical feature space. For the first... Each sample is searched in the acoustic physical feature space and the audio embedding space, respectively. The nearest neighbors, among which And calculate the overlap ratio of two nearest neighbor sets; take the average of the overlap ratios of all samples to obtain .

[0143] The weights are determined by the system, not generated freely by the agent. Acoustic consistency weights. ,in The acoustic credibility weight is assigned accordingly; the remaining weights for the other five indicators are allocated in the proportions of 0.30, 0.20, 0.20, 0.15, and 0.15. .

[0144] Step 8: Verify candidate solutions and update search results.

[0145] After the candidate audio clustering schemes are executed, the previous scheme is used as the parent trial and the current scheme as the child trial, and their unlabeled rewards are compared. If the reward of the current scheme is higher than that of the parent trial, it means that the current scheme is better under unlabeled evaluation and can be accepted; if the reward of the current scheme does not improve, it can be rolled back to the parent trial scheme.

[0146] Simultaneously verify the verifiable predictions given by the agent. Suppose we are predicting a certain index. It will increase at least The parent test index is The current plan's indicators are The prediction is verified when the following equation is satisfied:

[0147]

[0148] when When the prediction is contradictory, it is unverifiable when the parent trial is missing or the indicator is unavailable. The system decides whether to accept the proposed solution, roll back the solution, re-diagnose, or save the experience based on whether the reward increases and whether the prediction holds true, to determine the effectiveness of the solution itself and the reliability of the agent's interpretation, respectively. The updated verification results are used for subsequent searches and memory management.

[0149] Step 9: Form the system structure and output the optimal clustering scheme.

[0150] The audio identity clustering physical-guided intelligent agent neural architecture search system of the present invention includes a data construction module, a feature calculation module, a dataset fingerprint calculation module, an intelligent agent design module, an execution clustering module, an unlabeled evaluation module, a verification and update module, a memory management module, and an output module. The data construction module is used to construct segments from unlabeled raw audio samples, generating full-segment audio, first sub-segment audio, and second sub-segment audio, and recording view encodings. The feature calculation module is used to calculate the Log-Mel time-frequency matrix of the full-segment audio, first sub-segment audio, and second sub-segment audio, extract handcrafted acoustic features, and then concatenate and standardize the handcrafted acoustic features to form acoustic physical features. The dataset fingerprint calculation module is used to calculate the unlabeled dataset fingerprint and acoustic trust weight based on the acoustic physical features. Before generating candidate audio clustering schemes, the intelligent agent design module calculates search uncertainty based on the degree of search stagnation and the degree of cluster diagnosis anomalies, and retrieves historical memory, domain knowledge base, label-independent visualization results, or external technical materials as needed based on the search uncertainty, and calculates the unlabeled dataset fingerprint and acoustic trust weight based on the unlabeled dataset fingerprint and acoustic trust weight. The system generates candidate audio clustering schemes and verifiable predictions by combining the current clustering diagnosis results with the existing ones. When evidence is retrieved, candidate audio clustering schemes are generated in conjunction with that evidence. The clustering execution module executes the candidate audio clustering schemes to obtain audio embeddings and clustering results. The unlabeled evaluation module calculates unlabeled rewards based on audio embeddings, clustering results, view encoding, and acoustic physical features. The verification and update module compares the unlabeled rewards of the current candidate audio clustering scheme with those of the previous round, verifies verifiable predictions, and updates the search results based on the reward comparison results. The memory management module saves historical clustering schemes, corresponding unlabeled rewards, and verification results of verifiable predictions, and uses verified predictions for memory updates. The output module outputs the optimal audio clustering scheme obtained from the current search, its corresponding configuration, and the unlabeled evaluation results. During system operation, real identity labels do not participate in candidate audio clustering scheme generation, unlabeled reward calculation, agent retrieval, or memory updates.

[0151] The comparison results of this embodiment on the NAFP-500 dataset are shown in Table 1. ACC represents clustering accuracy, NMI represents normalized mutual information, ARI represents adjusted RAND index, Recall@1 represents the proportion of correct identities found in the first-ranked search, and mAP represents average search precision.

[0152] As shown in Table 1, on the NAFP-500 dataset, the proposed invention achieved 99.90%, 99.96%, and 99.80% in ACC, NMI, and ARI, respectively, and 99.75% and 95.25% in Recall@1 and mAP, respectively.

[0153]

[0154] The above embodiments are only used to illustrate the technical methods of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical methods of the present invention without departing from the spirit and scope of the technical methods of the present invention.

Claims

1. A physically guided intelligent agent neural architecture search method for audio identity clustering, characterized in that, include: The unlabeled original audio samples are used to construct segments, resulting in full segment audio, first sub-segment audio, and second sub-segment audio. Acoustic physical features are extracted and view codes are recorded. Calculate the unlabeled dataset fingerprint and acoustic confidence weight based on the acoustic physical characteristics; Search uncertainty is calculated based on the degree of search stagnation and the degree of cluster diagnosis anomaly, and evidence is retrieved on demand based on the search uncertainty; The intelligent agent generates candidate audio clustering schemes and verifiable predictions based on the fingerprint of the unlabeled dataset, acoustic confidence weights, and the current clustering diagnosis results, and generates the candidate audio clustering schemes in conjunction with the evidence when it is retrieved. The candidate audio clustering scheme is executed to obtain audio embeddings and clustering results, and an unlabeled reward is calculated based on the audio embeddings, clustering results, view encoding, and acoustic physical features. Compare the unlabeled rewards of the current candidate audio clustering scheme with those of the previous scheme, verify the verifiable predictions, update the search results based on the reward comparison results, use the verifiable predictions that have been determined to be verified for memory updates, and output the optimal audio clustering scheme obtained from the current search. Among them, real identity tags do not participate in candidate audio clustering scheme generation, unlabeled reward calculation, agent retrieval, and memory update.

2. The physically guided intelligent agent neural architecture search method for audio identity clustering according to claim 1, characterized in that, The fragment construction and extraction of acoustic physical features include: The original audio samples are unified into mono audio with a fixed sampling rate. A fixed-length main audio segment is extracted as the full segment audio, and the full segment audio is divided into a first sub-segment audio and a second sub-segment audio on an even basis. Calculate the Log-Mel time-frequency matrix of the full audio segment, the first sub-segment audio segment, and the second sub-segment audio segment, respectively; The handcrafted acoustic feature set is extracted from the full audio segment and then spliced ​​and normalized to form the acoustic physical features. The handcrafted acoustic feature set includes Mel-frequency cepstral coefficients, chromaticity features, and spectral centroid features.

3. The physically guided intelligent agent neural architecture search method for audio identity clustering according to claim 1, characterized in that, include: The view encoding is used to record the source conditions or perturbation conditions of the audio samples. The source conditions or perturbation conditions include clean recording, noisy recording, reverberant recording, different recording devices, different data division sources, or different segmentation positions. The unlabeled dataset fingerprint includes the number of samples, the expected number of clusters, the acoustic physical feature dimension, the proportion of each group of acoustic features in all acoustic features, the silhouette coefficient, the Calinski-Harabasz index, and the Davies-Bouldin index.

4. The physically guided intelligent agent neural architecture search method for audio identity clustering according to claim 1, characterized in that, The calculation of the acoustic credibility weight includes: The acoustic physical features are used to perform initial clustering of all audio samples and calculate the silhouette coefficients. Based on the lower and upper reference values ​​of the profile coefficient, the profile coefficient is normalized, and the normalization result is restricted to the range of 0 to 1 to obtain the acoustic confidence weight.

5. The physically guided intelligent agent neural architecture search method for audio identity clustering according to claim 1, characterized in that, include: The degree of search stagnation is calculated as the proportion of the number of rounds in which the reward for no tag is not increased within a preset number of rounds to the preset number of rounds; The degree of abnormality in cluster diagnosis is determined based on whether the silhouette coefficient of the current candidate clustering result is lower than the silhouette coefficient threshold, whether the actual number of non-empty clusters is equal to the expected number of clusters, and whether the inter-cluster separation ratio is lower than the inter-cluster separation ratio threshold. Based on the comparison results between the search uncertainty and the corresponding retrieval threshold, the search can be performed using historical memory, domain knowledge base, or tag-independent visualization results. When neither historical memory nor domain knowledge base returns evidence and the search uncertainty reaches the external technical data retrieval threshold, external technical data is retrieved. Assign evidence numbers to the retrieved evidence, and record the source of the evidence, the content of the evidence, and the search stall or cluster diagnosis results that triggered the retrieval of the evidence.

6. The physically guided intelligent agent neural architecture search method for audio identity clustering according to claim 1, characterized in that, include: The candidate audio clustering scheme includes a preprocessing method, a feature selection method, an encoding method, a loss function, and a clustering algorithm. The preprocessing method includes normalization, L2 normalization, or whitening. The encoding method includes a pre-trained audio encoder, a dynamic neural network, or a hand-crafted feature path. The clustering algorithm includes KMeans, Gaussian mixture model, spectral clustering, or hierarchical clustering. When evidence is retrieved, the candidate audio clustering scheme records the evidence number cited, the search stall situation that triggered the corresponding evidence retrieval or the clustering diagnosis result, and the configuration modifications supported by the evidence. The verifiable predictions include the expected improvement in unlabeled metrics, the direction of improvement, and the minimum amount of improvement.

7. The physically guided intelligent agent neural architecture search method for audio identity clustering according to claim 6, characterized in that, Executing the candidate audio clustering scheme includes: When the encoding method selects a pre-trained audio encoder, the audio embedding generated by the pre-trained audio encoder is read. When the encoding method selects a dynamic neural network, a model is constructed according to the network layer type given by the agent, and unsupervised loss is used to train on the Log-Mel time-frequency matrix to obtain audio embeddings. When the encoding method selects a handcrafted feature path, the selected handcrafted acoustic features are standardized and dimensionality reduced to obtain the audio embedding. Clustering is performed on the full-segment audio embeddings, and the embeddings of the first and second sub-segments of audio are assigned to the nearest full-segment cluster centers, respectively, to obtain the full-segment clustering results, the first sub-segment clustering results, and the second sub-segment clustering results.

8. The physically guided intelligent agent neural architecture search method for audio identity clustering according to claim 1, characterized in that, include: The unlabeled rewards include cluster clarity, cluster stability, temporal consistency, view invariance, cluster balance, and acoustic consistency. The clustering clarity is determined based on the silhouette coefficient and the Davies-Bouldin index; the clustering stability is determined by repeatedly clustering the same audio embedding with different random seeds and calculating the adjusted Rand index between different clustering results; the temporal consistency is determined based on the consistency between the full segment clustering result and the first sub-segment clustering result and the second sub-segment clustering result; the view invariance is determined based on the distribution of view encoding within the same cluster; the cluster balance is determined based on the distribution of the number of samples in each cluster; and the acoustic consistency is determined based on the overlap ratio of nearest neighbor samples in the acoustic physical feature space and the audio embedding space. The unlabeled reward is a weighted combination of various indicators.

9. The physically guided intelligent agent neural architecture search method for audio identity clustering according to claim 1, characterized in that, include: The previous round of schemes is used as the parent trial, and the current candidate audio clustering scheme is used as the child trial. The unlabeled rewards of the parent trial and the child trial are compared. The sub-trial is accepted when its unlabeled reward is higher than that of the parent trial; the sub-trial is rolled back to the parent trial when its unlabeled reward does not increase. The improvement of the unlabeled index corresponding to the verifiable prediction in the sub-experiment relative to the parent experiment is compared with the minimum improvement of the verifiable prediction. When the improvement reaches the minimum improvement, the verifiable prediction is determined to be verified. When the improvement does not reach the minimum improvement, it is determined to be contradictory. When the parent experiment or the corresponding unlabeled index is missing, it is determined to be unverifiable. Verifiable predictions identified as verified will be used for memory updates.

10. A physically guided intelligent agent neural architecture search system for audio identity clustering, used to implement the physically guided intelligent agent neural architecture search method for audio identity clustering as described in any one of claims 1-9, characterized in that, include: The data construction module is used to construct segments from unlabeled raw audio samples, generating full segment audio, first sub-segment audio, and second sub-segment audio, and recording the view encoding; The feature calculation module is used to calculate the Log-Mel time-frequency matrix of the full audio segment, the first sub-segment audio segment, and the second sub-segment audio segment, extract the handcrafted acoustic features, and then splice and standardize the handcrafted acoustic features to form acoustic physical features; The dataset fingerprint calculation module is used to calculate the unlabeled dataset fingerprint and acoustic confidence weight based on the acoustic physical characteristics. The intelligent agent design module is used to calculate search uncertainty based on the degree of search stagnation and the degree of cluster diagnosis anomaly, retrieve evidence on demand based on the search uncertainty, and generate candidate audio clustering schemes and verifiable predictions based on the fingerprint of the unlabeled dataset, acoustic credibility weights and current cluster diagnosis results. When evidence is retrieved, the candidate audio clustering scheme is generated in combination with the evidence. The clustering module is used to execute the candidate audio clustering scheme to obtain audio embeddings and clustering results; The unlabeled evaluation module is used to calculate the unlabeled reward based on the audio embedding, clustering results, view encoding, and acoustic physical features. The verification update module is used to compare the unlabeled rewards of the current candidate audio clustering scheme with those of the previous scheme, verify the verifiable prediction, and update the search results based on the reward comparison results. The memory management module is used to save historical clustering schemes, corresponding unlabeled rewards, and verification results of verifiable predictions, and uses verifiable predictions that have been identified as verified for memory updates. The output module is used to output the optimal audio clustering scheme obtained from the current search, the corresponding configuration, and the unlabeled evaluation results; Among them, real identity tags do not participate in candidate audio clustering scheme generation, unlabeled reward calculation, agent retrieval, and memory update.

Images