A dialect and minority language speech unsupervised automatic semantic annotation method

Abstract

The application discloses a dialect and small language voice unsupervised automatic semantic labeling method, comprising: obtaining target language voice data without manual semantic labeling label and text transcription data; mapping the voice data to a unified semantic vector space to obtain semantic representation vectors of each voice segment; unsupervised clustering the semantic representation vectors to obtain a plurality of voice clustering clusters and corresponding clustering cluster center semantic vectors; obtaining a cross-language general voice semantic prototype library containing hierarchical semantic prototypes and readable semantic labels; matching the clustering cluster center semantic vectors with the semantic prototypes to determine candidate hierarchical semantic labels and label confidence; when the candidate hierarchical semantic labels meet a preset abnormal condition, calibrating and correcting based on the semantic association relationship between the voice clustering clusters, and outputting voice labeling data. The application can convert unsupervised clustering results into readable hierarchical semantic labels without manual labeling and text transcription, improving labeling consistency and reliability.

Description

Technical Field

[0001] This application belongs to the fields of speech signal processing, natural language processing and artificial intelligence technology, and specifically relates to an unsupervised automatic semantic annotation method for dialects and minority languages. Background Technology

[0002] With the development of AI speech technology, high-resource languages ​​such as Mandarin and English have accumulated a large amount of manually annotated speech data. Based on speech recognition, text classification, and intent recognition models, high-precision speech transcription and semantic annotation can be achieved. However, low-resource languages ​​such as dialects, minority languages, and ethnic minority languages ​​generally suffer from problems such as small corpus size, high cost of manual annotation, insufficient professional annotators, inconsistent pronunciation rules, and lack of a standard writing system.

[0003] Existing speech and semantic annotation technologies typically rely on manually labeled samples, speech-to-text transcription data, or speech-to-text-to-semantic mapping datasets. For example, some existing solutions first convert speech into text, and then generate labels based on text similarity, keywords, classification models, or manual verification results. Some annotation solutions for less commonly spoken languages ​​require target language text samples, target language speech samples, or a small number of manually labeled samples as a basis for supervision or fine-tuning. The above solutions are applicable to languages ​​with large-scale annotation resources, but are difficult to directly apply to dialect and less commonly spoken language speech data that lacks annotation, text transcription, or dedicated dictionaries.

[0004] Furthermore, while unsupervised clustering can divide speech segments into several clusters based on the similarity of speech features, the clustering results usually only represent machine-recognizable category numbers or feature sets, lacking a stable mapping relationship with human-readable semantic labels. This makes it difficult to directly generate hierarchical annotation results such as scene, semantics, emotion, and attributes. For dialects and minority languages, different accents, non-standard pronunciations, short-duration speech, ambiguous semantic expressions, and environmental noise can lead to problems such as discrete label distribution within the same cluster, logical conflicts between labels at different levels, concentration of low-confidence samples, and inconsistent labels among semantically strongly correlated clusters. Relying solely on single clustering results or single similarity matching results for annotation can easily lead to local mislabeling and global label inconsistencies, failing to meet the requirements of accuracy, stability, and readability for annotation data in the construction of low-resource speech corpora and subsequent model training. Summary of the Invention

[0005] The purpose of this application is to address the problems of existing dialect and minority language speech semantic annotation technologies, which rely on manually labeled samples, text transcription data, and unsupervised clustering results that are difficult to convert into readable labels, and labeling results are prone to low confidence and semantic conflicts. This application aims to provide an unsupervised automatic semantic annotation method for dialect and minority language speech.

[0006] To achieve the above objectives, the present application adopts the following technical solution.

[0007] An unsupervised automatic semantic annotation method for dialects and minority languages ​​includes:

[0008] Acquire target language speech data without configured manual semantic annotation labels and text transcription data;

[0009] The speech data is mapped to a unified semantic vector space to obtain the semantic representation vector of each speech segment;

[0010] Unsupervised clustering is performed on the semantic representation vector to obtain several speech clusters and the corresponding semantic vector of the cluster center of each cluster;

[0011] Obtain a cross-language universal speech and semantic prototype library, which contains several levels of semantic prototypes and readable semantic tags corresponding to each semantic prototype;

[0012] The semantic vector of the cluster center is matched with the semantic prototype in the prototype library, and the candidate hierarchical semantic label and label confidence of each cluster are determined according to the matching result.

[0013] When the candidate hierarchical semantic labels meet the preset abnormal conditions, the candidate hierarchical semantic labels are calibrated and corrected based on the semantic association between each speech cluster.

[0014] Based on the candidate hierarchical semantic tags or the calibrated and corrected hierarchical semantic tags, output speech annotation data with the hierarchical semantic tags.

[0015] Furthermore, the unsupervised clustering includes two stages: coarse clustering and fine clustering. In the coarse clustering stage, all speech segments are clustered based on the feature similarity between semantic representation vectors. Speech segments with feature similarity higher than the coarse clustering threshold are grouped into the same basic cluster, resulting in several basic clusters. In the fine clustering stage, speech segments within each basic cluster are further clustered. Based on the feature differences within each basic cluster, speech segments within the same basic cluster are split into at least two refined clusters.

[0016] Furthermore, the cross-language universal speech semantic prototype library includes at least two levels of semantic prototypes. The basic scenario-level semantic prototype is used to represent the basic speech behavior scenario of a speech segment, the subdivided semantic level semantic prototype is used to further semantically distinguish speech clusters under the determined basic speech behavior scenario, and the attribute feature-level semantic prototype is used to represent the speech attribute features of a speech segment.

[0017] Furthermore, the cross-lingual universal speech semantic prototype library is constructed based on cross-lingual universal speech data. It aggregates, normalizes, centers, or solidifies the multi-dimensional fused speech feature vectors corresponding to each semantic prototype, generating a corresponding prototype center vector. The semantic prototypes in the prototype library are not bound to any specific language's characters, vocabulary, or grammatical rules.

[0018] Furthermore, the matching includes hierarchical constraint matching. First, the semantic vector of the cluster center is matched with the basic scene hierarchical semantic prototype to determine the first-level basic scene label. Then, within the prototype range corresponding to the matched first-level basic scene label, the semantic vector of the cluster center is matched with the subdivided semantic hierarchical semantic prototype to determine the second-level subdivided semantic label. When the second-level subdivided semantic label obtained by the second-level matching does not satisfy the preset hierarchical correspondence with the first-level basic scene label, the second-level subdivided semantic label is marked as a hierarchical semantic conflict label.

[0019] Furthermore, the semantic association is obtained by weighted fusion of semantic vector similarity, temporal adjacency association degree and pronunciation structure similarity, and is used to determine the semantic association edge weight between two speech clusters.

[0020] Furthermore, the preset abnormal conditions include at least one of the following: the dispersion of candidate hierarchical semantic tags within the same cluster exceeds a preset dispersion threshold; among the candidate hierarchical semantic tags of a cluster, the lower-level semantic tags and the upper-level basic scene tags do not satisfy a preset hierarchical correspondence; the proportion of speech segments with matching confidence lower than a preset admission threshold within the same cluster exceeds a preset proportion threshold; and the candidate hierarchical semantic tags of two clusters with semantic association strength exceeding a preset strength are inconsistent.

[0021] Furthermore, the calibration correction includes at least one of intra-cluster majority voting calibration, upper-level semantic constraint calibration, and strongly correlated cluster label alignment calibration.

[0022] Furthermore, when the matching confidence is lower than a preset admission threshold, the corresponding speech cluster is marked as a semantically undetermined category, and the speech clusters of the semantically undetermined category are returned to the clustering module for feature optimization and re-clustering. When the accumulated number of speech samples of the same semantically undetermined category reaches a preset threshold, a new semantic prototype is generated based on the speech samples. The hierarchical affiliation and readable semantic labels corresponding to the new semantic prototype are determined according to the general semantic topology, and the new semantic prototype and its corresponding readable semantic labels are included in the prototype library.

[0023] Furthermore, mapping the target language speech data to a unified semantic vector space includes: extracting speech features from the target language speech data; inputting the speech features into a cross-language speech semantic encoder to obtain a semantic representation vector; wherein the cross-language speech semantic encoder is configured to enable speech features of different languages ​​or different accents to have comparable semantic representations in the unified semantic vector space.

[0024] Compared with the prior art, the present invention has the following significant advantages:

[0025] This application maps target language speech data without manually labeled semantic tags and text transcription data to a unified semantic vector space, and performs unsupervised clustering based on semantic representation vectors. This enables the formation of speech clusters without relying on manually labeled samples and text transcription data of the target language, thus solving the problem of the inability to automatically process speech data of low-resource dialects and minority languages.

[0026] This application constructs a cross-language universal speech semantic prototype library and matches the semantic vectors of cluster centers with hierarchical semantic prototypes. This enables the conversion of unnamed speech clusters obtained from unsupervised clustering into readable hierarchical semantic labels, solving the problem that traditional unsupervised clustering results cannot be directly converted into usable semantic annotation data.

[0027] This application uses a hierarchical constraint matching mechanism to first determine basic scene labels and then determine subdivided semantic labels within the corresponding prototype scope. This can reduce the probability of semantic conflicts between lower-level semantic labels and upper-level basic scene labels and improve the logical consistency of hierarchical semantic labels.

[0028] This application calibrates and corrects candidate hierarchical semantic labels through semantic association. When candidate labels have low confidence, are discrete within clusters, have hierarchical conflicts, or have inconsistent labels of strongly associated clusters, it can use the label information of associated clusters to correct local mislabeling, thereby improving the overall consistency and stability of the annotation results.

[0029] This application uses a low-confidence label iterative processing mechanism to return low-confidence speech clusters to the clustering module for reprocessing, and expands the prototype library when conditions are met, which can improve the system's ability to adapt to unknown speech patterns, special accents and newly added speech data.

[0030] This application achieves comparable representation of speech features from different languages ​​and accents in a unified semantic vector space through a cross-linguistic speech-semantic encoder. This reduces the interference of language and accent differences on semantic clustering and prototype matching, and improves the generalization ability of dialect and minority language speech-semantic annotation. Attached Figure Description

[0031] The accompanying drawings are provided to further understand this application and form part of the specification. They are used together with the embodiments of this application to explain this application and do not constitute a limitation thereof.

[0032] Figure 1 This is a flowchart illustrating the unsupervised automatic semantic annotation method for dialects and minority languages ​​provided in this application embodiment. Detailed Implementation

[0033] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the scope of protection of this application.

[0034] In this application, "target language speech data" can be understood as speech data of dialects, minor languages, ethnic minority languages, low-resource languages, or other languages ​​that require semantic annotation. "No manually labeled semantic tags and text transcription data" means that the target language speech data itself is not pre-configured with corresponding manually labeled semantic categories, nor is the target language text transcription content pre-configured. This does not exclude the possibility that the system has a general speech semantic prototype library, a hierarchical semantic tagging system, or other general prior rules.

[0035] In this application, "semantic representation vector" refers to the vector representation of the target language speech data after feature extraction and semantic mapping, located in a unified semantic vector space; "cluster center semantic vector" refers to the cluster-level representative vector used to represent the overall semantic features of a speech cluster; "cross-language universal speech semantic prototype library" refers to a reference library including hierarchical semantic prototypes, prototype center vectors, and corresponding readable semantic tags; "readable semantic tags" refers to standardized tags that can be understood and used by humans, such as basic scene tags, subdivided semantic tags, or attribute feature tags.

[0036] Example 1

[0037] like Figure 1 As shown in the figure, this application provides an unsupervised automatic semantic annotation method for dialects and minority languages, which includes the following steps.

[0038] S1, acquire the target language speech data without configured manual semantic annotation labels and text transcription data.

[0039] Target language speech data can include dialect speech, speech of less commonly spoken languages, speech of minority languages, speech of endangered languages, etc. Target language speech data can originate from field collection, recordings of everyday conversations, telephone conversations, recordings from mobile devices, historical recordings from corpora, or other audio acquisition devices.

[0040] In this embodiment, the target language speech data may include noisy speech, speech with uneven speech rate, and speech with different accents. Since this type of speech data does not have manually labeled semantic tags or text transcription data, it is impossible to directly complete semantic annotation through supervised semantic classification models or text classification models.

[0041] In this embodiment, the target language speech data can be preprocessed before semantic mapping. The preprocessing includes at least one of the following: noise reduction, endpoint detection, silence segment filtering, invalid noise segment filtering, sampling rate unification, volume amplitude normalization, speech rate range normalization, and single-channel processing. For example, an adaptive Wiener filtering algorithm can be used for noise reduction, a dual-threshold endpoint detection algorithm can be used to segment effective speech segments, and the speech sampling rate can be unified to 16kHz to reduce interference from the acquisition device, recording scene, and background noise on subsequent semantic representation.

[0042] Through the above processing, effective speech segments can be obtained from the original speech, providing a stable data foundation for subsequent semantic mapping and unsupervised clustering.

[0043] S2 maps the target language speech data to a unified semantic vector space to obtain the semantic representation vector of each speech segment.

[0044] In this embodiment, speech features of the target language speech data can be extracted first, and then the speech features can be mapped to a unified semantic vector space. Speech features may include at least one of acoustic features, prosodic features, articulation structure features, and contextual features.

[0045] Among them, acoustic features can include Mel spectrum, Mel cepstral coefficients, spectral entropy, etc., to characterize the basic pronunciation characteristics of speech; prosodic features can include fundamental frequency, formants, speech rate, duration, pitch change rate, etc., to characterize speech emotion, tone and context differences; and articulation structure features can include articulation boundaries, syllable articulation features, linking features, tone sandhi features, etc., to adapt to dialect linking, accent tone sandhi and special pronunciation rules of minority languages.

[0046] In this embodiment, the aforementioned speech features can be input into a cross-lingual speech-semantic encoder to obtain semantic representation vectors located in a unified semantic vector space. The cross-lingual speech-semantic encoder is configured to ensure that speech features from different languages ​​or accents have comparable semantic representations in the unified semantic vector space. In other words, for speech segments from different languages ​​or accents but with similar semantic behaviors, the semantic representation vectors output by the cross-lingual speech-semantic encoder are relatively close in the unified semantic vector space; for speech segments with significantly different semantic behaviors, their semantic representation vectors are relatively far apart in the unified semantic vector space.

[0047] In this embodiment, the cross-lingual speech-semantic encoder can be trained using a text-free speech pre-training paradigm. This encoder can learn common semantic relationships of speech behaviors across languages ​​based on multilingual universal speech data, without relying on target language text transcription data. This reduces the incomparability issues between different languages, accents, and pronunciation rules, providing a comparable semantic vector foundation for subsequent clustering and prototype matching.

[0048] S3. Perform unsupervised clustering on the semantic representation vector to obtain several speech clusters and the corresponding semantic vector of the cluster center.

[0049] In this application, unsupervised clustering refers to automatically grouping speech segments based on the similarity between semantic representation vectors without using manually labeled semantic tags in the target language as supervision signals.

[0050] In this embodiment, unsupervised clustering may include two stages: coarse clustering and fine clustering.

[0051] The coarse clustering stage can cluster all speech segments based on the feature similarity between semantic representation vectors. Speech segments with feature similarity higher than the coarse clustering threshold are grouped into the same basic cluster, resulting in several basic clusters. These basic clusters can be used to distinguish different basic speech behavior scenarios, such as statements, inquiries, exclamations, instructions, responses, and casual conversation.

[0052] The fine-clustering stage can perform secondary clustering on speech segments within each basic cluster. Based on the feature differences within the basic clusters, speech segments within the same basic cluster are split into at least two refined clusters. These feature differences can include at least one of the following: semantic representation vector differences, prosodic feature differences, pronunciation structure differences, scene semantic differences, and emotional tendency differences. Through fine-clustering, further distinctions can be made regarding semantic details, emotional tendencies, or scene attributes within the same basic speech behavior scenario.

[0053] After completing coarse and fine clustering, the system can generate a cluster center semantic vector corresponding to the fine cluster based on the semantic representation vectors of each speech segment within the same fine cluster. This cluster center semantic vector is used to represent the overall semantic features of the fine cluster and serves as a benchmark for subsequent hierarchical semantic tag matching with a cross-language universal speech semantic prototype library.

[0054] S4, obtain the cross-language universal speech and semantic prototype library.

[0055] The cross-language universal speech and semantic prototype library contains several hierarchical semantic prototypes and corresponding readable semantic tags for each prototype. Each hierarchical semantic prototype may have a corresponding prototype center vector. The prototype center vector is used to characterize the typical speech expression features of the corresponding readable semantic tag.

[0056] In this embodiment, the cross-language universal speech and semantic prototype library may include at least two levels of semantic prototypes. Optionally, the cross-language universal speech and semantic prototype library includes a basic scene-level semantic prototype, a subdivided semantic level semantic prototype, and an attribute feature level semantic prototype.

[0057] The basic scenario-level semantic prototype is used to represent the basic speech behavior scenario of a speech segment. The basic speech behavior scenario can include at least one of statement, inquiry, exclamation, instruction, response, and small talk. The basic scenario-level semantic prototype is mainly used to determine what basic behavior mode the speech segment is expressed in.

[0058] The subdivided semantic level semantic prototype is used to further semantically distinguish speech clusters within a defined basic speech behavior scenario, so as to determine the subdivided semantic labels of the speech clusters in the matched basic scenario during subsequent hierarchical matching. The subdivided semantic level semantic prototype may include at least one of the following: daily life, transportation, emotional expression, business consultation, needs and demands, and social interaction.

[0059] The hierarchical semantic prototype of attribute features is used to represent the speech attribute features of speech segments. The speech attribute features may include at least one of emotional polarity, formality of the scene, urgency, and speech rate type. For example, high fundamental frequency, fast speech rate, and large pitch fluctuations may correspond to urgent, urgent, or positive attributes; low fundamental frequency, steady speech rate, and calm tone may correspond to negative, calm, or formal attributes.

[0060] In this embodiment, the cross-linguistic universal speech semantic prototype library can be constructed based on the general rules of human speech behavior. Specifically, it can extract common acoustic features, prosodic features, and intonation semantic features from high-resource universal speech, extract multi-dimensional fused speech feature vectors in batches for each type of semantic prototype, and store them as corresponding fixed semantic prototype center vectors through prototype feature solidification processing; in the prototype feature solidification process, only universal speech expression features are retained, and regional accents and language-specific pronunciation differences are removed.

[0061] It is worth noting that the semantic prototypes in the prototype library are not bound to the characters, vocabulary or grammatical rules of any specific language. Instead, they are used to represent common speech behaviors, semantic scenarios and attribute features across languages, thus serving as a general label reference for the target language under conditions where there are no manually labeled tags or text transcription data.

[0062] S5 matches the semantic vector of the cluster center with the semantic prototype in the prototype library, and determines the candidate hierarchical semantic labels and label confidence based on the matching results.

[0063] In this embodiment, the similarity between the semantic vector of the cluster center and the prototype center vector in the prototype library can be calculated, and cosine similarity can be used as the matching metric. The similarity calculation formula is as follows:

[0064] (Formula 1),

[0065] Where X represents the semantic vector of the cluster center, Y represents the prototype center vector, and Sim(X,Y) represents the similarity between the two.

[0066] In this embodiment, the matching process employs hierarchical constraint matching. Specifically, firstly, the semantic vector of the cluster center is matched with the basic scene hierarchical semantic prototype in the prototype library to determine the first-level basic scene label of each cluster; then, within the prototype range corresponding to the matched first-level basic scene label, the semantic vector of the cluster center is matched with the subdivided semantic hierarchical semantic prototype to determine the second-level subdivided semantic label of each cluster; furthermore, tertiary attribute labels can be determined based on the attribute feature hierarchical semantic prototype.

[0067] In this embodiment, the primary basic scene tags have priority. When the secondary sub-semantic tags obtained from the second-level matching do not satisfy the preset hierarchical correspondence with the primary basic scene tags, the secondary sub-semantic tags can be marked as hierarchical semantic conflict tags. The preset hierarchical correspondence is a pre-set correspondence between the primary basic scene tags and the candidate lower-level tag set. For example, when the primary basic scene tag is "query", the system prioritizes determining the secondary sub-semantic tags from the candidate sub-semantic tag set corresponding to "query".

[0068] In this embodiment, candidate hierarchical semantic labels and label confidence levels can be determined based on the similarity between the semantic vector of the cluster center and the semantic prototypes of each level, as well as the corresponding preset matching threshold. Specifically, when the similarity between the semantic vector of the cluster center and the semantic prototype of a certain level meets the preset matching threshold of the corresponding level, the readable semantic label corresponding to the semantic prototype of that level is determined as the candidate hierarchical semantic label of the corresponding level.

[0069] Preferably, the preset matching threshold for the first-level basic scene label is 0.75. When the similarity between the semantic vector of the cluster center and the semantic prototype of a certain basic scene level is not less than 0.75, the readable semantic label corresponding to the semantic prototype of the basic scene level is determined as a candidate first-level basic scene label. The preset matching threshold for the second-level subdivision semantic label is 0.80, and matching is performed within the prototype range corresponding to the matched first-level basic scene label. The preset matching threshold for the third-level attribute label is 0.72, and matching is performed based on the semantic prototype of the attribute feature level related to prosodic features.

[0070] When the matching similarity at any level is lower than a preset threshold, the corresponding speech cluster is marked as a low-confidence cluster or a semantically undetermined cluster, and enters the low-confidence label processing flow; wherein, the preset threshold is preferably 0.60. For candidate labels that do not reach the preset matching threshold of the corresponding level but are not lower than the preset threshold, they can be retained as medium-confidence candidate labels and attached with corresponding similarity scores.

[0071] S6. When the candidate hierarchical semantic labels meet the preset abnormal conditions, the candidate hierarchical semantic labels are calibrated and corrected based on the semantic association between each speech cluster.

[0072] In this embodiment, the calibration correction is initiated when a preset abnormal condition occurs in the candidate hierarchical semantic label, so as to avoid over-correction of candidate labels with high confidence, no conflict and good intra-cluster label consistency.

[0073] In this embodiment, the preset abnormal conditions include at least one of the following:

[0074] The initial matching labels of samples within the same refined cluster are mixed with multiple labels, and the coverage of the highest percentage label within the cluster is lower than the preset coverage threshold.

[0075] The lower-level semantic tags and the upper-level basic scene tags do not satisfy the preset hierarchical correspondence;

[0076] The proportion of low-confidence samples within the same cluster exceeds a preset threshold.

[0077] Candidate hierarchical semantic labels are inconsistent between two clusters whose semantic association strength exceeds a preset strong association threshold.

[0078] Preferably, the preset coverage threshold is 0.85, the low confidence sample is the sample with a matching similarity of less than 0.60, the preset proportion threshold is 15%, and the preset strong association threshold is 0.85.

[0079] In this embodiment, a semantic association graph can be constructed using refined speech clusters as nodes and the semantic association strength between speech clusters as edge weights. The semantic association edge weights can be determined by weighting the similarity of the semantic vectors of the cluster centers of two speech clusters, their temporal adjacency, and their pronunciation structure similarity. Specifically, it can be expressed as follows:

[0080] (Formula 2),

[0081] Among them, W ij S represents the semantic association edge weight between the i-th speech cluster and the j-th speech cluster. vec (i,j) represents the similarity between the semantic vectors of the centers of two clusters, S time (i,j) represents the temporal adjacency correlation between speech segments corresponding to two speech clusters, S pron (i,j) represents the similarity of the pronunciation structure between the corresponding speech segments of two speech clusters, and α, β, and γ represent the weighting coefficients. Preferably, α=0.6, β=0.2, and γ=0.2.

[0082] In this embodiment, when W ij When W ≥ 0.85, the two speech clusters are identified as strongly semantically related clusters; when W ≤ 0.60, the two speech clusters are identified as strongly semantically related clusters. ij When W < 0.85, the two speech clusters are identified as weakly semantically related clusters; when W ij When the value is less than 0.60, it is determined that there is no substantial semantic association between the two speech clusters.

[0083] In this embodiment, the calibration correction includes at least one of intra-cluster majority voting correction, upper-level semantic constraint correction, strongly correlated cluster label alignment correction, and accent deviation secondary correction.

[0084] The intra-cluster majority voting correction includes: counting the candidate hierarchical semantic labels of each speech segment within the same speech cluster, and when the candidate hierarchical semantic label with the highest proportion meets the preset majority condition, the candidate hierarchical semantic label is determined as the corrected label of the speech cluster.

[0085] The upper-level semantic constraint correction includes: when the lower-level semantic tag and the upper-level basic scene tag do not satisfy the preset hierarchical correspondence, and the upper-level basic scene tag satisfies the preset high confidence condition, the lower-level semantic tag is re-determined in the candidate lower-level tag set corresponding to the upper-level basic scene tag.

[0086] The strong association cluster label alignment correction includes: selecting the associated clusters that satisfy the strong semantic association condition with the speech clusters to be corrected as reference clusters, and performing weighted correction on the candidate hierarchical semantic labels of the speech clusters to be corrected according to the semantic association edge weights between the reference clusters and the speech clusters to be corrected.

[0087] For short-term fragmented or semantically ambiguous speech samples, candidate hierarchical semantic labels that meet the conditions of strong semantic association or have high semantic association edge weights can be called, and weighted fusion can be performed according to the weight ratio of semantic association edges to complete the missing readable semantic labels.

[0088] The secondary correction of accent deviation includes: when the candidate-level semantic label anomaly is caused by dialect accent shift or special pronunciation of a minority language, the corresponding speech segment is re-corrected by accent adaptive features, the semantic representation vector is reconstructed, and prototype library matching or calibration correction is re-executed.

[0089] Through the above calibration and correction, this application can utilize the global semantic relationships in the semantic association graph to correct local low-confidence labels, hierarchical conflict labels, inconsistent labels of strong association clusters, and accent mismatch labels, thereby improving the consistency and reliability of the annotation results.

[0090] S7 outputs speech annotation data with hierarchical semantic labels based on candidate hierarchical semantic labels or calibrated and corrected hierarchical semantic labels.

[0091] In this embodiment, for speech clusters whose candidate hierarchical semantic label confidence meets preset requirements but does not meet preset anomaly conditions, the candidate hierarchical semantic labels can be directly used as the final hierarchical semantic labels. For speech clusters that meet preset anomaly conditions and have been calibrated and corrected to obtain corrected labels, the calibrated and corrected hierarchical semantic labels can be used as the final hierarchical semantic labels. For medium-confidence labels with a single-level similarity slightly lower than the corresponding matching threshold but not lower than the preset admission threshold, the corresponding candidate hierarchical semantic labels can be retained, along with similarity scores or label confidence information.

[0092] The output speech annotation data may include at least one of the following: speech segment identifier, speech cluster to which it belongs, hierarchical semantic label, label confidence, semantic association information, anomaly marker, and feature parameters. The hierarchical semantic label may include at least one of the following: basic scene label, subdivided semantic label, and attribute feature label.

[0093] In this embodiment, low-confidence labels can also be iteratively processed. When the matching confidence is lower than a preset threshold, the corresponding speech cluster is marked as a semantically undetermined category; the speech clusters of the semantically undetermined category are returned to the clustering module for feature optimization and re-clustering. When the accumulation of similar low-confidence speech samples or semantically undetermined speech samples reaches a preset threshold, and multiple matches with the existing prototype library still fail to achieve the required similarity, it is determined that they correspond to a novel unknown semantic pattern, and a new semantic prototype center vector is generated based on this type of speech sample. The hierarchical affiliation and readable semantic label corresponding to the new semantic prototype are determined according to the general semantic topology. The new readable semantic label can be generated based on the feature distribution of the new semantic prototype in dimensions such as basic scene, subdivided semantics, emotional attributes, and urgency, combined with preset semantic label naming rules, and the new semantic prototype center vector and the corresponding readable semantic label are included in the prototype library. Among them, the general semantic topology is used to represent the hierarchical correspondence and combination rules between basic scene, subdivided semantics, and attribute features.

[0094] In this embodiment, speech samples that still fail to meet the confidence requirements after two reflows, and whose speech features are severely distorted, abnormal, or have insufficient or ambiguous semantic information, can be marked as invalid samples or removed from the final labeled dataset.

[0095] Example 2

[0096] In this embodiment, several unlabeled dialect and minority language speech samples are selected as the target language speech data. This target language speech data is not equipped with manually labeled semantic tags, nor is it configured with corresponding text transcription data.

[0097] First, the target language speech data is subjected to noise reduction, endpoint detection, and sampling rate normalization to obtain multiple valid speech segments. Then, the acoustic features, prosodic features, and articulation structure features of each valid speech segment are extracted, and these speech features are input into a cross-language speech semantic encoder to obtain the semantic representation vector of each speech segment in a unified semantic vector space.

[0098] Then, coarse clustering is performed on the semantic representation vectors to obtain multiple basic clusters; further fine clustering is then performed within each basic cluster to obtain multiple refined clusters. The system generates cluster center semantic vectors based on the semantic representation vectors of speech segments within each refined cluster.

[0099] Next, the system invokes a cross-language universal speech semantic prototype library to perform a first-level match between the semantic vector of the cluster center and the basic scene-level semantic prototype, obtaining first-level basic scene labels. Within the candidate range corresponding to the determined first-level basic scene labels, the semantic vector of the cluster center is matched with the semantic prototype of the subdivided semantic level, obtaining second-level subdivided semantic labels. Finally, it is matched with the semantic prototype of the attribute feature level, obtaining third-level attribute labels. Simultaneously, the system determines candidate level semantic labels and label confidence based on the similarity between the semantic vector of the cluster center and each level of semantic prototype, and the corresponding preset matching thresholds.

[0100] For clusters with high label confidence and no hierarchical conflicts, discrete intra-cluster labels, or inconsistent labels in strongly correlated clusters, the system directly outputs candidate hierarchical semantic labels. For clusters with low confidence, hierarchical conflicts, discrete intra-cluster labels, or inconsistent labels in strongly correlated clusters, the system constructs a semantic association graph, selects strongly correlated and high-confidence reference clusters, and calibrates and corrects the candidate hierarchical semantic labels. Finally, the system outputs speech-annotated data with hierarchical semantic labels, label confidence, and semantic association information.

[0101] Example 3

[0102] This application also provides an unsupervised automatic semantic annotation system for dialects and minority languages, including a data acquisition module, a semantic mapping module, an unsupervised clustering module, a prototype library module, a label matching module, a semantic association calibration and correction module, and an annotation output module.

[0103] The data acquisition module is used to acquire target language speech data that has not been configured with artificial semantic annotation labels and text transcription data.

[0104] The semantic mapping module is used to map target language speech data to a unified semantic vector space, obtaining semantic representation vectors for each speech segment. The semantic mapping module may include a speech feature extraction unit and a cross-language speech semantic coding unit.

[0105] The unsupervised clustering module is used to perform unsupervised clustering on semantic representation vectors, resulting in several speech clusters and the corresponding semantic vectors of the cluster centers. The unsupervised clustering module can include coarse clustering units and fine clustering units.

[0106] The prototype library module is used to store a cross-language universal speech and semantic prototype library, which contains several levels of semantic prototypes and readable semantic tags corresponding to each semantic prototype.

[0107] The label matching module is used to match the semantic vector of the cluster center with the semantic prototype in the prototype library, and determine the candidate hierarchical semantic labels and label confidence scores for each cluster based on the matching results.

[0108] The semantic association calibration and correction module is used to calibrate and correct the candidate-level semantic labels based on the semantic association between each speech cluster when the candidate-level semantic labels meet the preset abnormal conditions.

[0109] The annotation output module is used to output speech annotation data with hierarchical semantic labels based on candidate hierarchical semantic labels or calibrated and corrected hierarchical semantic labels.

[0110] For specific limitations on each module in the above system, please refer to the limitations of the unsupervised automatic semantic annotation method for dialects and minority languages ​​mentioned above. The corresponding technical effects can be obtained equally, and will not be repeated here. Each module in the above system can be implemented entirely or partially through software, hardware, or a combination thereof.

[0111] Example 4

[0112] This application also provides a computer device. The computer device may be a server, a terminal device, an edge computing device, or a cloud computing platform. The computer device includes a processor, a memory, a network interface, and a computer program stored in the memory and executable on the processor.

[0113] When the processor executes the computer program, it implements the steps of the unsupervised automatic semantic annotation method for dialects and minority languages ​​in any of the above embodiments.

[0114] Memory can include non-volatile storage media and internal memory. Non-volatile storage media can store the operating system and computer programs. Internal memory provides the environment for the operation of the operating system and computer programs. Network interfaces can be used to communicate with external devices, voice data acquisition terminals, cloud corpora, or user terminals.

[0115] This application also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the unsupervised automatic semantic annotation method for dialects and minority languages ​​in any of the above embodiments.

[0116] Those skilled in the art will understand that the structure of the computer device described above is merely an example and does not constitute a limitation on the device to which the present application applies. Specific computer devices may include more or fewer components than the structure described above, or combine certain components, or have different component arrangements.

[0117] The above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit it. This application is not limited to the exact structures described above and illustrated in the accompanying drawings, and it should not be considered that the specific implementation of this application is limited to these descriptions. For those skilled in the art, various changes and modifications made without departing from the concept of this application should be considered to fall within the protection scope of this application.

[0118] In summary, the unsupervised automatic semantic annotation method, system, computer equipment, and storage medium for dialects and minority languages ​​provided in this application achieve automatic semantic annotation of target language speech data without manually configured semantic annotation labels and text transcription data by using a unified semantic vector space, two-layer unsupervised clustering, a cross-language universal speech semantic prototype library, hierarchical constraint matching, semantic association calibration and correction, and a low-confidence iterative processing mechanism. This scheme can convert speech clusters obtained from unsupervised clustering into standardized, readable hierarchical semantic labels, and perform calibration and correction when there is low confidence in the labels, hierarchical conflicts, intra-cluster discreteness, or inconsistency in labels of strongly correlated clusters, thereby improving the automation level, label consistency, and result usability of dialect and minority language speech annotation.

Claims

1. An unsupervised automatic semantic annotation method for dialects and minority languages, characterized in that, include: Acquire target language speech data without configured manual semantic annotation labels and text transcription data; The speech data is mapped to a unified semantic vector space to obtain the semantic representation vector of each speech segment; Unsupervised clustering is performed on the semantic representation vector to obtain several speech clusters and the corresponding semantic vector of the cluster center of each cluster; Obtain a cross-language universal speech and semantic prototype library, which contains several levels of semantic prototypes and readable semantic tags corresponding to each semantic prototype; The semantic vector of the cluster center is matched with the semantic prototype in the prototype library, and the candidate hierarchical semantic label and label confidence of each cluster are determined according to the matching result. When the candidate hierarchical semantic labels meet the preset abnormal conditions, the candidate hierarchical semantic labels are calibrated and corrected based on the semantic association between each speech cluster. Based on the candidate hierarchical semantic tags or the calibrated and corrected hierarchical semantic tags, output speech annotation data with the hierarchical semantic tags.

2. The method according to claim 1, characterized in that, The unsupervised clustering includes two stages: coarse clustering and fine clustering. The coarse clustering stage is as follows: all speech segments are clustered based on the feature similarity between the semantic representation vectors, and speech segments with feature similarity higher than the coarse clustering threshold are assigned to the same basic cluster, resulting in several basic clusters; The fine clustering stage is as follows: secondary clustering is performed on the speech segments within each of the basic clusters, and the speech segments within the same basic cluster are split into at least two fine clusters based on the feature differences within the basic clusters.

3. The method according to claim 1, characterized in that, The cross-language universal speech and semantic prototype library includes at least two levels of semantic prototypes, wherein: The basic scenario-level semantic prototype is used to represent the basic speech behavior scenario of a speech segment. The basic speech behavior scenario includes at least one of statement, inquiry, exclamation, instruction, response and chatter. The subdivided semantic level semantic prototype is used to further semantically distinguish speech clusters in a defined basic speech behavior scenario. The subdivided semantic level semantic prototype is used to determine the subdivided semantic labels of the speech clusters in the matched basic scenario. The attribute feature hierarchical semantic prototype is used to characterize the speech attribute features of speech segments, which include at least one of emotional polarity, formality of the scene, urgency, and speech rate type.

4. The method according to claim 1, characterized in that, The construction methods of the cross-language universal speech and semantic prototype library include: Based on cross-language universal speech data, multi-dimensional fused speech feature vectors corresponding to various semantic prototypes are extracted. The multi-dimensional fused speech feature vectors extracted for each semantic prototype are solidified and stored as the corresponding prototype center vector. The semantic prototypes in the prototype library are not bound to any specific language's text, vocabulary, or grammatical rules.

5. The method according to claim 1, characterized in that, The step of matching the semantic vector of the cluster center with the semantic prototype in the prototype library includes hierarchical constraint matching: First, perform a first-level matching between the semantic vector of the cluster center and the basic scene hierarchical semantic prototype in the prototype library to determine the first-level basic scene label of each cluster. Then, within the prototype range corresponding to the matched first-level basic scene label, the semantic vector of the cluster center is matched with the semantic prototype of the subdivided semantic level to determine the second-level subdivided semantic label of each cluster. If the second-level subdivided semantic tags obtained by the second-level matching do not satisfy the preset hierarchical correspondence with the first-level basic scene tags, then the second-level subdivided semantic tags are marked as hierarchical semantic conflict tags.

6. The method according to claim 1, characterized in that, The semantic association is determined in the following way: Obtain the semantic vector similarity between the semantic vectors of the cluster centers of any two speech clusters; Obtain the temporal adjacency correlation between the speech segments corresponding to the two speech clusters; Obtain the similarity of the pronunciation structure between the speech segments corresponding to the two speech clusters; The semantic vector similarity, temporal adjacency correlation, and pronunciation structure similarity are weighted and fused to calculate the semantic association edge weight between the two speech clusters.

7. The method according to claim 1, characterized in that, The preset abnormal conditions include at least one of the following: The dispersion of candidate hierarchical semantic tags within the same cluster exceeds a preset dispersion threshold; In the candidate hierarchical semantic labels of the cluster, the lower-level semantic labels and the upper-level basic scene labels do not satisfy the preset hierarchical correspondence relationship; The proportion of speech segments with matching confidence levels below a preset admission threshold within the same cluster exceeds a preset proportion threshold. Candidate hierarchical semantic labels are inconsistent between two clusters whose semantic association strength exceeds a preset strength.

8. The method according to claim 1, characterized in that, The calibration correction includes at least one of the following calibration methods: Intra-cluster majority voting calibration: Count the candidate labels of each speech segment within the same cluster, and use the label with the highest percentage as the unified standard label for that cluster; Upper-level semantic constraint calibration: When there is a logical conflict between the lower-level tag and the upper-level basic scene tag, the lower-level tag is corrected in reverse based on the upper-level tag; Strongly Associated Cluster Label Alignment Calibration: Based on the labels of strongly associated clusters whose semantic association strength exceeds a preset threshold, the labels in the global semantic domain are standardized in a unified manner.

9. The method according to claim 1, characterized in that, It also includes iterative processing of low-confidence labels: When the confidence level of the match is lower than the preset admission threshold, the corresponding speech cluster is marked as a semantic undetermined category; The speech clusters with undetermined semantic categories are fed back to the clustering module for feature optimization and clustering again; When the number of speech samples of the same semantic undetermined category reaches a preset threshold, a new semantic prototype is generated based on the speech samples. The hierarchical affiliation and readable semantic tags corresponding to the new semantic prototype are determined according to the general semantic topology. The new semantic prototype and its corresponding readable semantic tags are then included in the prototype library.

10. The method according to claim 1, characterized in that, Mapping the target language speech data to a unified semantic vector space includes: Extract speech features from the target language speech data; The speech features are input into a cross-linguistic speech semantic encoder to obtain the semantic representation vector; The cross-linguistic speech semantic encoder is configured to enable speech features of different languages ​​or accents to have comparable semantic representations in the unified semantic vector space.

Images