Personnel identity recognition method and device based on audio sample, and medium

Abstract

The invention provides a personnel identity recognition method and device based on an audio sample and a medium, and relates to the technical field of personnel identity recognition, and the method comprises the steps: obtaining each fine-granularity human voice segment corresponding to a to-be-recognized voice segment, so as to obtain a fine-granularity human voice segment list QC; obtaining a voiceprint feature vector corresponding to each human voice segment in the QC to obtain a voiceprint feature column vector table HC corresponding to the QC; obtaining a preset voiceprint feature vector list HA; obtaining the maximum similarity HX between the HCx and each preset voiceprint feature vector in the HA; the method comprises the steps of determining a user corresponding to HX as a user corresponding to QCx to obtain an identified user list GA if HX is larger than or equal to H ', determining QCx as an unidentified human voice fragment if HX is smaller than H', and determining a user corresponding to QCx according to the human voice fragment, corresponding to each user in QC, in GA. The method can enhance robustness and continuity of identity identification in a complex acoustic environment.

Description

Technical Field

[0001] The present invention relates to the technical field of personnel identification, and in particular to a personnel identification method, device and medium based on audio samples. Background Art

[0002] Voice interaction in civil aviation cabin environments is often accompanied by overlapping conversations between multiple people (such as crew collaboration and cross-transmission of emergency commands). Existing voiceprint recognition technology faces significant limitations in such scenarios: traditional methods rely on independent voiceprint matching of a single voice segment. When multiple voices are mixed, it is difficult to effectively separate and identify the identity of the target speaker due to the superposition of spectral energy and interference of acoustic features; the voiceprint judgment mechanism based on a fixed threshold cannot adapt to the feature distortion caused by dynamic noise environments (such as high-frequency engine noise and equipment alarm sounds), and often misjudges mixed voices as invalid segments or incorrectly matches non-target users; in addition, existing technologies lack the ability to backtrack unrecognized voice segments. When key instructions are not successfully parsed due to instantaneous noise or multiple people speaking at the same time, the system cannot complete the identity through context association, resulting in a lack of integrity in the aviation communication link. Summary of the Invention

[0003] In view of the above technical problems, the technical solution adopted by the present invention is: According to a first aspect of the present application, a method for identifying a person based on an audio sample is provided, the method comprising the following steps: G100 obtains each fine-grained vocal segment corresponding to the speech segment to be recognized, so as to obtain a fine-grained vocal segment list QC = (QC1, QC2, ..., QC x ,…,QC y ), x=1, 2, …, y; where QC x is the xth fine-grained vocal segment corresponding to the speech to be recognized, and y is the number of fine-grained vocal segments corresponding to the speech to be recognized; G200, obtain the voiceprint feature vector corresponding to each human voice segment in QC to obtain the voiceprint feature column vector table HC corresponding to QC = (HC1, HC2, ..., HC x ,…,HC y ); Among them, HC x For QC x The corresponding voiceprint feature vector; G300 obtains each preset voiceprint feature vector to obtain a preset voiceprint feature vector list HA=(HA1, HA2, ..., HA c ,…,HA d ), c = 1, 2, ..., d; where HA c is the voiceprint feature vector of the cth user, and d is the number of users; G400, get HC x The maximum similarity HX with each preset voiceprint feature vector in HA; G500, if HX ≥ H', the user corresponding to HX is determined as QC x Corresponding users, to obtain the identified user list GA = (GA1, GA2, ..., GA C1 ,…,GA d1 ), c1=1, 2, …, d1; where GA C1 is the c1th user identified, d1 is the number of identified users; G600, if HX<H', then QC x Determined to be an unrecognized vocal segment; enter G700; G700, according to the corresponding vocal segment of each user in QC in GA, determines QC x The corresponding user.

[0004] According to another aspect of the present application, a non-transitory computer-readable storage medium is also provided, in which at least one instruction or at least one program is stored, and the at least one instruction or at least one program is loaded and executed by a processor to implement the above-mentioned audio sample-based person identification method.

[0005] According to another aspect of the present application, an electronic device is provided, including a processor and the above-mentioned non-transitory computer-readable storage medium.

[0006] The present invention has at least the following beneficial effects: The personnel identification method based on audio samples of the present invention effectively copes with the challenge of multi-person mixed voice identification through the collaborative mechanism of fine-grained speech segmentation and dynamic voiceprint matching: a hierarchical voiceprint feature extraction strategy is adopted to suppress environmental noise interference while retaining individual biometric characteristics, significantly improving the target voiceprint separation accuracy in high-noise scenarios; through iterative comparison of preset voiceprint libraries and adaptive backtracking analysis of unidentified segments, layer-by-layer analysis of multiple identities in mixed voices is achieved, avoiding identity omissions due to instantaneous interference or multi-person concurrency; combined with the context association model, the system can dynamically correct the judgment threshold of unmatched segments based on the voiceprint distribution of identified segments, thereby enhancing the robustness and continuity of identity identification in complex acoustic environments, meeting the security authentication requirements of multi-person collaborative operations in civil aviation scenarios, and avoiding the problem of lack of integrity of aviation communication links. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0008] Figure 1 This is a flowchart of a method for identifying a person based on audio samples provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0009] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without making any creative efforts shall fall within the scope of protection of the present invention.

[0010] It should be noted that, based on this disclosure, those skilled in the art will appreciate that an aspect described herein can be implemented independently of any other aspect, and that two or more of these aspects can be combined in various ways. For example, any number of the aspects described herein can be used to implement an apparatus and / or practice a method. In addition, other structures and / or functionalities other than one or more of the aspects described herein can be used to implement such an apparatus and / or practice such a method.

[0011] Example 1: In some scenarios, it is necessary to determine the user corresponding to a vocal clip. Therefore, the following method is provided: The human voice recognition method based on fusion comparison includes the following steps: H100, obtain each vocal segment corresponding to the speech segment to be recognized to obtain a vocal segment list QC = (QC1, QC2, ..., QC x ,…,QC y ), x=1, 2, …, y; where QC x is the xth vocal segment corresponding to the speech to be recognized, and y is the number of vocal segments corresponding to the speech to be recognized.

[0012] In this embodiment, it should be noted that the voice segments to be recognized include vocal segments and non-vocal segments, and the vocal segments in the voice segments to be recognized can be recognized to obtain the vocal segment list QC; those skilled in the art can use the existing vocal segment recognition method according to actual needs to recognize the voice segments to be recognized to obtain the vocal segment list QC, which will not be elaborated here.

[0013] H200, obtain the voiceprint feature vector corresponding to each human voice segment in QC to obtain the voiceprint feature vector list HC corresponding to QC = (HC1, HC2, ..., HC x ,…,HC y ); Among them, HC x For QC x The corresponding voiceprint feature vector.

[0014] In this embodiment, the fundamental frequency, MFCC differential coefficient, resonance peak bandwidth and other features corresponding to each vocal segment in the QC can be obtained, and then the corresponding voiceprint feature vector can be constructed based on the above features. It should be noted that those skilled in the art can use the existing voiceprint feature vector acquisition method according to actual needs to obtain the voiceprint feature vector corresponding to each vocal segment in the QC, which will not be elaborated here.

[0015] H300, obtain each preset voiceprint feature vector to obtain a preset voiceprint feature vector list HA = (HA1, HA2, ..., HA c ,…,HA d ), c = 1, 2, ..., d; where HA c is the voiceprint feature vector of the cth user, and d is the number of users.

[0016] In this embodiment, the method is applied in the cockpit of an airplane. For the same airplane, the users who frequently appear in the cabin are known. Voiceprint features can be extracted from each known user in an exhaustive manner to obtain HA.

[0017] H400, get HC x The maximum similarity HX with each preset voiceprint feature vector in HA.

[0018] In this embodiment, it should be noted that those skilled in the art can use existing vector similarity acquisition methods according to actual needs to obtain HC x The maximum similarity HX with each preset voiceprint feature vector in HA is not described here.

[0019] H500, if HX ≥ H', the user corresponding to HX is determined as QC x Corresponding user; otherwise, enter H600; where H' is the preset voiceprint feature similarity threshold.

[0020] In this embodiment, if HX≥H', it means that the voice characteristics of the user corresponding to HX are consistent with QC x The voice characteristics of the corresponding users are relatively similar, so it can be determined that the known user corresponding to HX is QC xCorresponding user; and if HX<H', it is necessary to perform another judgment through step H600; H' is an empirical value, which can be obtained through statistical analysis of a large amount of sound data.

[0021] H600, according to HC x Determine QC with the preset fusion voiceprint feature vector list x The corresponding user.

[0022] Furthermore, step H600 includes the following steps: H610, get the preset fusion voiceprint feature list HB=(HB1, HB2, ..., HB e ,…,HB g ), e=1, 2,…, g; where HB e is the preset e-th fusion voiceprint feature, g is the number of preset fusion voiceprint features; HB e It is obtained by fusing speech clips of at least two known users.

[0023] Furthermore, HB is obtained by the following steps: H660, obtain the voice segments of each known user to obtain a known user voice segment list HE = (HE1, HE2, ..., HE h ,…,HE k ), h=1, 2, …, k; where HE h is the speech clip of the hth known user, and k is the number of known users.

[0024] H661, fuse at least two known user voice segments in HE to obtain a fused voice segment list HF = (HE1, HE2, ..., HE e ,…,HE g ); Among them, HE e is the e-th fused speech segment obtained by fusion; g=2 k -k-1.

[0025] H662, perform fusion voiceprint feature extraction on each fusion speech segment in HF to obtain HB.

[0026] In this embodiment, for example, if there are three known users, there are three ways to combine them in pairs and one way to combine them in three pairs, for a total of four combinations. The voice segments of two known users are fused, and then the voice segments of three known users are fused, and so on, until the voice segments of k known users are fused, thereby obtaining g fused voice segments. The fusion method can be the superposition of voice segments to simulate the actual scenario of multiple people speaking at the same time. Voiceprint features are extracted from the fused voice segments to obtain HB.

[0027] The above HB can be acquired in advance and stored in a preset storage location, and can be directly called when needed, so as to improve the efficiency of human voice recognition.

[0028] H620, get HC x The first similarity with each fused voiceprint feature in HB to obtain the first similarity list HD x =(HD x，1 , HD x，2 ,…,HD x，e ,…,HD x，g ); Among them, HD x，e HC x With HB e The first similarity between .

[0029] H630, according to HD x , get the first target similarity HD x '=MAX(HD x ); where MAX() is the preset maximum value function.

[0030] H640, if HD x '≥H', then determine HC x The corresponding user is the known user corresponding to H'.

[0031] In this embodiment, HD x '≥H', means HD x 'The sound characteristics of the corresponding fusion speech segment and HC x The sound characteristics of HC are the same, so it can be determined that x The corresponding user is the known user corresponding to H'; for example: the known users corresponding to H' are user A and user B, then HC x The corresponding users are user A and user B, that is, HC x The generated audio clips are for users A and B speaking simultaneously.

[0032] Furthermore, after step H640, the method further includes the following steps: H650, if HD x '<H', then determine HC x The corresponding user is a user other than the known user or includes a user other than the known user.

[0033] In this embodiment, if HD x '<H', indicates that the sound characteristics of each fused speech segment in HB are different from those in HC x The sound characteristics of the HC are different, then determine xThe corresponding users are other users other than known users or include other users other than known users; a prompt message can be generated to remind the management personnel that other users have entered the cabin, thereby realizing the identification and reminder of other users other than known users.

[0034] In this embodiment, first, the voiceprint features corresponding to each voice segment are obtained by voice segmentation and multi-dimensional voiceprint feature fusion extraction, and the voiceprint feature vector corresponding to each voice segment is compared with each preset voiceprint feature vector for similarity to obtain the maximum similarity. If the maximum similarity is greater than the preset voiceprint feature vector similarity threshold, the user corresponding to the currently identified voice segment is determined to be the user corresponding to the voiceprint feature vector with the maximum similarity. If the corresponding user cannot be identified, it may be that multiple people are speaking at the same time, and then re-determination is performed based on the preset fusion voiceprint feature vector list, thereby achieving the purpose of accurate voice recognition in a more complex environment.

[0035] Furthermore, after step H600, the method further includes the following steps: H700, extract audio features from each vocal segment in QC to obtain the audio feature vector list YA=(YA1, YA2, ..., YA x ,…,YA y ); among them, YA x For QC x The corresponding audio features.

[0036] In this embodiment, audio features can be extracted for each vocal segment in QC. The audio features include: Mel-frequency cepstral coefficients (MFCC), fundamental frequency trajectory and other acoustic parameters, which capture emotional information such as intonation and stress; corresponding audio feature vectors are constructed based on the above audio features; it should be noted that those skilled in the art can use existing audio feature construction methods according to actual needs to obtain the audio feature vector list YA corresponding to QC, which will not be elaborated here.

[0037] H800, obtain the text features corresponding to each vocal segment in QC to obtain the text feature vector YB=(YB1,YB2,…,YB x ,…,YB y ); Among them, YB x For QC x The corresponding text feature vector.

[0038] Furthermore, H800 includes the following steps: H810, QC xInput to a preset ASR model; wherein the ASR model includes an encoder and a decoder, the encoder is used to encode the text features of the input human voice segment to obtain a text feature vector; the decoder is used to decode the text feature vector output by the encoder.

[0039] In this embodiment, the vocal segment QC x Input to the preset automatic speech recognition (ASR) model, the encoder performs QC x It performs deep feature extraction to generate text feature vectors (such as latent representations of speech semantic distribution and contextual relevance), rather than directly outputting text. The encoder captures the time-frequency domain correlation features of speech using a convolutional neural network (CNN) or Transformer architecture.

[0040] H820, the text feature vector output by the decoder is determined as YB x .

[0041] The decoder receives the text feature vector output by the encoder and generates a text sequence through an attention mechanism or a recurrent neural network (RNN). The key point is that this step does not directly use the decoded text result, but uses the intermediate feature vector output by the decoder (such as the decoder hidden state or attention weight) as the text feature vector YB x The vector combines the acoustic characteristics of the speech with the semantic context information.

[0042] H900, concatenate YA and YB to obtain the speech and text feature vector list YC corresponding to QC = (YC1, YC2, ..., YC x ,…,YC y ); Among them, YC x For QC x Corresponding speech and text feature vector; YC x =(YA x , YB x ).

[0043] In this embodiment, the concatenation of the above-mentioned speech feature vector and text feature vector has the following beneficial effects: 1. Anti-ASR error interference: Traditional methods directly use ASR output text for semantic analysis. However, noise, accents, and specialized terminology in civil aviation scenarios can easily lead to ASR text errors (for example, misidentifying "FL310" as "FLT10"). This solution extracts the decoder's intermediate feature vector (YBx) to preserve the original semantic distribution of the speech rather than the erroneous text, preventing erroneous results from being passed to subsequent classification modules.

[0044] 2. Deep semantic expression: The decoder's intermediate feature vectors can encode the underlying semantic relationships in speech (such as the contextual logic of instructions and the complex meanings of professional terms). This makes them more suitable for parsing the complex intent of aviation conversations than simple text keywords. For example, the feature vector can capture the strong association between "MAYDAY" and emergency situations, even if the ASR text contains some spelling errors.

[0045] 3. Multi-granularity feature fusion: Characteristic vector (YB x ) and audio features (YA x ) can simultaneously utilize complementary information from acoustic sentiment (such as urgency and increased speech rate) and semantic intent (such as keyword weight). For example, when engine noise masks some audio features, the text feature vector can still infer the conversation type through semantic associations.

[0046] 4. Computational efficiency optimization: The joint training of the encoder and decoder gives the intermediate feature vectors a high information density. Compared with the traditional text processing process (ASR → text cleaning → feature extraction), the direct output of feature vectors reduces redundant calculation steps, improves real-time performance, and meets the millisecond-level response requirements of civil aviation.

[0047] For example: When QC x When the pilot issues the command "Descend to FL250" for a flight with engine noise: The encoder may generate slightly distorted feature vectors due to noise interference, but the decoder focuses on the key frequency bands through the attention mechanism, and the intermediate features of its output can still retain the strong correlation between "FL250" and the height adjustment instructions.

[0048] The subsequent classification model combines audio features (such as the falling tone trend) with the text feature vector to accurately determine the "flight instruction" type, rather than relying on potentially erroneous ASR text (such as the mistaken conversion of "Descend to FL2H0").

[0049] In this embodiment, the accuracy and robustness of conversation type classification in complex acoustic environments are significantly improved through deep semantic feature extraction and error isolation mechanism.

[0050] H1000 inputs YC into the preset classification model to obtain the dialogue type corresponding to each vocal segment in QC.

[0051] Furthermore, step H1000 includes the following steps: H1100, input YC into the preset classification model to obtain the confidence list set ω=(ω1,ω2,…,ω x ,…,ω y ); where ω x For QCx The corresponding confidence list; ω x =(ω x，1 ,ω x，2 ,…,ω x，α ,…,ω x，β ), α=1, 2,...,β; among them, ω x，α For QC x is the confidence of the αth preset dialogue type, and β is the number of preset dialogue types.

[0052] H1200, according to ω, determines the target confidence corresponding to each vocal segment in QC to obtain the target confidence list μ = (μ1, μ2, ..., μ x ,…,μ y ); where μ x For QC x The corresponding target confidence; μ x =MAX(ω x );MAX() is the preset maximum value function.

[0053] H1300, traverse μ, if μ x ≥μ', then μ x The corresponding preset dialogue type is determined to be QC x The corresponding dialogue type; μ' is the preset confidence threshold.

[0054] In this embodiment, low-confidence results are filtered through thresholds to avoid misclassification caused by noise interference or voice aliasing, especially to prevent non-critical conversations from being misclassified as high-risk instructions (such as mistakenly identifying small talk as an emergency broadcast). Only results are directly output for high-confidence segments to reduce the subsequent processing load and meet the real-time requirements of civil aviation scenarios.

[0055] Furthermore, after step H1300, the method further includes the following steps: H1400, if μ x <μ', and μ x The corresponding preset dialogue type is the preset conversation dialogue type, then obtain QC x Adjacent vocal clips.

[0056] If the low confidence segment μ x If the corresponding type is a conversation (such as a question and answer between a flight attendant and a passenger), the preceding and following adjacent segments (such as speech within ±1 second) are obtained.

[0057] H1410, QC x Splice with adjacent vocal clips to form QC x Corresponding spliced vocal segment QC x '.

[0058] QCx Splice with adjacent fragments to form QC x ', forming a longer time sequence of speech segments.

[0059] H1420, QC x 'Reclassify.

[0060] QC x 'Re-extract features and classify, and use the expanded context information to correct the results.

[0061] In this embodiment, conversational dialogues are usually continuous, and after splicing, they provide more complete semantic information (such as the relevance of questions and answers), solving the problem of missing fragment information caused by short-term noise or voice interruption; avoiding directly discarding low-confidence fragments, and automatically repairing the classification results through dynamic expansion of the window, thereby improving the recognition consistency of scenarios such as cabin or aircraft cabin conversations.

[0062] Furthermore, after step H1300, the method further includes the following steps: H1500, if μ x <μ', and μ x The corresponding preset dialogue type is the preset instruction dialogue type, then obtain QC x Corresponding keywords to get QC x Corresponding keyword list QE x =(QE x，1 , QE x，2 ,…,QE x，γ ,…,QE x，λ（x） ), γ=1, 2,..., λ(x); among them, QE x，γ For QC x Corresponding to the γth keyword, λ(x) is QC x The number of corresponding keywords.

[0063] When a certain segment (QC x )’s classification confidence μ x When the value is lower than the threshold μ' and the type corresponding to the highest confidence is the preset command dialogue type (such as pilot operation instructions, emergency broadcast), the system extracts all keywords from the speech recognition (ASR) results of the segment and generates a keyword list QE x For example, if QC x The ASR text is "Request descent to FL250 due turbulence", QE x It may contain keywords such as "Request", "descent", "FL250", "turbulence", etc.

[0064] H1510, traversing QEx , determine QE x The number NUM1 of preset instruction keywords.

[0065] Traversing QE x , count the number of preset command keyword libraries NUM1. The preset library contains civil aviation-specific terms (such as "MAYDAY", "FLXXX", "Emergency", etc.), NUM1 represents QC x The number of words related to key operations in .

[0066] H1520, if NUM1 / λ(x) ≥ QR, then determine QC x The corresponding dialogue type is the command dialogue type; otherwise, it is determined manually; QR is the preset keyword quantity weight threshold.

[0067] Calculate the instruction keyword ratio (NUM1 / λ(x)). If the ratio is greater than or equal to the preset threshold QR (such as 40%), QC is determined. x It is a command dialogue type; otherwise, it will be transferred to manual review and confirmation.

[0068] In this embodiment, the above method has the following beneficial effects: 1. Anti-noise interference instruction fault-tolerant recognition: In scenarios with engine noise (interference in the 80Hz-2kHz frequency band) or voice aliasing, ASR may produce some erroneous text (e.g., mistranslating "FL250" as "FL2H0"). By counting the percentage of command keywords rather than relying on full text matching, the system can tolerate local recognition errors. For example, if "FL250" in QCx is mistakenly recognized as "FL2H0," but other keywords such as "descent" and "turbulence" are still correctly extracted, NUM1 / λ(x) may reach 50% (QR = 40%), and the system can still determine the command type, avoiding missing critical operation instructions.

[0069] 2. Double protection mechanism with safety as the priority: Implementing strong keyword verification for low-confidence instruction fragments complies with the civil aviation safety principle of "preferring false alarms to false negatives". x =0.65, μ'=0.7). As long as the command keyword ratio meets the standard, the system will still prioritize it as a command type to ensure that high-risk operations such as emergency broadcasts and flight mode switching are captured in a timely manner.

[0070] 3. Precise intervention of manual review: When the keyword ratio is insufficient (such as QC xContains a lot of non-critical dialogue interference), and is transferred to manual intervention for review to avoid misjudgment by the automated system due to noise or semantic ambiguity (such as misclassifying a passenger's question "Do you want to make an emergency landing" as a real instruction).

[0071] 4. Dynamic weight thresholds adapt to scenario requirements: The threshold QR can be flexibly adjusted according to different scenarios. For example: Cockpit commands: QR is set low (e.g. 30%) because commands are usually short and keyword-intensive (e.g. "ClimbFL300").

[0072] Cabin announcement: The QR setting is higher (e.g. 50%) because it may contain service terms (e.g. "Please fasten your seat belts" and "Emergency evacuation" coexisting).

[0073] Confidence reflects the model's grasp of overall features, while keyword verification focuses on the semantic core. The combination of the two avoids the one-sidedness of a single criterion and forms redundant verification, meeting the fault-tolerant design requirements of aviation safety systems. Keyword analysis is only triggered for low-confidence instruction fragments rather than full processing, saving computing resources and meeting the real-time requirements of airborne systems.

[0074] Example 2: When identifying the user corresponding to a vocal clip, the application scenario is an aircraft cabin. However, a vocal clip may be generated by multiple people speaking simultaneously. To identify users other than the known user, the following method is provided: Q100, obtain the voice segment to be recognized with a preset length.

[0075] Q200, performing preset coarse-grained human voice segment recognition on the speech segment to be recognized, so as to obtain a coarse-grained human voice segment list QA = (QA1, QA2, ..., QA u ,…,QA v ), u=1, 2,...,v; among them, QA u is the u-th coarse-grained vocal segment corresponding to the speech to be recognized, and v is the number of coarse-grained vocal segments corresponding to the speech to be recognized.

[0076] Furthermore, step Q200 includes the following steps: Q210, divide the speech segment to be recognized into several continuous speech frames of the same duration to obtain a speech frame list SA = (SA1, SA2, ..., SA i ,…,SA n ), i=1, 2,...,n; among them, SA i is the i-th speech frame obtained by dividing the speech segment to be recognized, and n is the number of speech frames obtained by dividing the speech segment to be recognized.

[0077] Q220, input SA into the preset VAD model to obtain the voice confidence list SA'=(SA'1, SA'2, ..., SA' i ,…,SA' n ); where SA' i for SA i The confidence level of the vocal frame.

[0078] Q230, perform a first sliding window operation on SA'; wherein the first sliding window includes m voice confidences.

[0079] Q240, each time the first sliding window is slid, the number QN of voice confidences in the first sliding window that is greater than a preset first voice confidence threshold is obtained.

[0080] Q250, if QN / m>η1, then determine the speech frame corresponding to the first human voice confidence in the corresponding first sliding window as a human voice frame; otherwise, determine it as a non-human voice frame; wherein η1 is a preset first weight.

[0081] Q260: Determine a coarse-grained vocal segment corresponding to the speech to be recognized based on the vocal frames and non-vocal frames in the SA.

[0082] In this embodiment, the method for obtaining the coarse-grained vocal segment list QA is the same as that used in the first embodiment, and is not described in detail here.

[0083] Furthermore, step Q260 includes the following steps: Q261, set the human voice frame in SA as the first preset character and the non-human voice frame as the second preset character to obtain the coarse-grained character list ZA corresponding to SA = (ZA1, ZA2, ..., ZA j ,…,ZA n+1-m ), j=1, 2,…, n+1-m; ZA j for SA j The corresponding preset character, n+1-m is the number of times the first sliding window slides; the first preset character is different from the second preset character.

[0084] Q262, traverse ZA, divide a plurality of consecutive first preset characters in ZA into a group, to obtain a coarse-grained first preset character group list LA = (LA1, LA2, ..., LA p ,…,LA q ), p = 1, 2, ..., q; where LA p is the pth first preset character group obtained by dividing the first preset character in ZA, and q is the number of first preset character groups obtained by dividing the first preset character in ZA.

[0085] Q263, traverse LA, if NU p >NU', then LA p The corresponding recognized speech segment is determined to be a coarse-grained human voice segment; p For LA p NU' is the first preset threshold.

[0086] In this embodiment, the methods in the above steps Q261-Q263 are the same as the methods in steps S710-S730 in the first embodiment, and are not repeated here.

[0087] Furthermore, after step Q262 and before step Q263, the method further includes the following steps: Q21, traverse LA, if NU p >MU', then LA p The first preset number of second preset characters that are consecutive thereafter are modified into the first preset character; wherein MU' is a preset second threshold.

[0088] Q22, add the modified first preset number of first preset characters to LA p To obtain the modified first preset character group list LA' corresponding to LA = (LA'1, LA'2, ..., LA' p ,…,LA' q ); Among them, LA' p For LA p The corresponding modified first preset character group.

[0089] Q23, traverse LA, if SU p >NU', then LA' p The corresponding recognized speech segment is determined to be a coarse-grained human voice segment; SU p for LA' p The number of the first preset characters in .

[0090] In this embodiment, the above steps Q21-Q23 are the same as the methods in steps S721-S723 in the first embodiment, and are not repeated here.

[0091] Q300, obtain the text list set QB corresponding to each coarse-grained vocal segment in QA = (QB1, QB2, ..., QB u ,…,QB v );QB u For QA u Corresponding text list; QB u =(QB u，1 , QB u，2 ,…,QB u，a ,…,QBu，f（u） ), a=1,2,…,f(u); QB u，a For QA u Corresponding to the a-th text, f(u) is QA u The number of corresponding characters.

[0092] In this embodiment, each coarse-grained vocal segment in QA can be input into a preset ASR model to obtain a text list set QB corresponding to each coarse-grained vocal segment in QA; it should be noted that each text is obtained with a timestamp, which indicates the time point corresponding to the corresponding text on the coarse-grained vocal segment.

[0093] Q400, performing preset fine-grained vocal segment recognition on the speech segment to be recognized, to obtain a fine-grained vocal segment list QC = (QC1, QC2, ..., QC x ,…,QC y ), x=1, 2, …, y; where QC x is the xth fine-grained vocal segment corresponding to the speech to be recognized, and y is the number of fine-grained vocal segments corresponding to the speech to be recognized.

[0094] Furthermore, step Q400 includes the following steps: Q410 , perform a second sliding window operation on SA′; wherein the second sliding window includes m′ individual voice confidences; wherein m′<m.

[0095] Q420, each time the first sliding window is slid, the number QN' of voice confidences in the first sliding window that is greater than a preset second voice confidence threshold is obtained; wherein the second voice confidence is greater than the first voice confidence.

[0096] Q430, if QN' / m'>η2, the speech frame corresponding to the first human voice confidence in the corresponding second sliding window is determined to be a human voice frame; otherwise, it is determined to be a non-human voice frame; wherein η2 is a preset second weight; η2>η1.

[0097] Q440: Determine a fine-grained vocal segment corresponding to the speech to be recognized based on the vocal frames and non-vocal frames in the SA.

[0098] In this embodiment, in the above steps, the first sliding window in the method steps for obtaining coarse granularity is modified to a second sliding window, and the second sliding window includes m' vocal confidences, that is, the length of the second sliding window is smaller; the first vocal confidence is adjusted to a larger second vocal confidence; the preset first weight is adjusted to a larger preset second weight; through the above parameter adjustment, the obtained vocal segment is shorter, that is, a fine-grained vocal segment.

[0099] Furthermore, step Q440 includes the following steps: Q441, set the human voice frame in SA as the first preset character and the non-human voice frame as the second preset character to obtain the fine-grained character list ZA'=(ZA'1, ZA'2, ..., ZA' r ,…,ZA' n+1-m’ ), r=1, 2,..., n+1-m'; ZA' r for SA r The corresponding preset character, n+1-m' is the number of times the second sliding window slides.

[0100] Q442, traverse ZA', divide a plurality of consecutive first preset characters in ZA' into a group, so as to obtain a fine-grained first preset character group list LA' = (LA'1, LA'2, ..., LA' b ,…,LA' s ), b=1, 2,…, s; where LA b is the bth first preset character group obtained by dividing the first preset character in ZA, and s is the number of first preset character groups obtained by dividing the first preset character in ZA′.

[0101] Q443, traverse LA', if MU b >MU', then LA' b The corresponding recognized speech segment is determined to be a fine-grained human voice segment; among them, MU b for LA' b MU' is the number of the first preset characters; MU' is the preset second threshold; MU'>NU'.

[0102] In this embodiment, the method steps after parameter adjustment are the same as the corresponding method steps in embodiment 1 and are not repeated here.

[0103] Q500, traverse QB, if QB u，a If the corresponding timestamp is within the time range of any fine-grained vocal segment in QC, QB is retained. u，a ; Otherwise, delete QB u，a .

[0104] In this embodiment, the fine-grained vocal segments will filter out some noise, but at the same time, some very short vocal segments will be mistakenly identified, which may be generated by environmental noise. Therefore, by comparing the timestamp of each word with the time range of the fine-grained vocal segment, only the words with timestamps within the time range corresponding to the granular vocal segment are retained to filter out invalid words, thereby accurately identifying the words corresponding to the vocal segment.

[0105] Furthermore, step Q230 includes the following steps: Q231, obtain the first preset value SN=1.

[0106] Q232, control the first sliding window so that the first voice confidence in the first sliding window is SA' SN .

[0107] Q233, if SN<n+1-m, obtain SN=SN+1 and enter Q232; otherwise, jump out of the current processing.

[0108] In this embodiment, the control method of the first sliding window is the same as the control method of the first sliding window in the first embodiment, and is not described in detail here.

[0109] In this embodiment, by constructing a coarse-grained and fine-grained collaborative recognition architecture, non-human voice interference is quickly screened out at the coarse-grained level, significantly reducing the system computing power consumption; accurate voice analysis is implemented at the fine-grained level, and the double verification mechanism greatly improves the recognition accuracy of key instructions; timestamp dynamic calibration technology is used, combined with the time range corresponding to the coarse-grained human voice segment and the timestamp corresponding to each text, to filter out invalid or misrecognized text, improve the accuracy of text recognition, and effectively solve the problem of command truncation caused by blurred voice segment boundaries in traditional methods; thereby avoiding the occurrence of semantic deviation due to environmental noise interference and improving the safety of aviation operations.

[0110] Furthermore, the model in this embodiment needs to be trained before use. Training the speech model requires annotating the speech samples. Traditional annotation methods are usually manual annotation, which has low annotation efficiency. Therefore, before step R100, the method further includes the following steps: R010, obtain the sub-audio samples of each channel corresponding to the audio sample to obtain a sub-audio sample list TA = (TA1, TA2, ..., TA σ ,…,TA θ ),σ=1,2,…,θ; among them, TA σ is the sub-audio sample of the σth channel corresponding to the audio sample, and θ is the number of channels corresponding to the audio sample.

[0111] In this embodiment, the original audio samples are separated by channel to generate a sub-audio sample list TA. Each sub-audio sample corresponds to an independent channel (such as the cockpit main microphone channel, the co-pilot headphone channel, the cabin broadcast channel, etc.), ensuring the independence of the multi-channel data.

[0112] R020, randomly select a sub-audio sample TA' from TA.

[0113] A sub-audio sample TA' is randomly selected from TA as the reference channel for annotation. Random selection avoids bias towards a specific channel (such as the channel with the strongest signal), enhancing the system's adaptability to different channel qualities.

[0114] R030, obtain each vocal segment corresponding to TA' to obtain a vocal segment list TB corresponding to TA' = (TB1, TB2, ..., TB a2 ,…,TB b2 ), a2=1, 2, …, b2; where TB a2 is the a2th vocal segment corresponding to TA', and b2 is the number of vocal segments corresponding to TA'.

[0115] Perform voice activity detection (VAD) and segmentation on TA' to extract a list of valid vocal segments, TB. Each vocal segment represents a continuous vocal interval (e.g., a single pilot instruction or a flight attendant announcement), excluding silent or noisy segments.

[0116] R040, obtain the time range and dialogue type of each vocal segment in TB to obtain the time range dialogue type group list TC corresponding to TA' = (TC1, TC2, ..., TC a2 ,…,TC b2 ); Among them, TC a2 TB a2 Corresponding time range dialogue type group; TC a2 =(TC a2，1 , TC a2，2 );TC a2，1 TB a2 Corresponding time range, TC a2，2 TB a2 The corresponding conversation type.

[0117] Furthermore, TC a2，2 Determine this by following these steps: R041, for TB a2 Perform audio feature extraction to obtain TB a2 The corresponding audio feature vector list TG.

[0118] R042, Get TB a2 The corresponding text feature TL.

[0119] R043, splicing TG and TL to obtain TB a2 The corresponding speech-to-text feature vector TU=(TG, TL).

[0120] R044, input TU into the preset classification model to obtain TB a2 The corresponding conversation type.

[0121] Furthermore, TL is obtained by the following steps: R41, TB a2 Input to a preset ASR model; wherein the ASR model includes an encoder and a decoder, the encoder is used to encode the text features of the input human voice segment to obtain a text feature vector; the decoder is used to decode the text feature vector output by the encoder.

[0122] R42, determines the text feature vector output by the decoder as TL.

[0123] Furthermore, step R044 includes the following steps: R01, input TU into the preset classification model to obtain the confidence list TV corresponding to TU = (TV1, TV2, ..., TV α ,…,TV β ), α=1, 2,...,β; among them, TV α is the confidence that TU is the αth preset dialogue type, and β is the number of preset dialogue types.

[0124] R02, based on TV, determines the target confidence TU'=MAX(TU) corresponding to TU; MAX() is the preset maximum value function.

[0125] R03, if TU'≥μ', the preset conversation type corresponding to TU' is determined as the conversation type corresponding to TU; μ' is a preset confidence threshold.

[0126] R04, if TU'<μ', and the preset dialogue type corresponding to TU' is the preset conversation dialogue type, then obtain TB a2 Adjacent vocal clips.

[0127] R05, TB a2 Splice with adjacent vocal clips to form TB a2 Corresponding spliced vocal segment TB a2 '.

[0128] R06, for TB a2 'Reclassify.

[0129] Furthermore, the method further comprises the following steps: If TU'<μ', and the default dialogue type corresponding to TU' is the default command dialogue type, then obtain TB a2 Corresponding keywords to get TB a2 The corresponding keyword list.

[0130] Identify TB a2 The corresponding keyword list contains the number NUM2 of preset instruction keywords.

[0131] If NUM2 / NUM3 ≥ QR, then determine QC x The corresponding dialogue type is the command dialogue type; otherwise, it is determined manually; NUM3 is TB a2 The number of keywords in the corresponding keyword list; QR is the preset keyword quantity weight threshold.

[0132] In this embodiment, the above-mentioned method for classifying conversation types is the same as the method for classifying conversation types in the fourth embodiment, and is not described in detail here.

[0133] R050: Synchronously mark the sub-audio samples in TA except TA' according to the preset dialogue type and TC.

[0134] Furthermore, step R050 includes the following steps: R051, traverse TC, if TC a2，2 For the default conversation type, TC a2 Determine the target time range.

[0135] R052: Label each sub-audio sample in TA except TA' with the corresponding target time range and the preset dialogue type.

[0136] In this embodiment, the annotation information (TC) of the main channel TA' is synchronously mapped to all other channels (subsamples in TA except TA'). That is, within the same time range, the corresponding segments of other channels automatically inherit the same dialogue type label.

[0137] In this embodiment, the technical effects of the above method are as follows: 1. A leap in annotation efficiency: Only a single main channel (TA') needs to be manually or finely labeled, and other channels automatically inherit the label through time synchronization, avoiding the repetitive work of channel-by-channel labeling in traditional methods. The efficiency improvement is proportional to the number of channels θ (for example, when θ=5 channels, the efficiency is improved by nearly 5 times).

[0138] 2. Multi-channel spatiotemporal alignment: Timestamp synchronization solves the propagation delay problem of multi-channel audio (for example, cockpit instructions are played with a 0.5-second delay in the cabin broadcast channel), ensuring that the marks of the same voice event in different channels are strictly aligned, providing a consistent data foundation for voice analysis (such as command tracing and event replay).

[0139] 3. Robustness to noisy scenes: The random selection of the primary channel implicitly prioritizes high-quality channels (such as low-noise pilot headset microphones), and the annotations generated based on them are synchronized to the noisy secondary channel (such as the cabin ambient microphone), avoiding the problem of missing or mislabeled segments when the noisy channel is independently annotated.

[0140] 4. Adaptability to civil aviation scenarios: For concurrent conversations between multiple roles in the cabin (such as overlapping pilot-to-tower conversations and flight attendant broadcasts), the main channel is accurately labeled to guide the automatic association of auxiliary channels, resolving the conflicts in conversation types caused by independent channel processing in traditional methods (such as the same instruction being mistakenly labeled as different types in different channels).

[0141] 5. Labeling consistency assurance: All audio channels share the same annotation benchmark, eliminating subjective differences in manual annotation. This is particularly suitable for civil aviation voice recordings that require strict compliance (such as black box data analysis and accident investigations), ensuring that the annotation results are traceable and verifiable.

[0142] Furthermore, after step R052, the method further includes the following steps: R053, for any target time range, randomly select a labeled sub-audio clip from θ channels as a model training sample.

[0143] In this embodiment, after completing the labeling of the target time range (T511), for each labeled segment (such as the "urgent command" with timestamps 10.2s-12.5s), the system randomly selects a labeled sub-audio segment from the θ channels (such as the data from the co-pilot microphone channel) as a sample for subsequent model training.

[0144] Randomly select samples from different channels (e.g., the captain's microphone this time, the cabin announcement microphone next time) to ensure that the training data covers the acoustic characteristics of multiple devices and locations (e.g., frequency response differences and noise distribution between different microphones), improving the model's generalization to channel differences. This avoids the model's bias towards channel-specific characteristics (e.g., formants in a specific frequency band) caused by rigidly using a single channel (e.g., always selecting the captain's microphone with the highest signal-to-noise ratio), thereby enhancing the model's robustness to low-quality channels (e.g., the cabin ambient microphone). Only the primary channel (TA') needs to be labeled. Corresponding clips from other channels are labeled synchronously, but only one of them is randomly selected during training. This reduces the impact of duplicate samples on training efficiency and balances data size and computational cost. To accommodate asynchronous multi-channel recording scenarios in the cabin (e.g., delays between headset and ambient microphones), the model learns to tolerate propagation delays (typically 200-500ms) by randomly selecting implicit time alignment (e.g., timestamp differences between different channels for the same event are naturally captured in the training data).

[0145] Furthermore, step R050 includes the following steps: R053, determine the audio segments of each sub-audio sample in TA except TA' within the target time range as vocal segments, so as to obtain the vocal segment list set TD = (TD1, TD2, ..., TDσ ,…,TD θ ); Among them, TD σ For TA σ Corresponding vocal clip list; TD σ =(TD σ，1 , TD σ，2 ,…,TD σ，a2 ,…,TD σ，b2 );TD σ，a2 For TA σ The corresponding a2th vocal segment.

[0146] Based on the time range of the main channel TA' (TC a2,1 ), from all other channels (TA σ ) Extract the audio segments of the corresponding time period to form a multi-channel vocal segment list set TD. Each TD σ,a2 Represents the vocal segment of the σth channel in the a2th time range.

[0147] R054, input TD into the preset ASR model to obtain the text confidence corresponding to each human voice segment, and then obtain the text confidence list corresponding to the audio sample ψ = (ψ1, ψ2, ..., ψ σ ,…,ψ θ ); where ψ σ For TD σ The corresponding text confidence list; ψ σ =(ψ σ，1 , ψ σ，2 ,…,ψ σ，a2 ,…,ψ σ，b2 );ψ σ，a2 For TA σ The text confidence of the corresponding a2th vocal segment.

[0148] TD all vocal clips σ,a2 Input the ASR model to obtain the text recognition confidence ψ of each segment σ,a2 (0-1 range), generates the confidence matrix ψ of the entire channel. The confidence reflects the degree of ASR's confidence in the accuracy of the text segment.

[0149] R055, based on ψ, determine the maximum text confidence corresponding to each target time range to obtain the maximum text confidence list ψ max =(ψ max，1 , ψ max，2 ,…,ψ max，a2 ,…,ψ max，b2 ); where ψ max，a2 is the maximum text confidence list corresponding to the a22th target time range; ψ max，a2 =MAX(ψ1，a2 , ψ 2，a2 ,…,ψ σ，a2 ,…,ψ θ，a2 );MAX() is the preset maximum value function.

[0150] For each target time range, compare the confidence scores ψ of all channels in that time period σ,a2 , select the maximum value ψ max，a2 =MAX(ψ 1，a2 , ψ 2，a2 ,…,ψ σ，a2 ,…,ψ θ，a2 ); This step ensures that only the most reliable channel data is retained in each time period.

[0151] R056, change ψ max The vocal segment corresponding to each maximum text confidence is determined as the target vocal segment; wherein the target vocal segment is used for model training.

[0152] The optimal vocal segment corresponding to each target time period (i.e., the segment with the most accurate text recognition) is marked as the target vocal segment, which serves as a high-quality data source for subsequent model training.

[0153] The above method has the following technical effects: 1. Data quality optimization: Through the multi-channel confidence competition mechanism, the channel segments with the highest recognition quality in each time period are automatically screened and identified, eliminating low-quality samples caused by noise and equipment differences, thereby improving the signal-to-noise ratio of the training data.

[0154] 2. Cross-channel redundancy utilization: By leveraging the inherent advantages of multi-channel systems, even if one channel fails completely (such as a microphone failure), data integrity can still be guaranteed through valid segments from other channels, significantly improving system robustness.

[0155] 3. Adaptive channel selection: Dynamically select the optimal sound channel for different time periods (such as cockpit microphones during instruction periods and ceiling microphones during cabin broadcast periods), breaking through the limitations of fixed microphone deployment and being particularly suitable for the dynamically changing acoustic environment in civil aviation scenarios.

[0156] 4. Computational efficiency balance: Only deep training is performed on key fragments, which reduces computing power by 30-40% compared to full data training, while ensuring that model performance does not degrade.

[0157] 5. Semi-supervised learning compatible: High-confidence fragments can be directly used as labeled data, and low-confidence fragments can trigger manual review or downgrade use to form a hierarchical data utilization strategy.

[0158] The method in this embodiment achieves an optimal balance between data quality, system fault tolerance, and computational efficiency through a multi-channel confidence competition mechanism, and is particularly suitable for voice processing scenarios with high reliability requirements, such as civil aviation.

[0159] In this embodiment, the efficiency and consistency of audio labeling are significantly improved through a multi-channel collaborative labeling mechanism: based on randomly selected main channel human voice clips and their dialogue types, audio clips in the corresponding time range in other channels are automatically and synchronously labeled to ensure that the labeling results of the same voice event in different channels are temporally and spatially aligned, avoiding the repetitive work of manual channel-by-channel labeling; through the precise labeling of the main channel, the automatic labeling of the auxiliary channel is guided, effectively solving the problem of missed labeling of clips in multi-channel audio due to noise interference or signal attenuation, which is particularly suitable for scenarios with differentiated audio quality collected by microphones at different positions in the cabin; combined with dialogue type recognition technology, the system can automatically identify and associate the same dialogue content across channels (such as the delay matching of cockpit instructions and cabin broadcasts), providing better training samples for subsequent model training, and greatly improving the effect of model training.

[0160] Example 3: The following will refer to Figure 1 The flowchart of the method for identifying a person based on an audio sample is shown, which introduces a method for identifying a person based on an audio sample.

[0161] The method for identifying a person based on an audio sample comprises the following steps: G100, obtain each vocal segment corresponding to the speech segment to be recognized to obtain a vocal segment list QC = (QC1, QC2, ..., QC x ,…,QC y ), x=1, 2, …, y; where QC x is the xth vocal segment corresponding to the speech to be recognized, and y is the number of vocal segments corresponding to the speech to be recognized.

[0162] In this embodiment, it should be noted that the voice segments to be recognized include vocal segments and non-vocal segments, and the vocal segments in the voice segments to be recognized can be recognized to obtain the vocal segment list QC; those skilled in the art can use the existing vocal segment recognition method according to actual needs to recognize the voice segments to be recognized to obtain the vocal segment list QC, which will not be elaborated here.

[0163] G200, obtain the voiceprint feature vector corresponding to each human voice segment in QC to obtain the voiceprint feature vector list HC corresponding to QC = (HC1, HC2, ..., HC x ,…,HC y ); Among them, HC x For QC xThe corresponding voiceprint feature vector.

[0164] In this embodiment, voiceprint features are extracted from each segmented vocal segment to generate a corresponding voiceprint feature vector list HC; voiceprint features include: fundamental frequency, formant, MFCC parameters, etc. Based on the above voiceprint features, corresponding voiceprint feature vectors can be generated to provide a data basis for subsequent identity matching.

[0165] G300 obtains each preset voiceprint feature vector to obtain a preset voiceprint feature vector list HA=(HA1, HA2, ..., HA c ,…,HA d ), c = 1, 2, ..., d; where HA c is the voiceprint feature vector of the cth user, and d is the number of users.

[0166] In this embodiment, a voiceprint feature list HA of pre-registered legitimate users (such as pilots and crew members) is obtained to form a reference library for system comparison.

[0167] G400, get HC x The maximum similarity HX with each preset voiceprint feature in HA.

[0168] HC x Compare each voiceprint with the preset library HA one by one, calculate the similarity and extract the maximum value HX. This step quantifies HC through algorithms (such as dynamic time warping or neural network embedding) x The degree of matching with each preset user.

[0169] G500, if HX ≥ H', the user corresponding to HX is determined as QC x Corresponding users, to obtain the identified user list GA = (GA1, GA2, ..., GA C1 ,…,GA d1 ), c1=1, 2, …, d1; where GA C1 is the c1th user identified, and d1 is the number of identified users.

[0170] If the maximum similarity HX exceeds a preset threshold H', the segment is determined to belong to the user corresponding to HX and is added to the list of identified users GA. The threshold H' is dynamically adjusted based on the ambient noise and false positive rate to ensure a high-confidence match.

[0171] G600, if HX<H', then QC x Determined to be an unrecognized vocal segment; enter G700.

[0172] If the HX score does not reach the threshold, the fragment is marked as unidentified and enters the retrospective analysis process. This avoids direct discarding of low-quality fragments and preserves the possibility of subsequent remediation.

[0173] G700, according to the corresponding vocal segment of each user in QC in GA, determines QC x The corresponding user.

[0174] Furthermore, step G700 includes the following steps: G710: Determine the voice segment corresponding to each user in GA from QC.

[0175] In this embodiment, GA is an identified user, and the voice segment corresponding to each identified user can be determined from QC.

[0176] G720, fuse at least two vocal segments corresponding to each user in GA to obtain an initial fused vocal segment list GB = (GB1, GB2, ..., GB e1 ,…,GB δ ), e1=1, 2, …, δ; where GB e1 is the e1th initial fused vocal segment obtained by fusion, δ is the number of initial fused vocal segments obtained by fusion; δ=2 d1 -d1-1.

[0177] In this embodiment, for example, if three users are identified, there are three ways to combine them in pairs and one way to combine them into three groups, for a total of four combinations. The voice segments of two identified users are fused, and then the voice segments of three identified users are fused, and so on, until the voice segments of d1 users are fused, thereby obtaining δ initial fused voice segments. The fusion method can be the superposition of voice segments to simulate the actual scenario of multiple people speaking at the same time.

[0178] G730, obtain the initial voiceprint feature vector corresponding to each initial fused voice segment in GB to obtain the initial voiceprint feature vector list GC=(GC1, GC2, ..., GC e1 ,…,GC δ ); Among them, GC e1 GB e1 The corresponding initial voiceprint feature vector.

[0179] In this embodiment, feature vectors may be extracted from each initial fused vocal segment in the GB to obtain the GC.

[0180] G740, Get QC x The maximum similarity GD between the voiceprint feature vector and each initial voiceprint feature vector in GC x .

[0181] G750, if GD x ≥H', then GD xThe corresponding at least two identified users are determined as QC x The corresponding user.

[0182] In this embodiment, if GD x ≥H', indicating QC x Corresponding user voice features and GD x The corresponding voice features are the same, GD x The corresponding at least two identified users are determined as QC x The corresponding user; thereby achieving the purpose of identifying the identity of people when multiple people are speaking at the same time.

[0183] It should be noted that the above steps will only be performed if the vocal segment cannot be completely recognized. That is, the above initial fused vocal segment list GB is not pre-stored, but is fused when needed, thereby reducing storage pressure.

[0184] In this embodiment, through segmentation and independent voiceprint extraction, individual features in the mixed speech of multiple people are effectively separated, identity confusion caused by spectral overlap is reduced, and the target speaker resolution capability is improved; combined with the dynamic adjustment mechanism of the threshold H', feature distortion caused by engine noise or instantaneous interference is suppressed, the false positive rate is reduced, and the stability of recognition in high-noise environments is ensured; contextual association analysis is performed on unrecognized segments, and the spatiotemporal distribution characteristics of identified users are used to supplement missing identity information, avoiding the loss of key commands due to instantaneous interference and enhancing the fault tolerance of the system; support is provided for layer-by-layer parsing of multiple identities from mixed voice streams, meeting the security requirements of civil aviation scenarios requiring collaborative confirmation of operations by multiple people (such as emergency protocol activation) and ensuring the integrity of procedure execution; hierarchical processing (independent matching first, context completion later) avoids repeated calculation of the full data, reducing the real-time processing load while ensuring accuracy, and meeting the millisecond-level response requirements of aviation communications.

[0185] Furthermore, after step G750, the method further includes the following steps: G760, if GD x <H', then determine QC x The corresponding users include other users outside of GA.

[0186] In this embodiment, if GD x <H', indicating QC x It cannot match any of the initial fusion vocal segments in GB. At this point, it can be determined that QC x The corresponding user is other than the known user, or QC xThere may be other users besides the known users among the corresponding multiple users, thereby achieving the purpose of identifying other users; after identifying other users besides the known users, prompt information may be generated to remind the management personnel that there are other people in the cabin.

[0187] In this embodiment, the collaborative mechanism of speech segmentation and dynamic voiceprint matching is used to effectively cope with the challenge of identity recognition of multiple people's mixed voices: a hierarchical voiceprint feature extraction strategy is adopted to suppress environmental noise interference while retaining individual biometric characteristics, significantly improving the target voiceprint separation accuracy in high-noise scenarios; through iterative comparison of the preset voiceprint library and adaptive backtracking analysis of unrecognized segments, layer-by-layer analysis of multiple identities in mixed speech is achieved to avoid identity omissions due to instantaneous interference or multiple people's concurrency; combined with the context association model, the system can dynamically correct the judgment threshold of unmatched segments based on the voiceprint distribution of the recognized segments, thereby enhancing the robustness and continuity of identity recognition in complex acoustic environments, and meeting the security authentication requirements of multi-personnel collaborative operations in civil aviation scenarios.

[0188] Furthermore, before step G100, the method further includes the following steps: G010, obtain the sub-audio samples of each channel corresponding to the audio sample to obtain a sub-audio sample list TA = (TA1, TA2, ..., TA σ ,…,TA θ ),σ=1,2,…,θ; among them, TA σ is the sub-audio sample of the σth channel corresponding to the audio sample, and θ is the number of channels corresponding to the audio sample.

[0189] G020, randomly select a sub-audio sample TA' from TA.

[0190] G030, obtain each vocal segment corresponding to TA' to obtain a list of vocal segments corresponding to TA' TB = (TB1, TB2, ..., TB a2 ,…,TB b2 ), a2=1, 2, …, b2; where TB a2 is the a2th vocal segment corresponding to TA', and b2 is the number of vocal segments corresponding to TA'.

[0191] G040, obtain the time range and dialogue type of each vocal segment in TB to obtain the time range dialogue type group list TC = (TC1, TC2, ..., TC a2 ,…,TC b2 ); Among them, TC a2 TB a2 Corresponding time range dialogue type group; TC a2 =(TC a2，1 , TCa2，2 );TC a2，1 TB a2 Corresponding time range, TC a2，2 TB a2 The corresponding conversation type.

[0192] G050, based on the preset dialogue type and TC, synchronously mark the sub-audio samples in TA except TA'.

[0193] Furthermore, step G050 includes the following steps: G051, traverse TC, if TC a2，2 For the default conversation type, TC a2 Determine the target time range.

[0194] G052: Label each sub-audio sample in TA except TA' with the corresponding target time range and the preset dialogue type.

[0195] Furthermore, step R050 includes the following steps: G053, determine the audio segments of each sub-audio sample in TA except TA' within the target time range as vocal segments, so as to obtain the vocal segment list set TD = (TD1, TD2, ..., TD σ ,…,TD θ ); Among them, TD σ For TA σ Corresponding vocal clip list; TD σ =(TD σ，1 , TD σ，2 ,…,TD σ，a2 ,…,TD σ，b2 );TD σ，a2 For TA σ The corresponding a2th vocal segment.

[0196] G054, input TD into the preset ASR model to obtain the text confidence corresponding to each human voice segment, and then obtain the text confidence list corresponding to the audio sample ψ = (ψ1, ψ2, ..., ψ σ ,…,ψ θ ); where ψ σ For TD σ The corresponding text confidence list; ψ σ =(ψ σ，1 , ψ σ，2 ,…,ψ σ，a2 ,…,ψ σ，b2 );ψ σ，a2 For TA σ The text confidence of the corresponding a2th vocal segment.

[0197] G055, obtain the maximum text confidence corresponding to each target time range based on ψ to obtain the maximum text confidence list ψ max =(ψ max，1 , ψ max，2 ,…,ψ max，a2 ,…,ψ max，b2 ); where ψ max，a2 is the maximum text confidence list corresponding to the a22th target time range; ψ max，a2 =MAX(ψ 1，a2 , ψ 2，a2 ,…,ψ σ，a2 ,…,ψ θ，a2 );MAX() is the preset maximum value function.

[0198] G056, change ψ max The vocal segment corresponding to each maximum text confidence is determined as the target vocal segment; wherein the target vocal segment is used for model training.

[0199] In this embodiment, the speech sample labeling method in the above steps G010-G050 is the same as the speech sample labeling method in the second embodiment, and is not described in detail here.

[0200] Furthermore, TC a2，2 Determine this by following these steps: G041, for TB a2 Perform audio feature extraction to obtain TB a2 The corresponding audio feature vector list TG.

[0201] G042, get TB a2 The corresponding text feature TL.

[0202] G043, splices TG and TL to obtain TB a2 The corresponding speech-to-text feature vector TU=(TG, TL).

[0203] G044, input TU into the preset classification model to obtain TB a2 The corresponding conversation type.

[0204] In this embodiment, TC a2，2 The determination method of TC in Example 2 is the same as that in Example 2 a2，2 The determination method is the same and will not be repeated here.

[0205] In this embodiment, the efficiency and consistency of audio labeling are significantly improved through a multi-channel collaborative labeling mechanism: based on randomly selected main channel human voice clips and their dialogue types, audio clips in the corresponding time range in other channels are automatically and synchronously labeled to ensure that the labeling results of the same voice event in different channels are temporally and spatially aligned, avoiding the repetitive work of manual channel-by-channel labeling; through the precise labeling of the main channel, the automatic labeling of the auxiliary channel is guided, effectively solving the problem of missed labeling of clips in multi-channel audio due to noise interference or signal attenuation, which is particularly suitable for scenarios with differentiated audio quality collected by microphones at different positions in the cabin; combined with dialogue type recognition technology, the system can automatically identify and associate the same dialogue content across channels (such as the delay matching of cockpit instructions and cabin broadcasts), providing better training samples for subsequent model training, and greatly improving the effect of model training.

[0206] Example 4: After the human voice segment is recognized in the above embodiment, the conversation type needs to be classified. The following method is provided: Y100, obtain each vocal segment corresponding to the speech segment to be recognized to obtain a vocal segment list QC = (QC1, QC2, ..., QC x ,…,QC y ), x=1, 2, …, y; where QC x is the xth vocal segment corresponding to the speech to be recognized, and y is the number of vocal segments corresponding to the speech to be recognized.

[0207] In this embodiment, it should be noted that the voice segments to be recognized include vocal segments and non-vocal segments, and the vocal segments in the voice segments to be recognized can be recognized to obtain the vocal segment list QC; those skilled in the art can use the existing vocal segment recognition method according to actual needs to recognize the voice segments to be recognized to obtain the vocal segment list QC, which will not be elaborated here.

[0208] Y200, extract audio features from each vocal segment in QC to obtain the audio feature vector list YA=(YA1, YA2, ..., YA x ,…,YA y ); among them, YA x For QC x The corresponding audio features.

[0209] In this embodiment, audio features can be extracted for each vocal segment in QC. The audio features include: Mel-frequency cepstral coefficients (MFCC), fundamental frequency trajectory and other acoustic parameters, which capture emotional information such as intonation and stress; corresponding audio feature vectors are constructed based on the above audio features; it should be noted that those skilled in the art can use existing audio feature construction methods according to actual needs to obtain the audio feature vector list YA corresponding to QC, which will not be elaborated here.

[0210] Y300, obtain the text features corresponding to each vocal segment in QC to obtain the text feature vector YB corresponding to QC = (YB1, YB2, ..., YB x ,…,YB y ); Among them, YB x For QC x The corresponding text feature vector.

[0211] Furthermore, Y300 includes the following steps: Y310, QC x Input to a preset ASR model; wherein the ASR model includes an encoder and a decoder, the encoder is used to encode the text features of the input human voice segment to obtain a text feature vector; the decoder is used to decode the text feature vector output by the encoder.

[0212] In this embodiment, the vocal segment QC x Input to the preset automatic speech recognition (ASR) model, the encoder performs QC x It performs deep feature extraction to generate text feature vectors (such as latent representations of speech semantic distribution and contextual relevance), rather than directly outputting text. The encoder captures the time-frequency domain correlation features of speech using a convolutional neural network (CNN) or Transformer architecture.

[0213] Y320, the text feature vector output by the decoder is determined as YB x .

[0214] The decoder receives the text feature vector output by the encoder and generates a text sequence through an attention mechanism or a recurrent neural network (RNN). The key point is that this step does not directly use the decoded text result, but uses the intermediate feature vector output by the decoder (such as the decoder hidden state or attention weight) as the text feature vector YB x The vector combines the acoustic characteristics of the speech with the semantic context information.

[0215] Y400, concatenate YA and YB to obtain the speech and text feature vector list YC corresponding to QC = (YC1, YC2, ..., YC x ,…,YC y); Among them, YC x For QC x Corresponding speech and text feature vector; YC x =(YA x , YB x ).

[0216] In this embodiment, the concatenation of the above-mentioned speech feature vector and text feature vector has the following beneficial effects: 1. Anti-ASR error interference: Traditional methods directly use ASR output text for semantic analysis. However, noise, accents, and specialized terminology in civil aviation scenarios can easily lead to ASR text errors (for example, misidentifying "FL310" as "FLT10"). This solution extracts the decoder's intermediate feature vector (YBx) to preserve the original semantic distribution of the speech rather than the erroneous text, preventing erroneous results from being passed to subsequent classification modules.

[0217] 2. Deep semantic expression: The decoder's intermediate feature vectors can encode the underlying semantic relationships in speech (such as the contextual logic of instructions and the complex meanings of professional terms). This makes them more suitable for parsing the complex intent of aviation conversations than simple text keywords. For example, the feature vector can capture the strong association between "MAYDAY" and emergency situations, even if the ASR text contains some spelling errors.

[0218] 3. Multi-granularity feature fusion: Characteristic vector (YB x ) and audio features (YA x ) can simultaneously utilize complementary information from acoustic sentiment (such as urgency and increased speech rate) and semantic intent (such as keyword weight). For example, when engine noise masks some audio features, the text feature vector can still infer the conversation type through semantic associations.

[0219] 4. Computational efficiency optimization: The joint training of the encoder and decoder gives the intermediate feature vectors a high information density. Compared with the traditional text processing process (ASR → text cleaning → feature extraction), the direct output of feature vectors reduces redundant calculation steps, improves real-time performance, and meets the millisecond-level response requirements of civil aviation.

[0220] For example: When QC x When the pilot issues the command "Descend to FL250" for a flight with engine noise: The encoder may generate slightly distorted feature vectors due to noise interference, but the decoder focuses on the key frequency bands through the attention mechanism, and the intermediate features of its output can still retain the strong correlation between "FL250" and the height adjustment instructions.

[0221] The subsequent classification model combines audio features (such as the falling tone trend) with the text feature vector to accurately determine the "flight instruction" type, rather than relying on potentially erroneous ASR text (such as the mistaken conversion of "Descend to FL2H0").

[0222] In this embodiment, the accuracy and robustness of conversation type classification in complex acoustic environments are significantly improved through deep semantic feature extraction and error isolation mechanism.

[0223] Y500 inputs YC into the preset classification model to obtain the dialogue type corresponding to each vocal segment in QC.

[0224] Furthermore, step Y500 includes the following steps: Y510, input YC into the preset classification model to obtain the confidence list set ω=(ω1,ω2,…,ω x ,…,ω y ); where ω x For QC x The corresponding confidence list; ω x =(ω x，1 ,ω x，2 ,…,ω x，α ,…,ω x，β ), α=1, 2,...,β; among them, ω x，α For QC x is the confidence of the αth preset dialogue type, and β is the number of preset dialogue types.

[0225] Y520, according to ω, determines the target confidence corresponding to each vocal segment of QC to obtain the target confidence list μ = (μ1, μ2, ..., μ x ,…,μ y ); where μ x For QC x The corresponding target confidence; μ x =MAX(ω x );MAX() is the preset maximum value function.

[0226] Y530, traverse μ, if μ x ≥μ', then μ x The corresponding preset dialogue type is determined to be QC x The corresponding dialogue type; μ' is the preset confidence threshold.

[0227] In this embodiment, low-confidence results are filtered through thresholds to avoid misclassification caused by noise interference or voice aliasing, especially to prevent non-critical conversations from being misclassified as high-risk instructions (such as mistakenly identifying small talk as an emergency broadcast). Only results are directly output for high-confidence segments to reduce the subsequent processing load and meet the real-time requirements of civil aviation scenarios.

[0228] Furthermore, after step Y530, the method further includes the following steps: Y540, if μ x <μ', and μ x The corresponding preset dialogue type is the preset conversation dialogue type, then obtain QC x Adjacent vocal clips.

[0229] If the low confidence segment μ x If the corresponding type is a conversation (such as a question and answer between a flight attendant and a passenger), the preceding and following adjacent segments (such as speech within ±1 second) are obtained.

[0230] Y541, QC x Splice with adjacent vocal clips to form QC x Corresponding spliced vocal segment QC x '.

[0231] QC x Splice with adjacent fragments to form QC x ', forming a longer time sequence of speech segments.

[0232] Y542, QC x 'Reclassify.

[0233] QC x 'Re-extract features and classify, and use the expanded context information to correct the results.

[0234] In this embodiment, conversational dialogues are usually continuous, and after splicing, they provide more complete semantic information (such as the relevance of questions and answers), solving the problem of missing fragment information caused by short-term noise or voice interruption; avoiding directly discarding low-confidence fragments, and automatically repairing the classification results through dynamic expansion of the window, thereby improving the recognition consistency of scenarios such as cabin or aircraft cabin conversations.

[0235] Furthermore, after step Y530, the method further includes the following steps: Y550, if μ x <μ', and μ x The corresponding preset dialogue type is the preset instruction dialogue type, then obtain QC x Corresponding keywords to get QC x Corresponding keyword list QE x =(QE x，1 , QE x，2 ,…,QE x，γ ,…,QE x，λ（x） ), γ=1, 2,..., λ(x); among them, QE x，γ For QC xCorresponding to the γth keyword, λ(x) is QC x The number of corresponding keywords.

[0236] When a certain segment (QC x ) classification confidence μ x When the value is lower than the threshold μ' and the type corresponding to the highest confidence is the preset command dialogue type (such as pilot operation instructions, emergency broadcast), the system extracts all keywords from the speech recognition (ASR) results of the segment and generates a keyword list QE x For example, if QC x The ASR text is "Request descent to FL250 due turbulence", QE x It may contain keywords such as "Request", "descent", "FL250", "turbulence", etc.

[0237] Y551, traversing QE x , determine QE x The number NUM1 of preset instruction keywords.

[0238] Traversing QE x , count the number of preset command keyword libraries NUM1. The preset library contains civil aviation-specific terms (such as "MAYDAY", "FLXXX", "Emergency", etc.), NUM1 represents QC x The number of words related to key operations in .

[0239] Y552, if NUM1 / λ(x) ≥ QR, then determine QC x The corresponding dialogue type is the command dialogue type; otherwise, it is determined manually; QR is the preset keyword quantity weight threshold.

[0240] Calculate the instruction keyword ratio (NUM1 / λ(x)). If the ratio is greater than or equal to the preset threshold QR (such as 40%), QC is determined. x It is a command dialogue type; otherwise, it will be transferred to manual review and confirmation.

[0241] In this embodiment, the above method has the following beneficial effects: 1. Anti-noise interference instruction fault-tolerant recognition: In scenarios with engine noise (interference in the 80Hz-2kHz frequency band) or voice aliasing, ASR may produce some erroneous text (e.g., mistranslating "FL250" as "FL2H0"). By counting the percentage of command keywords rather than relying on full text matching, the system can tolerate local recognition errors. For example, if "FL250" in QCx is mistakenly recognized as "FL2H0," but other keywords such as "descent" and "turbulence" are still correctly extracted, NUM1 / λ(x) may reach 50% (QR = 40%), and the system can still determine the command type, avoiding missing critical operation instructions.

[0242] 2. Double protection mechanism with safety as the priority: Implementing strong keyword verification for low-confidence instruction fragments complies with the civil aviation safety principle of "preferring false alarms to false negatives". x =0.65, μ'=0.7). As long as the command keyword ratio meets the standard, the system will still prioritize it as a command type to ensure that high-risk operations such as emergency broadcasts and flight mode switching are captured in a timely manner.

[0243] 3. Precise intervention of manual review: When the keyword ratio is insufficient (such as QC x Contains a lot of non-critical dialogue interference), and is transferred to manual intervention for review to avoid misjudgment by the automated system due to noise or semantic ambiguity (such as misclassifying a passenger's question "Do you want to make an emergency landing" as a real instruction).

[0244] 4. Dynamic weight thresholds adapt to scenario requirements: The threshold QR can be flexibly adjusted according to different scenarios. For example: Cockpit commands: QR is set low (e.g. 30%) because commands are usually short and keyword-intensive (e.g. "ClimbFL300").

[0245] Cabin announcement: The QR setting is higher (e.g. 50%) because it may contain service terms (e.g. "Please fasten your seat belts" and "Emergency evacuation" coexisting).

[0246] Confidence reflects the model's grasp of overall features, while keyword verification focuses on the semantic core. The combination of the two avoids the one-sidedness of a single criterion and forms redundant verification, meeting the fault-tolerant design requirements of aviation safety systems. Keyword analysis is only triggered for low-confidence instruction fragments rather than full processing, saving computing resources and meeting the real-time requirements of airborne systems.

[0247] In this embodiment, the multimodal fusion architecture of audio and text is used to significantly improve the robustness of conversation classification in complex acoustic environments: for multi-person mixed speech, voice segmentation is combined with cross-modal feature splicing to simultaneously capture acoustic features (such as intonation and stress) and semantic information (such as keywords and command structure), effectively distinguishing independent intentions in overlapping conversations; through audio-text joint modeling, the system can compensate for the recognition defects of a single modality (such as speech recognition errors or text ambiguity caused by noise), and enhance the contextual understanding of aviation professional terminology and emergency instructions; the dynamic feature fusion mechanism can also adaptively distinguish between routine communication and key operational instructions, ensuring priority identification and processing of high-risk conversation types such as tower instructions and cabin emergency broadcasts, thereby ensuring the security and timeliness of civil aviation communications.

[0248] Furthermore, after determining the conversation type, it is necessary to confirm the user corresponding to the voice to be recognized. Therefore, after step Y500, the method further includes the following steps: Y600, obtain the voiceprint feature vector corresponding to each human voice segment in QC to obtain the voiceprint feature column vector table HC corresponding to QC = (HC1, HC2, ..., HC x ,…,HC y ); Among them, HC x For QC x The corresponding voiceprint feature vector.

[0249] Y610, obtain each preset voiceprint feature vector to obtain a preset voiceprint feature vector list HA = (HA1, HA2, ..., HA c ,…,HA d ), c = 1, 2, ..., d; where HA c is the voiceprint feature vector of the cth user, and d is the number of users.

[0250] Y620, get HC x The maximum similarity HX with each preset voiceprint feature vector in HA.

[0251] Y630, if HX ≥ H', then the user corresponding to HX is determined as QC x Corresponding users, to obtain the identified user list GA = (GA1, GA2, ..., GA C1 ,…,GA d1 ), c1=1, 2, …, d1; where GA C1 is the c1th user identified, and d1 is the number of identified users.

[0252] Y640, if HX<H', then QC x Determined to be an unrecognized vocal segment; enter G700.

[0253] Y650, determines the QC according to the corresponding vocal segment of each user in GA x The corresponding user.

[0254] Furthermore, step Y650 includes the following steps: Y651, determine the vocal segment corresponding to each user in GA from QC.

[0255] Y652, fuse at least two vocal segments corresponding to each user in GA to obtain an initial fused vocal segment list GB = (GB1, GB2, ..., GB e1 ,…,GB δ ), e1=1, 2, …, δ; where GB e1 is the e1th initial fused vocal segment obtained by fusion, and δ is the number of initial fused vocal segments obtained by fusion.

[0256] Y653, obtain the initial voiceprint feature vector corresponding to each initial fused voice segment in GB to obtain the initial voiceprint feature vector list GC = (GC1, GC2, ..., GC e1 ,…,GC δ ); Among them, GC e1 GB e1 The corresponding initial voiceprint feature vector.

[0257] Y654, obtain QC x The maximum similarity GD between the voiceprint feature vector and each initial voiceprint feature vector in GC x .

[0258] Y655, if GD x ≥H', then GD x The corresponding at least two identified users are determined as QC x The corresponding user.

[0259] In this embodiment, the method for determining the user corresponding to the speech to be recognized is the same as the method for determining the user corresponding to the speech to be recognized in the first embodiment, and is not described in detail here.

[0260] In this embodiment, first, the voiceprint features corresponding to each voice segment are obtained by voice segmentation and multi-dimensional voiceprint feature fusion extraction, and the voiceprint feature vector corresponding to each voice segment is compared with each preset voiceprint feature vector for similarity to obtain the maximum similarity. If the maximum similarity is greater than the preset voiceprint feature vector similarity threshold, the user corresponding to the currently identified voice segment is determined to be the user corresponding to the voiceprint feature vector with the maximum similarity. If the corresponding user cannot be identified, it may be that multiple people are speaking at the same time, and then re-determination is performed based on the preset fusion voiceprint feature vector list, thereby achieving the purpose of accurate voice recognition in a more complex environment.

[0261] Embodiment 5: In the above embodiment, the model used needs to be trained before use. Therefore, the training samples need to be labeled. The traditional purely manual labeling method is inefficient. Therefore, the following method is provided: T100, obtain the sub-audio samples of each channel corresponding to the audio sample to obtain a sub-audio sample list TA = (TA1, TA2, ..., TA σ ,…,TA θ ),σ=1,2,…,θ; among them, TA σ is the sub-audio sample of the σth channel corresponding to the audio sample, and θ is the number of channels corresponding to the audio sample.

[0262] In this embodiment, the original audio samples are separated by channel to generate a sub-audio sample list TA. Each sub-audio sample corresponds to an independent channel (such as the cockpit main microphone channel, the co-pilot headphone channel, the cabin broadcast channel, etc.), ensuring the independence of the multi-channel data.

[0263] T200, randomly select a sub-audio sample TA' from TA.

[0264] A sub-audio sample TA' is randomly selected from TA as the reference channel for annotation. Random selection avoids bias towards a specific channel (such as the channel with the strongest signal), enhancing the system's adaptability to different channel qualities.

[0265] T300, obtain each vocal segment corresponding to TA' to obtain a vocal segment list TB corresponding to TA' = (TB1, TB2, ..., TB a2 ,…,TB b2 ), a2=1, 2, …, b2; where TB a2 is the a2th vocal segment corresponding to TA', and b2 is the number of vocal segments corresponding to TA'.

[0266] Perform voice activity detection (VAD) and segmentation on TA' to extract a list of valid vocal segments, TB. Each vocal segment represents a continuous vocal interval (e.g., a single pilot instruction or a flight attendant announcement), excluding silent or noisy segments.

[0267] T400, obtain the time range and dialogue type of each vocal segment in TB to obtain the time range dialogue type group list TC corresponding to TA' = (TC1, TC2, ..., TC a2 ,…,TC b2 ); Among them, TC a2 TB a2 Corresponding time range dialogue type group; TC a2 =(TC a2，1 , TC a2，2 );TC a2，1 TB a2 Corresponding time range, TC a2，2 TB a2 The corresponding conversation type.

[0268] Furthermore, TC a2，2 Determine this by following these steps: T410, for TB a2 Perform audio feature extraction to obtain TB a2 The corresponding audio feature vector list TG.

[0269] T420, get TB a2 The corresponding text feature TL.

[0270] T430, splices TG and TL to obtain TB a2 The corresponding speech-to-text feature vector TU=(TG, TL).

[0271] T440, input TU into the preset classification model to obtain TB a2 The corresponding conversation type.

[0272] Furthermore, TL is obtained by the following steps: T421, TB a2 Input to a preset ASR model; wherein the ASR model includes an encoder and a decoder, the encoder is used to encode the text features of the input human voice segment to obtain a text feature vector; the decoder is used to decode the text feature vector output by the encoder.

[0273] T422, the text feature vector output by the decoder is determined as TL.

[0274] Furthermore, step T440 includes the following steps: T441, input TU into the preset classification model to obtain the confidence list TV corresponding to TU = (TV1, TV2, ..., TV α ,…,TV β ), α=1, 2,...,β; among them, TV α is the confidence that TU is the αth preset dialogue type, and β is the number of preset dialogue types.

[0275] T442, based on TV, determine the target confidence TU'=MAX(TU) corresponding to TU; MAX() is a preset maximum value function.

[0276] T443: If TU'≥μ', the preset conversation type corresponding to TU' is determined as the conversation type corresponding to TU; μ' is a preset confidence threshold.

[0277] T444, if TU'<μ', and the preset dialogue type corresponding to TU' is the preset conversation dialogue type, then obtain TB a2 Adjacent vocal clips.

[0278] T445, TB a2 Splice with adjacent vocal clips to form TB a2 Corresponding spliced vocal segment TB a2 '.

[0279] T446, for TB a2 'Reclassify.

[0280] Furthermore, the method further comprises the following steps: T447, if TU'<μ', and the preset dialogue type corresponding to TU' is the preset command dialogue type, then obtain TB a2 Corresponding keywords to get TB a2 The corresponding keyword list.

[0281] T448, confirmed TB a2 The corresponding keyword list contains the number NUM2 of preset instruction keywords.

[0282] T449, if NUM2 / NUM3 ≥ QR, then determine QC x The corresponding dialogue type is the command dialogue type; otherwise, it is determined manually; NUM3 is TB a2 The number of keywords in the corresponding keyword list; QR is the preset keyword quantity weight threshold.

[0283] In this embodiment, the above-mentioned method for classifying conversation types is the same as the method for classifying conversation types in the sixth embodiment, and is not described in detail here.

[0284] T500, according to the preset dialogue type and TC, synchronously mark the sub-audio samples in TA except TA'.

[0285] Furthermore, step T500 includes the following steps: T510, traverse TC, if TC a2，2 For the default conversation type, TC a2 Determine the target time range.

[0286] T511 , labeling each sub-audio sample in TA except TA′ with the corresponding target time range and the preset dialogue type.

[0287] In this embodiment, the annotation information (TC) of the main channel TA' is synchronously mapped to all other channels (subsamples in TA except TA'). That is, within the same time range, the corresponding segments of other channels automatically inherit the same dialogue type label.

[0288] In this embodiment, the technical effects of the above method are as follows: 1. A leap in annotation efficiency: Only a single main channel (TA') needs to be manually or finely labeled, and other channels automatically inherit the label through time synchronization, avoiding the repetitive work of channel-by-channel labeling in traditional methods. The efficiency improvement is proportional to the number of channels θ (for example, when θ=5 channels, the efficiency is improved by nearly 5 times).

[0289] 2. Multi-channel spatiotemporal alignment: Timestamp synchronization solves the propagation delay problem of multi-channel audio (for example, cockpit instructions are played with a 0.5-second delay in the cabin broadcast channel), ensuring that the marks of the same voice event in different channels are strictly aligned, providing a consistent data foundation for voice analysis (such as command tracing and event replay).

[0290] 3. Robustness to noisy scenes: The random selection of the primary channel implicitly prioritizes high-quality channels (such as low-noise pilot headset microphones), and the annotations generated based on them are synchronized to the noisy secondary channel (such as the cabin ambient microphone), avoiding the problem of missing or mislabeled segments when the noisy channel is independently annotated.

[0291] 4. Adaptability to civil aviation scenarios: For concurrent conversations between multiple roles in the cabin (such as overlapping pilot-to-tower conversations and flight attendant broadcasts), the main channel is accurately labeled to guide the automatic association of auxiliary channels, resolving the conflicts in conversation types caused by independent channel processing in traditional methods (such as the same instruction being mistakenly labeled as different types in different channels).

[0292] 5. Labeling consistency assurance: All audio channels share the same annotation benchmark, eliminating subjective differences in manual annotation. This is particularly suitable for civil aviation voice recordings that require strict compliance (such as black box data analysis and accident investigations), ensuring that the annotation results are traceable and verifiable.

[0293] Furthermore, after step T511, the method further includes the following steps: T512: For any target time range, randomly select a labeled sub-audio clip from the θ channels as a model training sample.

[0294] In this embodiment, after completing the labeling of the target time range (T511), for each labeled segment (such as the "urgent command" with timestamps 10.2s-12.5s), the system randomly selects a labeled sub-audio segment from the θ channels (such as the data from the co-pilot microphone channel) as a sample for subsequent model training.

[0295] Randomly select samples from different channels (e.g., the captain's microphone this time, the cabin announcement microphone next time) to ensure that the training data covers the acoustic characteristics of multiple devices and locations (e.g., frequency response differences and noise distribution between different microphones), improving the model's generalization to channel differences. This avoids the model's bias towards channel-specific characteristics (e.g., formants in a specific frequency band) caused by rigidly using a single channel (e.g., always selecting the captain's microphone with the highest signal-to-noise ratio), thereby enhancing the model's robustness to low-quality channels (e.g., the cabin ambient microphone). Only the primary channel (TA') needs to be labeled. Corresponding clips from other channels are labeled synchronously, but only one of them is randomly selected during training. This reduces the impact of duplicate samples on training efficiency and balances data size and computational cost. To accommodate asynchronous multi-channel recording scenarios in the cabin (e.g., delays between headset and ambient microphones), the model learns to tolerate propagation delays (typically 200-500ms) by randomly selecting implicit time alignment (e.g., timestamp differences between different channels for the same event are naturally captured in the training data).

[0296] Furthermore, step T500 includes the following steps: T520, determine the audio segments of each sub-audio sample in TA except TA' within the target time range as vocal segments, so as to obtain a vocal segment list set TD = (TD1, TD2, ..., TD σ ,…,TD θ ); Among them, TD σ For TA σ Corresponding vocal clip list; TD σ =(TD σ，1 , TD σ，2 ,…,TD σ，a2 ,…,TD σ，b2 );TD σ，a2 For TA σThe corresponding a2th vocal segment.

[0297] Based on the time range of the main channel TA' (TC a2,1 ), from all other channels (TA σ ) Extract the audio segments of the corresponding time period to form a multi-channel vocal segment list set TD. Each TD σ,a2 Represents the vocal segment of the σth channel in the a2th time range.

[0298] T530, TD is input into the preset ASR model to obtain the text confidence corresponding to each human voice segment, and then obtain the text confidence list corresponding to the audio sample ψ = (ψ1, ψ2, ..., ψ σ ,…,ψ θ ); where ψ σ For TD σ The corresponding text confidence list; ψ σ =(ψ σ，1 , ψ σ，2 ,…,ψ σ，a2 ,…,ψ σ，b2 );ψ σ，a2 For TA σ The text confidence of the corresponding a2th vocal segment.

[0299] TD all vocal clips σ,a2 Input the ASR model to obtain the text recognition confidence ψ of each segment σ,a2 (0-1 range), generates the confidence matrix ψ of the entire channel. The confidence reflects the degree of ASR's confidence in the accuracy of the text segment.

[0300] T540, based on ψ, determine the maximum text confidence corresponding to each target time range to obtain a maximum text confidence list ψ max =(ψ max，1 , ψ max，2 ,…,ψ max，a2 ,…,ψ max，b2 ); where ψ max，a2 is the maximum text confidence list corresponding to the a22th target time range; ψ max，a2 =MAX(ψ 1，a2 , ψ 2，a2 ,…,ψ σ，a2 ,…,ψ θ，a2 );MAX() is the preset maximum value function.

[0301] For each target time range, compare the confidence scores ψ of all channels in that time period σ,a2 , select the maximum value ψ max，a2 =MAX(ψ 1，a2 , ψ 2，a2,…,ψ σ，a2 ,…,ψ θ，a2 ); This step ensures that only the most reliable channel data is retained in each time period.

[0302] T550, will max The vocal segment corresponding to each maximum text confidence is determined as the target vocal segment; wherein the target vocal segment is used for model training.

[0303] The optimal vocal segment corresponding to each target time period (i.e., the segment with the most accurate text recognition) is marked as the target vocal segment, which serves as a high-quality data source for subsequent model training.

[0304] The above method has the following technical effects: 1. Data quality optimization: Through the multi-channel confidence competition mechanism, the channel segments with the highest recognition quality in each time period are automatically screened and identified, eliminating low-quality samples caused by noise and equipment differences, thereby improving the signal-to-noise ratio of the training data.

[0305] 2. Cross-channel redundancy utilization: By leveraging the inherent advantages of multi-channel systems, even if one channel fails completely (such as a microphone failure), data integrity can still be guaranteed through valid segments from other channels, significantly improving system robustness.

[0306] 3. Adaptive channel selection: Dynamically select the optimal sound channel for different time periods (such as cockpit microphones during instruction periods and ceiling microphones during cabin broadcast periods), breaking through the limitations of fixed microphone deployment and being particularly suitable for the dynamically changing acoustic environment in civil aviation scenarios.

[0307] 4. Computational efficiency balance: Only deep training is performed on key fragments, which reduces computing power by 30-40% compared to full data training, while ensuring that model performance does not degrade.

[0308] 5. Semi-supervised learning compatible: High-confidence fragments can be directly used as labeled data, and low-confidence fragments can trigger manual review or downgrade use to form a hierarchical data utilization strategy.

[0309] The method in this embodiment achieves an optimal balance between data quality, system fault tolerance, and computational efficiency through a multi-channel confidence competition mechanism, and is particularly suitable for voice processing scenarios with high reliability requirements, such as civil aviation.

[0310] In this embodiment, the efficiency and consistency of audio labeling are significantly improved through a multi-channel collaborative labeling mechanism: based on randomly selected main channel human voice clips and their dialogue types, audio clips in the corresponding time range in other channels are automatically and synchronously labeled to ensure that the labeling results of the same voice event in different channels are temporally and spatially aligned, avoiding the repetitive work of manual channel-by-channel labeling; through the precise labeling of the main channel, the automatic labeling of the auxiliary channel is guided, effectively solving the problem of missed labeling of clips in multi-channel audio due to noise interference or signal attenuation, which is particularly suitable for scenarios with differentiated audio quality collected by microphones at different positions in the cabin; combined with dialogue type recognition technology, the system can automatically identify and associate the same dialogue content across channels (such as the delay matching of cockpit instructions and cabin broadcasts), providing better training samples for subsequent model training, and greatly improving the effect of model training.

[0311] In this embodiment, after the speech samples are labeled and the training of the preset speech model is completed, the method steps in Embodiments 1 to 4 can be implemented.

[0312] Furthermore, although the steps of the method of the present disclosure are described in a particular order in the accompanying drawings, this does not require or imply that the steps must be performed in this particular order, or that all steps shown must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps.

[0313] An embodiment of the present invention also provides a non-transitory computer-readable storage medium, which can be set in an electronic device to store at least one instruction or at least one program related to implementing a method in a method embodiment. The at least one instruction or the at least one program is loaded and executed by the processor to implement the method provided in the above embodiment.

[0314] The program product may utilize any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or component, or any combination thereof. More specific examples (a non-exhaustive list) of readable storage media include: an electrical connection having one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof.

[0315] A computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, which carries readable program code. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A readable signal medium may also be any readable medium other than a readable storage medium that can transmit, propagate, or transfer a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0316] The program code embodied on the readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0317] The program code for performing the operations of the present application can be written in any combination of one or more programming languages, including object-oriented programming languages such as Java, C++, etc., and conventional procedural programming languages such as "C" or similar programming languages. The program code can be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In the case of a remote computing device, the remote computing device can be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0318] An embodiment of the present invention further provides an electronic device including a processor and the aforementioned non-transitory computer-readable storage medium.

[0319] The electronic device is merely an example and should not limit the functions and scope of use of the embodiments of the present application.

[0320] The electronic device is implemented as a general-purpose computing device. Components of the electronic device may include, but are not limited to, the aforementioned at least one processor, the aforementioned at least one memory, and a bus connecting different system components (including the memory and the processor).

[0321] The memory stores program codes, which can be executed by the processor, so that the processor performs the steps of various embodiments described in this specification.

[0322] The memory may include readable media in the form of volatile memory, such as random access memory (RAM) and / or cache memory, and may further include read-only memory (ROM).

[0323] The memory may also include a program / utility having a set (at least one) of program modules, such program modules including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.

[0324] The bus may represent one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, a processor, or a local bus using any of a variety of bus architectures.

[0325] The electronic device may also communicate with one or more external devices (e.g., keyboards, pointing devices, Bluetooth devices, etc.), one or more devices that enable a user to interact with the electronic device, and / or any device that enables the electronic device to communicate with one or more other computing devices (e.g., routers, modems, etc.). Such communication may be performed via an input / output (I / O) interface. Furthermore, the electronic device may also communicate with one or more networks (e.g., local area networks (LANs), wide area networks (WANs), and / or public networks, such as the Internet) via a network adapter. The network adapter communicates with other modules of the electronic device via a bus. It should be understood that, although not shown in the figures, other hardware and / or software modules may be used in conjunction with the electronic device, including but not limited to: microcode, device drivers, redundant processors, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0326] Through the description of the above embodiments, it is easy for those skilled in the art to understand that the example embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solution according to the embodiments of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, and includes several instructions to enable a computing device (which can be a personal computer, a server, a terminal device, or a network device, etc.) to execute the method according to the embodiments of the present disclosure.

[0327] An embodiment of the present invention further provides a computer program product comprising program code. When the program product is run on an electronic device, the program code is used to enable the electronic device to execute the steps of the method according to various exemplary embodiments of the present invention described above in this specification.

[0328] Although some specific embodiments of the present invention have been described in detail by way of examples, it should be understood by those skilled in the art that the above examples are for illustration only and are not intended to limit the scope of the present invention. It should also be understood by those skilled in the art that various modifications may be made to the embodiments without departing from the scope and spirit of the present invention.

Claims

1. A method for identifying a person based on an audio sample, characterized in that: The method comprises the following steps: G100, obtain each vocal segment corresponding to the speech segment to be recognized to obtain a vocal segment list QC = (QC1, QC2, ..., QC x ,…,QC y ), x=1, 2, …, y; where QC x is the xth vocal segment corresponding to the speech to be recognized, and y is the number of vocal segments corresponding to the speech to be recognized; G200, obtain the voiceprint feature vector corresponding to each human voice segment in QC to obtain the voiceprint feature column vector table HC corresponding to QC = (HC1, HC2, ..., HC x ,…,HC y ); Among them, HC x For QC x The corresponding voiceprint feature vector; G300 obtains each preset voiceprint feature vector to obtain a preset voiceprint feature vector list HA=(HA1, HA2, ..., HA c ,…,HA d ), c = 1, 2, ..., d; where HA c is the voiceprint feature vector of the cth user, and d is the number of users; G400, get HC x The maximum similarity HX with each preset voiceprint feature vector in HA; G500, if HX ≥ H', the user corresponding to HX is determined as QC x Corresponding users, to obtain the identified user list GA = (GA1, GA2, ..., GA C1 ,…,GA d1 ), c1=1, 2, …, d1; where GA C1 is the c1th user identified, d1 is the number of identified users; G600, if HX<H', then QC x Determined to be an unrecognized vocal segment; enter G700; G700, according to the corresponding vocal segment of each user in QC in GA, determines QC x The corresponding user.

2. The method for identifying a person based on an audio sample according to claim 1, wherein: Step G700 includes the following steps: G710, determining the vocal segment corresponding to each user in GA from QC; G720, fuse at least two vocal segments corresponding to each user in GA to obtain an initial fused vocal segment list GB = (GB1, GB2, ..., GB e1 ,…,GB δ ), e1=1, 2, …, δ; where GB e1 is the e1th initial fused vocal segment obtained by fusion, δ is the number of initial fused vocal segments obtained by fusion; G730, obtain the initial voiceprint feature vector corresponding to each initial fused voice segment in GB to obtain the initial voiceprint feature vector list GC=(GC1, GC2, ..., GC e1 ,…,GC δ ); Among them, GC e1 GB e1 The corresponding initial voiceprint feature vector; G740, Get QC x The maximum similarity GD between the voiceprint feature vector and each initial voiceprint feature vector in GC x ; G750, if GD x ≥H', then GD x The corresponding at least two identified users are determined as QC x The corresponding user.

3. The method for identifying a person based on an audio sample according to claim 2, wherein: After step G750, the method further includes the following steps: G760, if GD x <H', then determine QC x The corresponding users include other users outside of GA.

4. The method for identifying a person based on an audio sample according to claim 1, wherein: Before step G100, the method further includes the following steps: G010, obtain the sub-audio samples of each channel corresponding to the audio sample to obtain a sub-audio sample list TA = (TA1, TA2, ..., TA σ ,…,TA θ ),σ=1,2,…,θ; among them, TA σ is the sub-audio sample of the σth channel corresponding to the audio sample, and θ is the number of channels corresponding to the audio sample; G020, randomly select a sub-audio sample TA' from TA; G030, obtain each vocal segment corresponding to TA' to obtain a vocal segment list TB corresponding to TA' = (TB1, TB2, ..., TB a2 ,…,TB b2 ), a2=1, 2, …, b2; where TB a2 is the a2th vocal segment corresponding to TA', and b2 is the number of vocal segments corresponding to TA'; G040, obtain the time range and dialogue type of each vocal segment in TB to obtain the time range dialogue type group list TC = (TC1, TC2, ..., TC a2 ,…,TC b2 ); Among them, TC a2 TB a2 Corresponding time range dialogue type group; TC a2 =(TC a2，1 , TC a2，2 );TC a2，1 TB a2 Corresponding time range, TC a2，2 TB a2 The corresponding conversation type; G050, based on the preset dialogue type and TC, synchronously mark the sub-audio samples in TA except TA'.

5. The method for identifying a person based on an audio sample according to claim 4, characterized in that: Step G050 includes the following steps: G051, traverse TC, if TC a2，2 For the default conversation type, TC a2 Determine the target time frame; G052: Label each sub-audio sample in TA except TA' with the corresponding target time range and the preset dialogue type.

6. The method for identifying a person based on an audio sample according to claim 4, wherein: Step R050 includes the following steps: G053, determine the audio segments of each sub-audio sample in TA except TA' within the target time range as vocal segments, so as to obtain the vocal segment list set TD = (TD1, TD2, ..., TD σ ,…,TD θ ); Among them, TD σ For TA σ Corresponding vocal clip list; TD σ =(TD σ，1 , TD σ，2 ,…,TD σ，a2 ,…,TD σ，b2 );TD σ，a2 For TA σ The corresponding a2th vocal segment; G054, input TD into the preset ASR model to obtain the text confidence corresponding to each human voice segment, and then obtain the text confidence list corresponding to the audio sample ψ = (ψ1, ψ2, ..., ψ σ ,…,ψ θ ); where ψ σ For TD σ The corresponding text confidence list; ψ σ =(ψ σ，1 , ψ σ，2 ,…,ψ σ，a2 ,…,ψ σ，b2 );ψ σ，a2 For TA σ The corresponding text confidence of the a2th vocal segment; G055, obtain the maximum text confidence corresponding to each target time range based on ψ to obtain the maximum text confidence list ψ max =(ψ max，1 , ψ max，2 ,…,ψ max，a2 ,…,ψ max，b2 ); where ψ max，a2 is the maximum text confidence list corresponding to the a22th target time range; ψ max，a2 =MAX(ψ 1，a2 , ψ 2，a2 ,…,ψ σ，a2 ,…,ψ θ，a2 );MAX() is the preset maximum value function; G056, change ψ max The vocal segment corresponding to each maximum text confidence is determined as the target vocal segment; wherein the target vocal segment is used for model training.

7. The method for identifying a person based on an audio sample according to claim 4, wherein: TC a2，2 Determine this by following these steps: G041, for TB a2 Perform audio feature extraction to obtain TB a2 The corresponding audio feature vector list TG; G042, get TB a2 The corresponding text feature TL; G043, splices TG and TL to obtain TB a2 The corresponding speech-to-text feature vector TU=(TG, TL); G044, input TU into the preset classification model to obtain TB a2 The corresponding conversation type.

8. A non-transitory computer-readable storage medium, wherein at least one instruction or at least one program is stored in the storage medium, characterized in that: The at least one instruction or the at least one program is loaded and executed by the processor to implement the person identification method based on audio samples as described in any one of claims 1 to 7.

9. An electronic device, characterized in that: The device comprises a processor and the non-transitory computer-readable storage medium of claim 8.

Images