An audio-text alignment method, apparatus, device, and storage medium

Abstract

The present application discloses an audio-text alignment method, apparatus, device, and storage medium, relating to the technical field of speech processing, including: obtaining initial audio data and the corresponding transcribed text, obtaining the rhythm change rate index corresponding to the initial audio data, and performing semantic analysis on the transcribed text to obtain the importance degree of each initial semantic unit; determining target semantic units according to the importance degree, and performing a preliminary match between each target semantic unit and the initial audio data to determine the anchor point positions corresponding to each target semantic unit; allocating timestamps to the initial audio data based on the rhythm change rate index to obtain corresponding target audio data, dividing the target audio data into different audio segments based on each anchor point position, and aligning the audio segments with the transcribed text based on the timestamps. By combining the rhythm characteristics of the audio and the semantics of the text to align the audio and the text, the alignment accuracy of the audio and the text is ensured.

Description

Technical Field

[0001] The present invention relates to the technical field of speech processing, and particularly relates to an audio-text alignment method, device, equipment and storage medium. Background Art

[0002] In the digital age, the precise synchronization of audio content and text has become a core requirement for numerous applications. From video subtitles, speech transcription to language learning platforms, accurate timestamp alignment is the key to providing a high-quality user experience. The core task of timestamp alignment technology is to determine the precise time points of each element (word, phrase or character) in the text corresponding to the corresponding segments in the audio stream, so as to establish a mapping relationship between the two.

[0003] Currently, the end-to-end model using the CTC algorithm (Connectionist Temporal Classification) and the attention mechanism based on Transformer are usually used to achieve the alignment of audio-text timestamps. However, this method has the problems of sparse key data distribution and inaccurate keyword timestamps detected, and cannot guarantee the accuracy when aligning natural speech and related text containing complex factors such as speech rate changes, pauses, intonation fluctuations and emotional expressions. Summary of the Invention

[0004] In view of this, the purpose of the present invention is to provide an audio-text alignment method, device, equipment and storage medium, which can align audio and text by combining the rhythm characteristics of audio and the semantics of text, and ensure the alignment accuracy of audio and text. The specific scheme is as follows:

[0005] In the first aspect, the present application provides an audio-text alignment method, including:

[0006] Obtain the initial audio data and the target transcription text corresponding to the initial audio data, obtain the rhythm change rate index corresponding to the initial audio data at different time nodes, and perform semantic analysis on the target transcription text to obtain the importance degree of each initial semantic unit in the target transcription text; wherein, the rhythm change rate index is used to characterize the degree of speech rhythm change in the initial audio data;

[0007] Determine target semantic units with an importance degree greater than a preset importance threshold from each of the initial semantic units according to each of the importance degrees, and perform a preliminary match between each of the target semantic units and the initial audio data to determine the anchor positions of each of the target semantic units in the initial audio data;

[0008] Dynamically allocate timestamps to the initial audio data based on each of the rhythm change rate indices to obtain corresponding target audio data, divide the target audio data into different target audio segments based on each of the anchor positions corresponding to each of the target semantic units, and align each of the target audio segments with the target transcription text based on each of the timestamps.

[0009] Optionally, obtaining the rhythm change rate indices respectively corresponding to the initial audio data at different time nodes includes:

[0010] Obtain the speech rate change situation, pitch information, and pause situation respectively corresponding to the initial audio data at different time nodes, and determine the target weights respectively corresponding to the speech rate change situation, the pitch information, and the pause situation;

[0011] Perform weighted fusion on the speech rate change situation, the pitch information, and the pause situation respectively corresponding to different time nodes based on each of the target weights to obtain the rhythm change rate indices respectively corresponding to the initial audio data at different time nodes.

[0012] Optionally, obtaining the rhythm change rate indices respectively corresponding to the initial audio data at different time nodes includes:

[0013] Obtain the speech scene and speech style corresponding to the initial audio data, automatically adjust the target parameters corresponding to the rhythm change rate index according to the speech scene and the speech style, and obtain the rhythm change rate indices respectively corresponding to the initial audio data at different time nodes according to the corresponding adjusted parameters.

[0014] Optionally, performing semantic analysis on the target transcription text to obtain the importance degree of each initial semantic unit in the target transcription text includes:

[0015] Perform semantic analysis on the target transcription text to obtain the syntactic centrality score, semantic density score, and topic relevance score corresponding to each of the initial semantic units in the target transcription text; wherein, the syntactic centrality score represents the status of each initial semantic unit in the corresponding syntactic structure, the semantic density score represents the amount of information contained in each initial semantic unit, and the topic relevance score represents the degree of association between each initial semantic unit and the core topic corresponding to the target transcription text;

[0016] Perform weighted fusion on the syntactic centrality score, the semantic density score, and the topic relevance score to obtain the importance degree of each initial semantic unit in the target transcription text.

[0017] Optionally, obtaining the syntactic centrality score, semantic density score, and topic relevance score corresponding to each of the initial semantic units in the target transcription text includes:

[0018] Obtaining the syntactic centrality score corresponding to each of the initial semantic units based on the part of speech of each of the initial semantic units;

[0019] Obtaining the semantic density score corresponding to each of the initial semantic units according to the inverse document frequency corresponding to each of the initial semantic units and the occurrence frequency of each of the initial semantic units in the target transcription text;

[0020] Mapping each of the initial semantic units and the core topic to a target semantic space to obtain a first vector corresponding to each of the initial semantic units and a second vector corresponding to the core topic, obtaining the cosine similarity between each of the first vectors and the second vector, and obtaining the topic relevance score corresponding to each of the initial semantic units according to each of the cosine similarities.

[0021] Optionally, dividing the target audio data into different target audio segments based on each of the anchor positions corresponding to each of the target semantic units includes:

[0022] Determining the information complexity corresponding to different parts of the target audio data, and dynamically setting the target paragraph length corresponding to different parts of the target audio data according to each of the information complexities and each of the anchor positions; wherein the information complexity is used to characterize the amount of information contained in different parts of the target audio data;

[0023] Dividing the target audio data into different target audio segments based on each of the target paragraph lengths; wherein the head of any target audio segment contains the same audio data as the tail of the previous adjacent target audio segment, and the tail of any target audio segment contains the same audio data as the head of the next adjacent target audio segment.

[0024] Optionally, aligning each of the target audio segments with the target transcription text based on each of the timestamps includes:

[0025] Constructing a similarity matrix between the target audio data and the target transcription text, and obtaining a matching pattern between the target audio data and the target transcription text according to the similarity matrix;

[0026] Obtaining a difference region between the target audio data and the target transcription text based on the matching pattern, obtaining a difference type corresponding to the difference region, and aligning each of the target audio segments with the target transcription text according to an alignment strategy corresponding to the difference type.

[0027] In a second aspect, the present application provides an audio-text alignment device, comprising:

[0028] A semantic analysis module, configured to obtain initial audio data and a target transcription text corresponding to the initial audio data, obtain a rhythm change rate index corresponding to the initial audio data at different time nodes, and perform semantic analysis on the target transcription text to obtain the importance degree of each initial semantic unit in the target transcription text; wherein, the rhythm change rate index is used to characterize the degree of voice rhythm change in the initial audio data;

[0029] An anchor point position determination module, configured to determine target semantic units with an importance degree greater than a preset importance threshold from each of the initial semantic units according to each of the importance degrees, and perform a preliminary match between each of the target semantic units and the initial audio data to determine the anchor point position of each of the target semantic units in the initial audio data;

[0030] An audio-text alignment module, configured to dynamically allocate time stamps to the initial audio data based on each of the rhythm change rate indexes to obtain corresponding target audio data, divide the target audio data into different target audio segments based on each of the anchor point positions corresponding to each of the target semantic units, and align each of the target audio segments with the target transcription text based on each of the time stamps.

[0031] In a third aspect, the present application provides an electronic device, comprising:

[0032] A memory, configured to store a computer program;

[0033] A processor, configured to execute the computer program to implement the foregoing audio-text alignment method.

[0034] In a fourth aspect, the present application provides a computer-readable storage medium, configured to store a computer program, and when the computer program is executed by a processor, the foregoing audio-text alignment method is implemented.

[0035] In this application, initial audio data and the corresponding target transcription text of the initial audio data are first obtained, the rhythm change rate indices corresponding to the initial audio data at different time nodes are obtained, and semantic analysis is performed on the target transcription text to obtain the importance levels of the respective initial semantic units in the target transcription text; wherein, the rhythm change rate index is used to characterize the degree of change in the speech rhythm in the initial audio data. Then, target semantic units with importance levels greater than a preset importance threshold are determined from the respective initial semantic units according to the respective importance levels, and each of the target semantic units is initially matched with the initial audio data to determine the anchor positions of each of the target semantic units in the initial audio data. Finally, timestamps are dynamically assigned to the initial audio data based on the respective rhythm change rate indices to obtain corresponding target audio data, the target audio data is divided into different target audio segments based on the respective anchor positions corresponding to each of the target semantic units, and each of the target audio segments is aligned with the target transcription text based on the respective timestamps. It can be seen that by deeply analyzing the rhythm characteristics of the audio data and the semantics corresponding to the transcription text, this application can obtain complex factors such as the speech rate change, pauses, intonation fluctuations, and emotional expressions in the audio data, thereby improving the accuracy of audio and text alignment; by assigning timestamps to the audio data according to the rhythm change rate index corresponding to the audio data, the simple uniform assignment strategy is avoided, and it can be ensured that the timestamps of the segments with high complexity in the audio are more dense, thus ensuring the alignment accuracy of the positions with high complexity in the audio. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required for use in the description of the embodiments or the prior art. Obviously, the drawings in the following description are only the embodiments of the present invention, and for those of ordinary skill in the art, other drawings can be obtained according to the provided drawings without creative efforts.

[0037] Figure 1 It is a schematic flowchart of a method for aligning audio and text disclosed in this application;

[0038] Figure 2 It is a specific flowchart of a method for aligning audio and text disclosed in this application;

[0039] Figure 3 It is a flowchart of text semantic analysis disclosed in this application;

[0040] Figure 4 It is a specific flowchart of a method for aligning audio and text disclosed in this application;

[0041] Figure 5A flowchart for detecting the mismatched region between audio and text disclosed in this application;

[0042] Figure 6 A schematic structural diagram of an audio-text alignment device disclosed in this application;

[0043] Figure 7 A structural diagram of an electronic device disclosed in this application. Detailed implementation manners

[0044] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all of the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0045] Currently, the existing audio-text alignment methods cannot guarantee the accuracy when aligning audio with text. For this reason, this application provides an audio-text alignment method, which aligns audio with text by combining the rhythm characteristics of audio and the semantics of text, thereby ensuring the alignment accuracy of audio and text.

[0046] See Figure 1 As shown, an audio-text alignment method disclosed in an embodiment of the present invention includes:

[0047] Step S11: Obtain the initial audio data and the target transcription text corresponding to the initial audio data, obtain the rhythm change rate indexes respectively corresponding to the initial audio data at different time nodes, and perform semantic analysis on the target transcription text to obtain the importance degrees of the respective initial semantic units in the target transcription text; wherein, the rhythm change rate index is used to characterize the degree of speech rhythm change in the initial audio data.

[0048] The audio-text alignment method provided in this embodiment is applied to the SRATA system (Speech Rhythm-Aware and Semantic-Guided Adaptive Timestamp Alignment System), hereinafter referred to as the system for short. This system stems from in-depth research on the process of human speech understanding. Research shows that when humans process speech information, they not only focus on the pronunciation content but also subconsciously perceive speech rhythm, stress position, and semantic structure. These factors jointly affect the perception and understanding of the temporal distribution of speech. Traditional alignment algorithms often ignore these natural characteristics, resulting in poor performance when dealing with real speech. This system redefines alignment as an integrated process of speech rhythm understanding and semantic structure recognition, rather than a simple signal matching problem. The system design solves several key technical problems: how to quantify speech rhythm features (i.e., the rhythm change rate index) and convert it into a basis for timestamp allocation; how to identify semantically important nodes in the text and use them as alignment anchors (i.e., anchor points); how to design an alignment algorithm that can ensure both accuracy and control the computational complexity. Among them, the overall process of the audio-text alignment method in this implementation is as follows Figure 2 As shown, first, it is necessary to perform semantic analysis on the text separately, and extract the rhythm change rate index of the audio in different time windows. Then, based on the semantic analysis results of the text and the rhythm change rate index of the audio, the audio and the text are aligned. The overall pseudocode of the audio-text alignment method in this embodiment is as follows:

[0049] Algorithm: SRATA;

[0050] Input: Audio file A, transcribed text T;

[0051] Output: Alignment timestamps for each word in T;

[0052] / / Step 1: Preprocess the audio and text;

[0053] preprocessed_audio = preprocess(A)

[0054] preprocessed_text = preprocess(T)

[0055] / / Step 2: Extract multi-scale rhythm features from the audio;

[0056] rhythm_features = extract_rhythm_features(preprocessed_audio)

[0057] / / Step 3: Calculate the rhythm change rate index;

[0058] RVI = calculate_RVI(rhythm_features)

[0059] / / Step 4: Analyze the semantic structure of the text;

[0060] semantic_units = analyze_semantics(preprocessed_text)

[0061] / / Step 5: Calculate semantic importance scores;

[0062] importance_scores = calculate_semantic_importance(semantic_units)

[0063] / / Step 6: Identify semantic anchors;

[0064] anchor_points = identify_anchors(semantic_units, importance_scores)

[0065] / / Step 7: Use the anchors for coarse alignment;

[0066] segments = coarse_align(preprocessed_audio, preprocessed_text, anchor_points)

[0067] / / Step 8: Perform fine alignment for each paragraph;

[0068] aligned_timestamps = []

[0069] for segment in segments:

[0070] / / Apply the adaptive error tolerance mechanism;

[0071] segment_with_tolerance = apply_error_tolerance(segment)

[0072] / / Perform rhythm-aware fine alignment;

[0073] segment_timestamps = fine_align(segment_with_tolerance, RVI)

[0074] aligned_timestamps.append(segment_timestamps)

[0075] / / Step 9: Apply global consistency constraints;

[0076] final_timestamps = apply_global_constraints(aligned_timestamps)

[0077] return final_timestamps

[0078] In this embodiment, it is necessary to obtain the rhythm change rate index of the initial audio data at different time points. It should be noted that speech rhythm is one of the core features of human language expression, which is reflected in multiple aspects such as speech rate changes, pause distributions, and stress patterns. Traditional alignment algorithms usually assume that speech has a constant rate, resulting in insufficient accuracy when dealing with real speech. SRATA designed a speech rhythm perception mechanism that can dynamically adjust the timestamp density distribution by analyzing the speech rate changes, pause patterns, and intonation contours in the audio. This mechanism is based on an in-depth study of the manifestation forms of speech rhythm: speech rate changes usually reflect the importance and complexity of the content; pauses often mark the boundaries of semantic units; stress emphasizes key information points. The system constructs a mathematical description model of speech rhythm through multi-dimensional speech rhythm feature extraction.

[0079] The system innovatively introduces the Rhythm Variation Index (RVI), which is a quantitative indicator comprehensively reflecting the degree of speech rhythm change. RVI is calculated by weighted fusion of factors such as speech rate changes, pause distributions, and stress patterns. Based on RVI, the system constructs a timestamp density mapping function to assign appropriate timestamp density values to each time point in the audio. This non-uniform distribution strategy makes the timestamp distribution more in line with the characteristics of real speech and significantly improves the alignment accuracy. Specifically, the system extracts multi-dimensional features from the audio signal, including short-time energy changes, pause point identification, pitch and pitch contour analysis, and local speech rate measurement. These features together constitute a comprehensive description of speech rhythm. Based on these features, the system calculates the Rhythm Variation Index (RVI), which is a quantitative indicator comprehensively reflecting the degree of speech rhythm change. The rhythm change rate index is a quantitative indicator reflecting the degree of speech rhythm change, and its mathematical expression is as follows:

[0080] ;

[0081] Where: represents the rhythm change rate index at time point t, represents the normalized speech rate change at time point t, represents the normalized pause density at time point t, represents the normalized stress pattern at time point t, is a weight factor that satisfies ; The calculation of each component is as follows:

[0082] ;

[0083] where, is the local speech rate at time point t (such as syllables per second), is the average speech rate of the entire audio, is the maximum speech rate deviation observed in the audio.

[0084] ;

[0085] where, is the local pause density at time point t (such as the total pause duration within the window around t), is the maximum pause density observed in the audio.

[0086] ;

[0087] where, is the local energy change at time point t (representing the stress pattern), is the maximum energy change observed in the audio.

[0088] To balance local and global rhythm features, the system adopts a multi-scale rhythm analysis method, considering rhythm features in different time windows simultaneously, and obtaining the final rhythm description through weighted fusion. This method not only retains sensitivity to local changes but also maintains global consistency, solving the difficulties of traditional methods in dealing with speech changes at different scales. Correspondingly, in this embodiment, the process of obtaining the rhythm change rate index corresponding to different time nodes of the initial audio data may specifically include: obtaining the speech rate change situation, pitch information, and pause situation corresponding to different time nodes of the initial audio data respectively, and determining the target weights corresponding to the speech rate change situation, pitch information, and pause situation respectively; performing weighted fusion on the speech rate change situation, pitch information, and pause situation corresponding to different time nodes respectively based on each target weight to obtain the rhythm change rate index corresponding to different time nodes of the initial audio data. Specifically, the system adopts a multi-scale analysis method, considering rhythm features in different time windows simultaneously, where the short-time window (100 - 300 ms) captures local speech rate changes, the medium-time window (0.5 - 2 s) captures the structural features of semantic units, and the long-time window (3 - 10 s) reflects the overall expression pattern. By weighted fusion of features at different scales, the system obtains a comprehensive and accurate rhythm description.

[0089] In addition, due to significant differences in the speech rhythm characteristics of different speakers and different scenarios (i.e., speech scenarios), such as lectures, conversations, and readings, the embodiments of the present application also consider the adaptability issues of different speaking styles (i.e., speech styles); correspondingly, the process of obtaining the rhythm change rate indices corresponding to different time nodes of the initial audio data may specifically include: obtaining the speech scenario and speech style corresponding to the initial audio data, automatically adjusting the target parameters corresponding to the rhythm change rate index according to the speech scenario and the speech style, and obtaining the rhythm change rate indices corresponding to different time nodes of the initial audio data according to the adjusted parameters; specifically, the system designs a style adaptation mechanism, which automatically adjusts the rhythm analysis parameters by analyzing the overall characteristics of the audio content to adapt to different speech styles. For example, for a fast-paced speech, the system will adjust the sensitivity of the speech rate change; for content with rich emotional expressions, the system will increase the weight of pitch changes. By integrating semantic analysis into the alignment process, the alignment is no longer a simple signal matching, but an intelligent process with language understanding capabilities. This method not only improves the alignment accuracy but also enhances the system's adaptability to different language styles and content types.

[0090] Step S12: Determine target semantic units with an importance level greater than a preset importance threshold from each of the initial semantic units, and perform a preliminary match between each of the target semantic units and the initial audio data to determine the anchor positions of each of the target semantic units in the initial audio data.

[0091] The semantic anchor technology is one of the core innovations of this system, aiming to identify the semantically important nodes in the text and use them as reliable reference points for alignment. This technology stems from the observation of the human speech understanding process: when people listen to speech content, they often first capture the key words and then understand the complete content around these key points. The system has developed a multi-dimensional semantic importance evaluation method through in-depth research on linguistic principles and natural language processing technologies, considering three dimensions: syntactic centrality, semantic density, and topic relevance. In addition, matching semantic anchors with audio features is another key challenge. The system designs a semantic-acoustic feature matching algorithm to find the most likely matching points by analyzing the possible acoustic manifestations of semantically important nodes in the audio. To improve the matching accuracy, the system adopts a multi-feature fusion strategy, considering energy features, frequency features, and time-domain features.

[0092] In this embodiment, the semantic anchor technology first performs semantic unit analysis. Among them, the process of scoring the importance level of semantic units is as Figure 3As shown, it includes syntactic analysis of the text to identify core grammatical components such as subject, predicate, and object; semantic role labeling to determine the function of each word in the semantic structure; and named entity recognition to find special entities. On this basis, the system evaluates the importance of each semantic unit and quantifies the importance degree of the semantic unit from three dimensions: grammatical centrality, semantic density, and topic relevance. Among them, grammatical centrality reflects the status of a word in the syntactic structure, and core components such as the subject and predicate have a higher grammatical centrality; semantic density measures the amount of information carried by a word, and technical terms, key concepts, etc. often have a higher semantic density; topic relevance evaluates the degree of association between a word and the core topic, and words directly related to the central issue have a higher topic relevance. Specifically, the semantic importance scoring model in this system evaluates the importance degree of semantic units from three dimensions:

[0093] ;

[0094] Among them, is the semantic importance score of the word or phrase w, is the grammatical centrality score, is the semantic density score, is the topic relevance score, is the weight factor, satisfying . The calculation of each component is as follows:

[0095] ;

[0096] Among them, is the importance level of the grammatical role i (such as subject = 1.0, predicate = 0.8, etc.), is the binary indicator (1 if w has the role i, otherwise 0).

[0097] ;

[0098] Among them, is the word frequency of w in the document, is the inverse document frequency of w in the reference corpus, is the specificity score measuring the professionalism / technicality of the term.

[0099] ;

[0100] Among them, is the vector representation of the word w in the semantic space, is the vector representation of the topic, is the cosine similarity function.

[0101] After identifying the semantically important nodes, the system matches them with the audio features to determine the anchor point positions. The system designs a semantic and acoustic feature matching algorithm to analyze the possible acoustic manifestations of semantically important nodes in the audio, such as stress positions, pitch variations, energy peaks, etc., and combines the context information to find the most likely matching points.

[0102] To improve the matching accuracy, the system adopts a multi-feature fusion strategy, comprehensively considering the energy feature, frequency feature, and time-domain feature. The energy feature reflects the change in speech intensity, and important words often have higher energy values; the frequency feature captures the pitch and tone contour, and important content is usually accompanied by specific pitch variations; the time-domain feature analyzes the duration and rhythm pattern, and the expression of core content often has a unique time structure. By weighted fusion of these features, the system can accurately find the positions of semantically important nodes in the audio.

[0103] Correspondingly, the process of performing semantic analysis on the target transcription text to obtain the importance degree of each initial semantic unit in the target transcription text may specifically include: performing semantic analysis on the target transcription text to obtain the syntactic centrality score, semantic density score, and topic relevance score corresponding to each initial semantic unit in the target transcription text; wherein, the syntactic centrality score represents the status of each initial semantic unit in the corresponding syntactic structure, the semantic density score represents the amount of information contained in each initial semantic unit, and the topic relevance score represents the degree of association between each initial semantic unit and the core topic corresponding to the target transcription text; performing weighted fusion on the syntactic centrality score, semantic density score, and topic relevance score to obtain the importance degree of each initial semantic unit in the target transcription text.

[0104] Among them, the process of obtaining the syntactic centrality score, semantic density score, and topic relevance score corresponding to each initial semantic unit in the target transcription text may specifically include: obtaining the syntactic centrality score corresponding to each initial semantic unit based on the part of speech of each initial semantic unit; obtaining the semantic density score corresponding to each initial semantic unit according to the inverse document frequency corresponding to each initial semantic unit and the occurrence frequency of each initial semantic unit in the target transcription text; mapping each initial semantic unit and the core topic to the target semantic space to obtain the first vector corresponding to each initial semantic unit and the second vector corresponding to the core topic, obtaining the cosine similarity between each first vector and the second vector, and obtaining the topic relevance score corresponding to each initial semantic unit according to each cosine similarity. By identifying the highly semantically important nodes in the text and using them as reliable reference points for alignment, the alignment process between semantics and audio text is combined, improving the accuracy of audio-text alignment.

[0105] Step S13: Dynamically assign timestamps to the initial audio data based on each of the rhythm change rate indices to obtain corresponding target audio data. Divide the target audio data into different target audio segments based on each of the anchor positions corresponding to the target semantic units, and align each of the target audio segments with the target transcription text based on each of the timestamps.

[0106] In this embodiment, based on the aforementioned RVI, this embodiment further defines a timestamp density allocation function , and assigns timestamps accordingly:

[0107] ;

[0108] where is the timestamp density at time point t, is the base timestamp density, is the RVI influence factor (value range [0, 1]), is the rhythm change rate index at time point t.

[0109] The timestamp density determines the number of timestamps allocated per unit time. In regions with large speech rate changes, many pauses, or obvious accents, the RVI value is higher, resulting in a higher timestamp density, thus more accurately reflecting the speech rhythm characteristics.

[0110] For each text unit (word, phrase, etc.), its time length is calculated as:

[0111] ;

[0112] where is the time length of unit w, is the content volume of unit w (such as the number of syllables), is the cumulative timestamp density of the corresponding audio passage.

[0113] One challenge faced by the audio-text alignment method in this embodiment is: how to control the computational complexity while ensuring accuracy, especially when dealing with long audio content. The computational complexity of traditional global alignment algorithms often grows quadratically, resulting in low processing efficiency. The system in this embodiment designs a hierarchical progressive alignment strategy, which significantly improves efficiency and accuracy through a divide-and-conquer method. This strategy decomposes the alignment task into two stages: rough alignment and fine alignment. In the rough alignment stage, a preliminary alignment framework is established using the semantic anchors identified previously, and the long audio content is segmented into multiple relatively independent paragraphs. This step significantly reduces the overall complexity of the algorithm because subsequent fine alignment only needs to be performed within smaller paragraphs, avoiding the high complexity of global calculation. The pseudocode of the above process is shown as follows:

[0114] Algorithm: hierarchical_alignment;

[0115] Input: Preprocessed audio A, preprocessed text T, anchors;

[0116] Output: Aligned timestamps;

[0117] function hierarchical_alignment(A, T, anchors):

[0118] / / Step 1: Coarse segmentation using anchors;

[0119] segments = segment_using_anchors(A, T, anchors)

[0120] / / Step 2: Initialize the final aligned timestamps;

[0121] final_timestamps = an empty array of the length of T

[0122] / / Step 3: Process each paragraph independently;

[0123] for segment in segments:

[0124] audio_segment = segment.audio

[0125] text_segment = segment.text

[0126] / / Step 3a: Check for potential mismatches;

[0127] mismatches = detect_mismatches(audio_segment, text_segment)

[0128] / / Step 3b: Apply the adaptive error tolerance mechanism;

[0129] if mismatches:

[0130] audio_segment, text_segment = apply_error_tolerance(audio_segment, text_segment, mismatches)

[0131] Step 3c: Extract the rhythm features of this paragraph;

[0132] rhythm_features = extract_rhythm_features(audio_segment)

[0133] / / Step 3d: Calculate the RVI for this paragraph;

[0134] segment_RVI = calculate_RVI(rhythm_features)

[0135] / / Step 3e: Perform fine alignment using rhythm-aware timestamp assignment;

[0136] segment_timestamps = fine_align(audio_segment, text_segment, segment_RVI)

[0137] / / Step 3f: Add to the final timestamps;

[0138] final_timestamps[segment.start_idx:segment.end_idx] = segment_timestamps

[0139] / / Step 4: Apply global consistency constraints;

[0140] final_timestamps = apply_global_constraints(final_timestamps)

[0141] return final_timestamps

[0142] In the coarse alignment stage, a preliminary alignment framework is first established using the previously identified semantic anchors. These anchors usually have high confidence and can provide a reliable reference for the overall alignment. The system uses these anchors to calculate a preliminary global mapping relationship and divides the long audio content into multiple relatively independent paragraphs. Although the accuracy of this step is not high, it can provide a global structure and lay a foundation for subsequent fine alignment.

[0143] The key to coarse alignment lies in how to construct a global mapping using limited anchors. The system designs an interpolation-based mapping construction method. For the content between two anchors, the system calculates the approximate mapping relationship of the intermediate points based on the time positions and content lengths of the anchors. This method takes into account the change in content density and avoids the deviation that may be caused by simple linear interpolation. For the content before or after the anchors, the system uses extrapolation to estimate the mapping relationship of the boundary regions based on the alignment characteristics of the adjacent regions.

[0144] This rough alignment method significantly reduces the overall complexity of the algorithm because it avoids detailed comparisons on a global scale and only needs to handle the matching relationships of a small number of key points. Even for long audio of the hour level, the rough alignment stage can be completed within a few seconds, providing a good initial value for the fine alignment stage.

[0145] In the fine alignment stage, a dedicated fine alignment algorithm is applied to each paragraph. In this stage, the system combines speech rhythm features and local semantic structures to achieve precise alignment within the paragraph. Since the paragraph size is controlled, even when using a fine algorithm with a relatively high computational complexity, the overall efficiency is still acceptable.

[0146] A key challenge of the hierarchical strategy is the selection of the granularity of paragraph division. The system designs an adaptive paragraph division method that dynamically adjusts the paragraph size according to content complexity, speech clarity, and anchor point distribution. In areas with simple content and clear speech, the system will divide larger paragraphs; while in areas with complex content or unclear speech, the system will divide smaller paragraphs to increase processing accuracy. Correspondingly, the process of dividing the target audio data into different target audio segments based on the positions of the respective anchor points corresponding to each target semantic unit may specifically include: determining the information complexity corresponding to different parts of the target audio data, and dynamically setting the target paragraph lengths corresponding to different parts of the target audio data according to each information complexity and each anchor point position; where the information complexity is used to characterize the amount of information contained in different parts of the target audio data; dividing the target audio data into different target audio segments based on each target paragraph length; where the head of any target audio segment contains the same audio data as the tail of the previous adjacent target audio segment, and the tail of any target audio segment contains the same audio data as the head of the next adjacent target audio segment; that is, to ensure the coherence between paragraphs, the system applies special processing at the paragraph boundaries. One method is to set an overlapping area, that is, adjacent paragraphs share a small part of the content, and by optimizing the alignment consistency of these overlapping areas, the incoherence that may occur at the paragraph splicing is eliminated. Another method is to apply global consistency constraints, requiring that the alignment results of all paragraphs satisfy the rationality of the global order and time distribution, and adjusting the relationships between paragraphs through iterative optimization.

[0147] In addition, for the case where the audio and the transcribed text are inconsistent, this embodiment designs a method for identifying mismatched regions based on a local similarity matrix. By analyzing the local correspondence between the text and audio features, possible mismatched regions are identified. This method can not only detect obvious inconsistencies (such as missing or extra content), but also identify more subtle inconsistencies (such as synonymous substitutions and word order adjustments), and perform alignment according to the type of inconsistency between the audio and the transcribed text. By using semantic anchors to perform a preliminary alignment between the audio and the transcribed text, and then performing precise alignment on a paragraph-by-paragraph basis, the overall complexity of the algorithm is significantly reduced, and the alignment error of one paragraph will not spread to other paragraphs, improving the robustness of the system. This solves the problem of error propagation in traditional global alignment algorithms; by allocating timestamps according to the speech rhythm, the timestamp allocation is more in line with the inherent law of natural speech, thus significantly improving the alignment accuracy, especially when dealing with complex real speech content. Experiments show that when facing complex real speech (including features such as speech rate changes, pauses, repetitions, and corrections), the alignment accuracy of the audio-text alignment method in this embodiment is improved by 15 - 30% compared with the traditional method, and the computational efficiency is improved by 40 - 60%. This system has broad application prospects in the field of audio-text alignment, especially suitable for scenarios such as automatic video caption generation, speech-assisted learning platforms, oral history archive processing, and meeting record automation. By implementing this system, the alignment accuracy can be effectively improved, especially having significant advantages in dealing with speech rate changes, oral features, and long audio content.

[0148] Thus, it can be seen that by using the non-uniform allocation strategy based on RVI in this application, the real speech rhythm characteristics can be more accurately reflected; by integrating semantic analysis into the audio-text alignment process, it provides guidance at the semantic level for the alignment process, improving the alignment accuracy; by identifying the mismatched regions and types between the audio and the text, and then selecting corresponding adjustment methods for audio-text alignment, the robustness of the alignment process is improved; by performing alignment in two stages of coarse and fine, the computational complexity is significantly reduced while maintaining high precision.

[0149] Based on the foregoing embodiments, this application describes the overall process of aligning audio and text. To make the technical solution in this embodiment more complete, next, this application will elaborate on the audio-text alignment process when the audio and text are mismatched. Refer to Figure 4 As shown, an embodiment of the present invention discloses a specific audio-text alignment method, including:

[0150] Step S21, constructing a similarity matrix between the target audio data and the target transcribed text, and obtaining the matching pattern between the target audio data and the target transcribed text according to the similarity matrix.

[0151] It can be understood that in practical applications, there are often inconsistencies between the transcribed text and the audio content, such as transcription errors, oral features (repetition, correction, filler words), etc. These inconsistencies can cause traditional alignment algorithms to fail or produce incorrect results. In this embodiment, an adaptive fault tolerance mechanism is designed to detect and intelligently handle various inconsistent situations.

[0152] In this embodiment, a method for identifying mismatched regions based on a local similarity matrix is designed, and its specific process is as follows Figure 5 shown. First, it is necessary to detect the mismatched regions in the audio and the text, and then moderately adjust the audio or the text according to the corresponding mismatch types in order to align the audio and the text. By analyzing the local correspondence between the text and audio features, possible mismatched regions are identified. This method can not only detect obvious inconsistencies (such as missing or extra content), but also identify more subtle inconsistencies (such as synonym substitution, word order adjustment). Specifically, the system constructs a local similarity matrix of the text and audio features, analyzes the matching patterns in it. In an ideal situation, the values on the main diagonal should be relatively high, indicating sequential matching; while in the regions with inconsistencies, the matrix will show specific patterns, such as diagonal breaks, discrete high-value points, etc. By analyzing these patterns, the system can accurately identify possible mismatched regions. Identifying mismatched regions is the key to the adaptive fault tolerance mechanism. The system identifies the mismatched regions through the local similarity matrix L:

[0153] ;

[0154] where is the local similarity between the audio segment and the text unit , is the similarity function, based on the matching degree between the acoustic features and the text representation.

[0155] Step S22: Obtain the difference region between the target audio data and the target transcribed text based on the matching pattern, obtain the corresponding difference type of the difference region, and align each target audio segment with the target transcribed text according to the alignment strategy corresponding to the difference type.

[0156] In this embodiment, after detecting the inconsistency, the system classifies the inconsistency into multiple types, such as transcription errors, oral corrections, repeated expressions, filler words, etc. This classification is based on the structural features and context relationships of the mismatched regions, and is implemented using linguistic knowledge and pattern matching techniques. For different types of inconsistencies, the system adopts specialized processing strategies to ensure that the alignment quality is not affected.

[0157] Specifically, the mismatched regions are manifested as specific patterns in the matrix. The system analyzes these patterns to identify possible mismatch types:

[0158] ;

[0159] wherein, is the type of mismatch being judged, is the probability that the mismatch belongs to a specific type given the local similarity matrix L and the context.

[0160] After identifying the mismatch region, the system performs mismatch type analysis. Different types of inconsistencies have different characteristic patterns and processing strategies. The system classifies the inconsistencies into the following main types: Transcription error: The content in the text does not match the audio, which may be due to recognition errors or manual transcription mistakes; Spoken correction: The speaker corrects themselves during the expression, such as "The meeting will be on Wednesday this week, no, Thursday"; Repetitive expression: The speaker repeats certain content to emphasize or clarify, such as "This time, this kind of problem must, must be solved"; Fillers and discourse markers: Common fillers, pause words, etc. in spoken language, such as "um", "that", "you know"; Content omission: Some content in the audio is omitted during transcription, or some content in the transcription is omitted in the audio.

[0161] For the mismatches of this type in this embodiment, dedicated processing strategies are adopted: For transcription errors, the system applies a fuzzy matching algorithm to find the actual text that the audio content may correspond to; For spoken corrections, the system identifies the content before and after the correction and appropriately adjusts the timestamp allocation; For repetitive expressions, the system identifies the repeated content units and looks for corresponding patterns that appear multiple times in the audio; For fillers and discourse markers, the system uses a dedicated model to identify these special elements and gives appropriate treatment during the alignment process.

[0162] In addition, this embodiment does not simply apply predefined rules, but dynamically adjusts the processing strategy according to the specific characteristics and context of the inconsistency. For example, when it is recognized that a specific speaker is accustomed to using certain fillers, the system will optimize the corresponding recognition parameters; when processing content in a professional field, the system will adjust the matching sensitivity of professional terms; in transcripts with uneven quality, the system will dynamically adjust the credibility weights of different regions. This adaptive fault tolerance mechanism enables the system to handle various inconsistencies gracefully, significantly improving the robustness in real application scenarios. Even in cases where the transcription quality is poor or contains a large number of spoken language features, the system can still maintain a high alignment accuracy, greatly expanding the scope of application scenarios.

[0163] It can be seen that by deeply analyzing the rhythm characteristics of audio data and the semantics corresponding to the transcribed text, the present application can obtain complex factors such as the speech rate change, pause, intonation fluctuation, and emotional expression of the audio data, thereby improving the accuracy of audio and text alignment; by allocating timestamps to the audio data according to the rhythm change rate index corresponding to the audio data, the simple uniform allocation strategy is avoided, and it can be ensured that the timestamps of the segments with high complexity in the audio are more dense, thereby ensuring the alignment accuracy of the positions with high complexity in the audio.

[0164] See Figure 6 As shown, an audio text alignment device according to an embodiment of the present invention includes:

[0165] A semantic analysis module 11, configured to obtain initial audio data and a target transcribed text corresponding to the initial audio data, obtain a rhythm change rate index corresponding to the initial audio data at different time nodes, and perform semantic analysis on the target transcribed text to obtain the importance degree of each initial semantic unit in the target transcribed text; wherein, the rhythm change rate index is used to characterize the degree of speech rhythm change in the initial audio data;

[0166] An anchor point position determination module 12, configured to determine target semantic units with an importance degree greater than a preset importance threshold from each of the initial semantic units according to each of the importance degrees, and perform a preliminary match between each of the target semantic units and the initial audio data to determine the anchor point position of each of the target semantic units in the initial audio data;

[0167] An audio text alignment module 13, configured to dynamically allocate timestamps to the initial audio data based on each of the rhythm change rate indexes to obtain corresponding target audio data, divide the target audio data into different target audio segments based on each of the anchor point positions corresponding to each of the target semantic units, and align each of the target audio segments with the target transcribed text based on each of the timestamps.

[0168] It can be seen that by deeply analyzing the rhythm characteristics of audio data and the semantics corresponding to the transcribed text, the present application can obtain complex factors such as the speech rate change, pause, intonation fluctuation, and emotional expression of the audio data, thereby improving the accuracy of audio and text alignment; by allocating timestamps to the audio data according to the rhythm change rate index corresponding to the audio data, the simple uniform allocation strategy is avoided, and it can be ensured that the timestamps of the segments with high complexity in the audio are more dense, thereby ensuring the alignment accuracy of the positions with high complexity in the audio.

[0169] In some specific embodiments, the semantic analysis module 11 may specifically include:

[0170] A weight determination unit, configured to obtain the speech rate change situation, pitch information, and pause situation corresponding to the initial audio data at different time nodes respectively, and determine the target weights corresponding to the speech rate change situation, the pitch information, and the pause situation respectively;

[0171] A first data fusion unit, configured to perform weighted fusion on the speech rate change situation, the pitch information, and the pause situation corresponding to different time nodes respectively based on the respective target weights, so as to obtain the rhythm change rate index corresponding to the initial audio data at different time nodes respectively.

[0172] In some specific embodiments, the semantic analysis module 11 may specifically include:

[0173] A parameter adjustment unit, configured to obtain the speech scenario and speech style corresponding to the initial audio data, automatically adjust the target parameters corresponding to the rhythm change rate index according to the speech scenario and the speech style, and obtain the rhythm change rate index corresponding to the initial audio data at different time nodes respectively according to the adjusted parameters.

[0174] In some specific embodiments, the semantic analysis module 11 may specifically include:

[0175] A semantic analysis sub-module, configured to perform semantic analysis on the target transcription text to obtain the grammatical centrality score, semantic density score, and topic relevance score corresponding to each of the initial semantic units in the target transcription text; wherein, the grammatical centrality score represents the status of each of the initial semantic units in the corresponding syntactic structure, the semantic density score represents the amount of information contained in each of the initial semantic units, and the topic relevance score represents the degree of association between each of the initial semantic units and the core topic corresponding to the target transcription text;

[0176] A second data fusion unit, configured to perform weighted fusion on the grammatical centrality score, the semantic density score, and the topic relevance score to obtain the importance degree of each initial semantic unit in the target transcription text.

[0177] In some specific embodiments, the semantic analysis sub-module may specifically include:

[0178] A first score acquisition unit, configured to obtain the grammatical centrality score corresponding to each of the initial semantic units based on the part of speech of each of the initial semantic units;

[0179] A second score acquisition unit, configured to obtain the semantic density score corresponding to each of the initial semantic units according to the inverse document frequency corresponding to each of the initial semantic units and the occurrence frequency of each of the initial semantic units in the target transcription text;

[0180] A third score acquisition unit, configured to map each of the initial semantic units and the core topic to a target semantic space, so as to obtain a first vector corresponding to each of the initial semantic units and a second vector corresponding to the core topic, obtain a cosine similarity between each of the first vectors and the second vector, and obtain the topic relevance score corresponding to each of the initial semantic units according to each of the cosine similarities.

[0181] In some specific embodiments, the audio-text alignment module 13 may specifically include:

[0182] A paragraph length setting unit, configured to determine the information complexity corresponding to different parts of the target audio data, and dynamically set the target paragraph length corresponding to different parts of the target audio data according to each of the information complexities and each of the anchor positions; wherein, the information complexity is used to characterize the amount of information contained in different parts of the target audio data.

[0183] A paragraph division unit, configured to divide the target audio data into different target audio segments based on each of the target paragraph lengths; wherein, the head of any target audio segment contains the same audio data as the tail of the previous adjacent target audio segment, and the tail of any target audio segment contains the same audio data as the head of the next adjacent target audio segment.

[0184] In some specific embodiments, the audio-text alignment module 13 may specifically include:

[0185] A matching mode acquisition unit, configured to construct a similarity matrix between the target audio data and the target transcription text, and obtain a matching mode between the target audio data and the target transcription text according to the similarity matrix.

[0186] An audio-text alignment unit, configured to obtain a difference region between the target audio data and the target transcription text based on the matching mode, obtain a difference type corresponding to the difference region, and align each of the target audio segments with the target transcription text according to an alignment strategy corresponding to the difference type.

[0187] Furthermore, an embodiment of the present application also discloses an electronic device. Figure 7 It is a structural diagram of an electronic device 20 shown according to an exemplary embodiment, and the content in the figure cannot be considered as any limitation on the scope of use of the present application.

[0188] Figure 7Schematic diagram of the structure of an electronic device 20 provided by an embodiment of the present application. The electronic device 20 may specifically include: at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input / output interface 25, and a communication bus 26. Among them, the memory 22 is used to store a computer program, and the computer program is loaded and executed by the processor 21 to implement the relevant steps in the audio-text alignment method disclosed in any of the foregoing embodiments. In addition, the electronic device 20 in this embodiment may specifically be an electronic computer.

[0189] In this embodiment, the power supply 23 is used to provide working voltage for each hardware device on the electronic device 20; the communication interface 24 can create a data transmission channel between the electronic device 20 and external devices, and the communication protocol it follows is any communication protocol applicable to the technical solution of the present application, and specific limitations are not imposed here; the input / output interface 25 is used to obtain external input data or output data to the outside, and its specific interface type can be selected according to specific application needs, and specific limitations are not imposed here.

[0190] In addition, as a carrier for resource storage, the memory 22 may be a read-only memory, a random access memory, a disk, or an optical disc, etc., and the resources stored thereon may include an operating system 221, a computer program 222, etc., and the storage method may be temporary storage or permanent storage.

[0191] Among them, the operating system 221 is used to manage and control each hardware device and the computer program 222 on the electronic device 20, and it may be Windows Server, Netware, Unix, Linux, etc. In addition to the computer program that can be used to complete the audio-text alignment method executed by the electronic device 20 disclosed in any of the foregoing embodiments, the computer program 222 may further include a computer program that can be used to complete other specific tasks.

[0192] Furthermore, the present application also discloses a computer-readable storage medium for storing a computer program; wherein, when the computer program is executed by a processor, the audio-text alignment method disclosed above is implemented. For the specific steps of this method, reference may be made to the corresponding content disclosed in the foregoing embodiments, and details are not described herein again.

[0193] In this specification, the various embodiments are described in a progressive manner. Each embodiment focuses on the differences from other embodiments. The same or similar parts among the various embodiments can be referred to each other. For the device disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple, and the relevant parts can be referred to the description of the method part.

[0194] Those skilled in the art may further realize that the units and algorithm steps of each example described in combination with the embodiments disclosed herein can be implemented by electronic hardware, computer software, or a combination of the two. To clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described according to functions in the above description. Whether these functions are executed in a hardware or software manner depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of this application.

[0195] The steps of the methods or algorithms described in combination with the embodiments disclosed herein can be directly implemented by hardware, software modules executed by a processor, or a combination of the two. The software modules can be placed in a random access memory (RAM), internal memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium well-known in the technical field.

[0196] Finally, it should also be noted that in this document, relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the term "comprising", "including" or any other variation thereof is intended to cover non-exclusive inclusion, so that a process, method, article or device comprising a series of elements not only includes those elements, but also includes other elements not expressly listed, or elements inherent to such process, method, article or device. Without further limitation, an element defined by the statement "comprising an..." does not exclude the existence of additional identical elements in the process, method, article or device comprising the element.

[0197] The technical solutions provided in this application have been introduced in detail above. Specific examples have been used in this document to illustrate the principles and implementation manners of this application. The description of the above embodiments is only used to help understand the method and its core idea of this application; at the same time, for those of ordinary skill in the art, according to the idea of this application, there will be changes in the specific implementation manners and application scopes. In summary, the content of this specification should not be construed as a limitation to this application.

Claims

1. An audio-text alignment method, characterized in that Including: Obtain initial audio data and the corresponding target transcription text of the initial audio data, obtain the rhythm change rate index corresponding to the initial audio data at different time nodes, and perform semantic analysis on the target transcription text to obtain the importance degree of each initial semantic unit in the target transcription text; wherein, the rhythm change rate index is used to characterize the degree of speech rhythm change in the initial audio data; Determine target semantic units with an importance degree greater than a preset importance threshold from each of the initial semantic units according to each of the importance degrees, and perform a preliminary match between each of the target semantic units and the initial audio data to determine the anchor positions of each of the target semantic units in the initial audio data; Dynamically assign timestamps to the initial audio data based on each of the rhythm change rate indexes to obtain corresponding target audio data, divide the target audio data into different target audio segments based on each of the anchor positions corresponding to each of the target semantic units, and align each of the target audio segments with the target transcription text based on each of the timestamps.

2. The audio-text alignment method according to claim 1, wherein The obtaining the rhythm change rate index corresponding to the initial audio data at different time nodes includes: Obtain the speech rate change situation, pitch information, and pause situation corresponding to the initial audio data at different time nodes, and determine the target weights corresponding to the speech rate change situation, the pitch information, and the pause situation respectively; Perform weighted fusion on the speech rate change situation, the pitch information, and the pause situation corresponding to different time nodes based on each of the target weights to obtain the rhythm change rate index corresponding to the initial audio data at different time nodes.

3. The audio-text alignment method according to claim 1, wherein The obtaining the rhythm change rate index corresponding to the initial audio data at different time nodes includes: Obtain the speech scene and speech style corresponding to the initial audio data, automatically adjust the target parameters corresponding to the rhythm change rate index according to the speech scene and the speech style, and obtain the rhythm change rate index corresponding to the initial audio data at different time nodes according to the corresponding adjusted parameters.

4. The audio-text alignment method according to claim 1, wherein The performing semantic analysis on the target transcription text to obtain the importance degree of each initial semantic unit in the target transcription text includes: Perform semantic analysis on the target transcription text to obtain the syntactic centrality score, semantic density score, and topic relevance score corresponding to each of the initial semantic units in the target transcription text; wherein, the syntactic centrality score characterizes the status of each of the initial semantic units in the corresponding syntactic structure, the semantic density score characterizes the amount of information contained in each of the initial semantic units, and the topic relevance score characterizes the degree of association between each of the initial semantic units and the core topic corresponding to the target transcription text; Perform weighted fusion on the syntactic centrality score, the semantic density score, and the topic relevance score to obtain the importance degree of each initial semantic unit in the target transcription text.

5. The audio-text alignment method according to claim 4, wherein Obtaining the syntactic centrality score, semantic density score, and topic relevance score corresponding to each of the initial semantic units in the target transcription text includes: Obtaining the syntactic centrality score corresponding to each of the initial semantic units based on the part of speech of each of the initial semantic units; Obtaining the semantic density score corresponding to each of the initial semantic units according to the inverse document frequency corresponding to each of the initial semantic units and the occurrence frequency of each of the initial semantic units in the target transcription text; Mapping each of the initial semantic units and the core topic to a target semantic space to obtain a first vector corresponding to each of the initial semantic units and a second vector corresponding to the core topic, obtaining the cosine similarity between each of the first vectors and the second vector, and obtaining the topic relevance score corresponding to each of the initial semantic units according to each of the cosine similarities.

6. The audio-text alignment method according to claim 1, wherein Partitioning the target audio data into different target audio segments based on each of the anchor positions corresponding to each of the target semantic units includes: Determining the information complexity corresponding to different parts of the target audio data, and dynamically setting the target paragraph length corresponding to different parts of the target audio data according to each of the information complexities and each of the anchor positions; wherein, the information complexity is used to characterize the amount of information contained in different parts of the target audio data; Partitioning the target audio data into different target audio segments based on each of the target paragraph lengths; wherein, the head of any target audio segment contains the same audio data as the tail of the previous adjacent target audio segment, and the tail of any target audio segment contains the same audio data as the head of the next adjacent target audio segment.

7. The audio-text alignment method according to any one of claims 1 to 6, characterized in that Aligning each of the target audio segments with the target transcription text based on each of the timestamps includes: Constructing a similarity matrix between the target audio data and the target transcription text, and obtaining a matching pattern between the target audio data and the target transcription text according to the similarity matrix; Obtaining a difference region between the target audio data and the target transcription text based on the matching pattern, obtaining a difference type corresponding to the difference region, and aligning each of the target audio segments with the target transcription text according to the alignment strategy corresponding to the difference type.

8. An audio-text alignment device, characterized in that, Including: A semantic analysis module, configured to obtain initial audio data and a target transcription text corresponding to the initial audio data, obtain a rhythm change rate index corresponding to the initial audio data at different time nodes, and perform semantic analysis on the target transcription text to obtain the importance degree of each initial semantic unit in the target transcription text; wherein, the rhythm change rate index is used to characterize the degree of speech rhythm change in the initial audio data. The anchor position determination module is configured to determine target semantic units with importance levels greater than a preset importance threshold from each of the initial semantic units according to each of the importance levels, and perform a preliminary match between each of the target semantic units and the initial audio data to determine the anchor positions of each of the target semantic units in the initial audio data; The audio-text alignment module is configured to dynamically assign timestamps to the initial audio data based on each of the rhythm change rate indices to obtain corresponding target audio data, divide the target audio data into different target audio segments based on each of the anchor positions corresponding to each of the target semantic units, and align each of the target audio segments with the target transcription text based on each of the timestamps.

9. An electronic device, characterized in that, Comprising: A memory for storing a computer program; A processor for executing the computer program to implement the audio-text alignment method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, For storing a computer program, which when executed by a processor implements the audio-text alignment method according to any one of claims 1 to 7.

Images