Heterogeneous modal sequence-oriented alignment method, device and system and storage medium

Abstract

The invention discloses a heterogeneous modal sequence-oriented alignment method, device and system and a storage medium. The method comprises the following steps: extracting a frame-level acoustic vector and a fundamental frequency of an acoustic feature flow, and generating a first modal sequence; performing word-level probability coding on the language modality to obtain a word boundary probability vector; and extracting a lip movement track from the visual key point matrix, and constructing a second modal sequence. And further, calculating a mutual information function of the acoustic and language modals based on the joint probability density, calculating a distance function of the acoustic and visual modals based on the weighted distance, dynamically determining a modal weight according to the distance function, and generating a frame-level fusion feature vector. A multi-target cost matrix containing inter-frame differences, mutual information loss and key point matching errors is constructed through fusion features, path search is executed based on maximum slope constraint, a minimum cost path is obtained, a unified time alignment sequence is output, and accurate time sequence synchronization of multi-modal information is achieved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of big data, and particularly relates to a method, device and system for aligning heterogeneous modal sequences and a storage medium. BACKGROUND

[0002] In the field of multi-modal audio-video processing, the time alignment between heterogeneous modalities such as speech, image and text is a key technical link to realize content synchronization generation and multi-modal interaction. For example, in virtual digital human driving, speech subtitle synchronization, intelligent customer service question and answer generation, short video explanation, emotional broadcasting and other applications, the system usually needs to match the acoustic signal with the language and text information, the modal data such as the lip movement or facial expression track in the video for time consistency, so as to ensure the coordination and fluency of speech, subtitles and picture content at the perception level.

[0003] In the prior art, in order to reduce the implementation complexity, most schemes adopt a phased and pair-by-pair alignment strategy, that is, the speech signal is first matched with the text, and then the speech is registered with the video frame, so as to indirectly establish the time mapping relationship between the multi-modalities. Although this kind of scheme is relatively simple in structure, it has obvious limitations in actual application. Due to the essential differences in the acquisition, processing and expression of each modality, the step-by-step alignment strategy is prone to amplify the alignment error layer by layer, especially in unstable scenes such as fast pronunciation speed change, sudden change of spoken language expression, strong dialect interference or occlusion of mouth shape in the video, the generated subtitles, speech and pictures often have obvious time dislocation, which affects the synchronization experience of the final content.

[0004] In addition, some existing methods lack sufficient modeling capability when dealing with nonlinear changes between different modalities or dynamic modality weight adjustment, and cannot judge the stability and reliability of the current modality in real time during the alignment process, so they still rely heavily on the specific modality when the interference of the specific modality is intensified, further aggravating the alignment deviation.

[0005] In summary, the existing multi-modal alignment method still has obvious room for improvement in precision robustness, dynamic adaptability and perceptual consistency, and an alignment optimization method that can fuse multi-modal features, has high time sequence analysis capability and adapts to speech speed and modality changes is urgently needed. SUMMARY

[0006] To solve the above technical problems, the present application provides a method, device and system for aligning heterogeneous modal sequences and a storage medium.

[0007] The technical solutions provided in the present application are described as follows: The first aspect of the present application provides a method for aligning heterogeneous modal sequences, comprising: The acoustic feature vectors and the fundamental frequency features of each frame are extracted based on a window function and an overlap rate from an input acoustic feature stream AFS to form a first modal sequence; A word boundary probability vector sequence is generated by performing word-level probability coding on an input language modal sequence LBPV. Lip motion trajectories are extracted from an input visual spatial key point matrix VSKM, and a second modal sequence is constructed based on a target key point set. A mutual information function of the first modal sequence and the word boundary probability vector sequence is calculated based on a joint probability density. A distance function of the first modal sequence and the second modal sequence is calculated based on a weighted distance. A modal weight coefficient is dynamically calculated based on the mutual information function and the distance function, and a frame-level fusion feature vector is constructed according to the modal weight coefficient. A multi-objective cost matrix is constructed based on the frame-level fusion feature vector, each element of the multi-objective cost matrix including an inter-frame difference between the frame-level fusion feature vectors, a mutual information loss term, and a key point matching error term. A path search is performed on the multi-objective cost matrix based on a preset maximum slope constraint parameter, and at least one minimum cost path satisfying continuity and monotonicity is obtained. A unified time alignment sequence is output according to the minimum cost path.

[0008] Optionally, the mutual information function of the first modal sequence and the word boundary probability vector sequence is calculated based on a joint probability density, including: Time window sliding sampling is performed on the first modal sequence and the word boundary probability vector sequence at a fixed step length, and local feature vector pairs in a corresponding time segment are extracted. A joint probability density function of each set of local feature vector pairs is constructed using a KDE method, wherein the kernel function is a Gaussian kernel, and the bandwidth parameter is adaptively adjusted according to the sampled local entropy value. The marginal probability density function and the joint probability density function are estimated respectively. The mutual information function is calculated based on a preset information entropy formula.

[0009] Optionally, the distance function of the first modal sequence and the second modal sequence is calculated based on a weighted distance, including: A preset key point subset is selected from the normalized key point set of each frame to construct a lip motion feature vector. The acoustic feature vector of each frame in the first modal sequence is linearly mapped to a feature space with the same dimension as the lip motion feature vector, and a comparison pair is obtained. A distance function between the two frames in the comparison pair is calculated by a preset distance function.

[0010] Optionally, the multi-objective cost matrix is constructed based on the frame-level fusion feature vector, each element of the multi-objective cost matrix including an inter-frame difference value between the frame-level fusion feature vectors, an mutual information loss term and a key point matching error term, comprising: For any two frame-level fusion feature vectors, an inter-frame difference value is calculated, which is the square of the Euclidean distance between the two frame-level fusion feature vectors, for measuring the degree of change of the features between the current time and the adjacent time; The mutual information loss term is constructed by taking the negative value of the logarithm of the preset minimum constant after the calculated frame-level mutual information value is added; The key point matching error term is calculated based on the weighted squared difference between the visual key point coordinates estimated by the acoustic feature flow and the real visual key point coordinates, wherein the weighting coefficients are set according to the importance of each key point; The inter-frame difference value, the mutual information loss term and the key point matching error term are linearly combined according to the preset weighting coefficients to obtain the value of the corresponding element in the multi-objective cost matrix, the weighting coefficients are used to control the proportion of each loss term in the overall cost, and the sum of the three weighting coefficients is one; The numerical values of all elements in the multi-objective cost matrix are mapped to a preset interval with the maximum value in each term as a reference.

[0011] Optionally, based on the preset maximum slope constraint parameter, path search is performed on the multi-objective cost matrix, and at least one minimum cost path satisfying continuity and monotonicity is obtained, comprising: The maximum slope constraint parameter in path search is determined in advance to limit the maximum moving step length of the path in the time dimension and the modal dimension; Based on the maximum slope constraint parameter, dynamic path search is performed on the multi-objective cost matrix under the constraints of monotonicity and continuity to obtain a path sequence that minimizes the cumulative cost, the path sequence is composed of multiple nodes, each node corresponding to a multi-modal frame combination; The uniform time alignment sequence is output according to the minimum cost path, comprising: The minimum cost path is backtracked from the starting boundary to the terminal boundary of the multi-objective cost matrix, and the modal time index combination corresponding to the minimum cost path is extracted; According to each modal time index combination contained in the minimum cost path, a uniform reference time axis is established, the original data sequences of each heterogeneous modal are synchronously mapped onto the uniform reference time axis, and a uniform time alignment sequence is output.

[0012] Optionally, the method further comprises: receiving a face key point sequence matrix containing multiple frames of video images, the matrix being two-dimensional spatial coordinate information of each key point in a face region identified from consecutive frame images based on a visual acquisition module; extracting a lip region motion trajectory corresponding to a target frame from the face key point sequence matrix based on a predefined lip key point index set, the lip key point index set containing coordinate indexes of key anatomical feature points including at least upper lip, lower lip, and corner of mouth; converting the lip region motion trajectory sequence into a second modality sequence.

[0013] Optionally, the method further comprises: performing first-order and second-order difference calculations on the frame-level fusion feature vectors of consecutive multiple time frames in the frame-level fusion feature vector to obtain local change rates and local change curvatures of the frame-level fusion feature vectors; comparing the local change curvatures with a preset curvature threshold, and identifying a target time segment with a local change that is violent according to a comparison result; based on the target time segment, adjusting a matching penalty weight of a corresponding target time segment in the multi-objective cost matrix; performing smoothing processing on the adjusted multi-objective cost matrix.

[0014] Optionally, the method further comprises: calculating a modality sequence confidence between the current frame and the previous and next frames for each single modality sequence in the frame-level fusion feature vector; when a modality sequence confidence of a target single modality sequence is lower than a preset confidence threshold, reducing a contribution degree of the target single modality sequence to the frame-level fusion feature vector through a scaling coefficient.

[0015] The second aspect of the present application provides an alignment device for heterogeneous modality sequences, the device comprising: a first modality sequence generation unit configured to extract acoustic feature vectors and fundamental frequency features containing each frame based on a window function and an overlap rate from an input acoustic feature stream AFS to form a first modality sequence; a second modality sequence generation unit configured to perform word-level probability coding on an input language modality sequence LBPV to generate a word boundary probability vector sequence; a third modality sequence generation unit configured to extract a lip motion trajectory in an input visual spatial key point matrix VSKM and construct a second modality sequence based on a target key point set; a first calculation unit configured to calculate a mutual information function of the first modality sequence and a word boundary probability vector sequence based on a joint probability density; a second calculation unit configured to calculate a distance function of the first modality sequence and the second modality sequence based on a weighted distance; a third calculation unit configured to dynamically calculate a modality weight coefficient based on the mutual information function and the distance function, and construct a frame-level fusion feature vector according to the modality weight coefficient; a matrix construction unit configured to construct a multi-objective cost matrix based on the frame-level fusion feature vector, each element of the multi-objective cost matrix including an inter-frame difference between the frame-level fusion feature vectors, a mutual information loss term, and a key point matching error term; a path searching unit configured to perform path searching on the multi-objective cost matrix based on a preset maximum slope constraint parameter, and obtain at least one minimum cost path satisfying continuity and monotonicity; an output unit configured to output a unified time alignment sequence according to the minimum cost path.

[0016] The third aspect of the present application provides an alignment system for heterogeneous modality sequences, the system comprising: a processor, a memory, an input / output unit, and a bus; the processor is connected to the memory, the input / output unit, and the bus; the memory stores a program, and the processor invokes the program to execute the method of the first aspect and any optional aspect of the first aspect.

[0017] The fourth aspect of the present application provides a computer readable storage medium, the computer readable storage medium stores a program, and the program executes the method of the first aspect and any optional aspect of the first aspect when executed on a computer.

[0018] As can be seen from the above technical solutions, the present application has the following beneficial effects: The present application effectively breaks through the problem of error accumulation caused by "pairwise modality alignment" in the prior art by constructing a frame-level fusion feature vector and introducing the coupling relationship among the three types of modalities of speech, text, and vision, and performing time sequence modeling and alignment in a unified feature space. Compared with the traditional double modality stage processing method, the present application realizes global joint optimization of multi-modality time sequence synchronization by constructing a fusion cost matrix at the frame level and uniformly performing optimal path searching, so that the output of the subtitle text, speech broadcast, and lip movement has higher consistency and synchronization accuracy, and significantly improves the interactive naturalness and viewing quality of the multi-modality content generation system. BRIEF DESCRIPTION OF DRAWINGS

[0019] In order to more clearly illustrate the technical solutions in the present application, the drawings needed in the embodiments will be briefly introduced as follows. Obviously, the drawings in the following description only constitute some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained without creative labor based on these drawings.

[0020] Figure 1 An embodiment flowchart of the alignment method for heterogeneous modal sequences provided in the present application; Figure 2 An embodiment flowchart of step S103 in the alignment method for heterogeneous modal sequences provided in the present application; Figure 3 An embodiment flowchart of step S104 in the alignment method for heterogeneous modal sequences provided in the present application; Figure 4 An embodiment flowchart of step S107 in the alignment method for heterogeneous modal sequences provided in the present application; Figure 5 An embodiment flowchart of step S108 in the alignment method for heterogeneous modal sequences provided in the present application; Figure 6 An embodiment structure diagram of the alignment device for heterogeneous modal sequences provided in the present application; Figure 7 An embodiment structure diagram of the alignment system for heterogeneous modal sequences provided in the present application. DETAILED DESCRIPTION

[0021] Please refer to Figure 1 , the present application first provides an embodiment of an alignment method for heterogeneous modal sequences, which comprises: S101, based on a window function and an overlap rate, extracting acoustic feature vectors and fundamental frequency features containing each frame from an input acoustic feature flow AFS to form a first modal sequence; In the present embodiment, the "heterogeneous modal sequence" includes but is not limited to an acoustic modal sequence (for example: MFCC features, fundamental frequency features, etc.), a language modal sequence (for example: word boundary encoding vectors corresponding to text content), and a second modal sequence (for example: lip movement key point trajectories, etc.). The method is suitable for multi-modal speech broadcasting, virtual digital person driving, subtitle alignment, etc. scenes, and can realize high-precision alignment of three modalities in the time dimension.

[0022] In this step, a sliding window sampling process is performed on the input acoustic feature stream (AFS), and the sampling window size can be set to 25 ms, and the frame shift is set to 10 ms. Acoustic features are extracted for each time window, including but not limited to 13-dimensional Mel frequency cepstral coefficients (MFCC), first-order derivatives, and second-order derivatives, which are combined into a 39-dimensional feature vector; further, the pitch and pitch contour are extracted to form a one-dimensional or two-dimensional pitch feature sequence. Finally, a first modal sequence composed of multiple frames is obtained, as well as a corresponding pitch sequence.

[0023] In this embodiment, it is assumed that the input acoustic feature stream AFS is a 10-second long speech signal with a sampling rate of 16 kHz. In order to extract frame-level features, a conventional speech feature extraction strategy is used for preprocessing.

[0024] Frame division is performed on the original speech signal, with a window length of 25 ms (i.e. 400 sampling points) and a frame shift of 10 ms (160 sampling points), resulting in about 1000 time frames in the 10-second audio.

[0025] For each frame, the following feature extraction operations are performed: using the Mel frequency cepstral coefficient (MFCC) method, 13-dimensional MFCC vectors are extracted; further, first-order difference (Δ) and second-order difference (ΔΔ) are calculated, totaling 26 dimensions; and 39-dimensional MFCC feature vectors are obtained by merging statics. The pitch of the frame, its log-F0, and the pitch contour (if any) are extracted in parallel to form pitch features (e.g. 2-4 dimensions); finally, each frame forms an acoustic feature vector with a length of about 41-43 dimensions.

[0026] Next, the first modal sequence and the pitch sequence are constructed. The MFCC features of each frame are combined in chronological order to form a first modal feature sequence with a length of about 1000; correspondingly, the pitch features of each frame are combined to form a pitch sequence; both share the time axis as the acoustic modal input in the method of the present application.

[0027] For example, at frame 300, the MFCC feature vector is [12.1, -4.3, 0.7,..., -1.2] (39 dimensions in total), and the corresponding pitch is F0 = 110 Hz, so the complete acoustic description of this frame is: Frame_300={ 'mfcc':[12.1,-4.3,0.7,...,-1.2], 'pitch':110.0 } S102, word-level probability encoding is performed on the input language modality sequence LBPV to generate a word boundary probability vector sequence; The input language modality sequence LBPV is processed at the word level, and a BIO (Begin-Inside-Outside) labeling method is performed on each word boundary to generate a time-aligned boundary probability vector. Each time corresponds to a real-valued probability vector, which is used to describe the probability of belonging to the beginning of a word, the middle of a word, or a non-word state at the current time point, as a structured representation of the language modality, forming a language modality vector sequence with the same length as the acoustic modality.

[0028] In this embodiment, it is assumed that the language modality input corresponding to the 10-second speech described in S101 is a text sequence, and the transcription result is: "Welcome to use the multi-modal alignment system." To realize the structured processing of the text, the input language modality sequence (Language Boundary Probability Vector, LBPV) is processed at the word level in this step, mainly through the following specific ways.

[0029] First, the original text is segmented to obtain a word sequence, for example: ["Welcome", "use", "multi-modal", "alignment", "system", "."] It is assumed that there are 1000 frames corresponding to the audio, and based on the preliminary alignment result of the speech recognition system (ASR) or the prediction result of the acoustic model, the start and end time of each word is estimated.

[0030] For example, "Welcome" corresponds to frames [0-80], "use" corresponds to frames [81-160], "multi-modal" corresponds to frames [161-350], and so on.

[0031] At each frame position, it is labeled whether the frame is the beginning (B), middle (I), or does not belong to any word (O) of a word.

[0032] A word boundary probability estimator (BiLSTM-CRF, Transformer, or attention-based model can be used) is introduced, which outputs a 3-dimensional probability vector for each frame: P_frame_t=[P_B(t), P_I(t), P_O(t)] For example: Frame 162 (belongs to the first word of "multi-modal") → [0.89, 0.08, 0.03] Frame 163 (belongs to the middle word of "multi-modal") → [0.03, 0.95, 0.02] The frame-level probability vectors are spliced in time sequence to form a language mode sequence with the same length as the acoustic mode: LBPV=[ [0.82, 0.13, 0.05], #Frame0 [0.91, 0.06, 0.03], #Frame1 ... [0.03, 0.95, 0.02], #Frame163 ... ] Each frame is represented by a three-dimensional probability vector to represent its semantic boundary features in the word structure, which can be directly used for mutual information estimation with the acoustic mode.

[0033] Finally, the text is mapped to the frame-level probability structure, and discrete language units are introduced into the continuous time domain, making them comparable with the acoustic mode, and forming the basis for subsequent mode alignment.

[0034] S103, extracting the lip motion trajectory in the input visual space key point matrix VSKM, and constructing a second mode sequence based on the target key point set; Based on the pre-acquired video key point matrix (VSKM), the key point coordinates of the face region including the lips, the lower jaw, etc. are extracted, and the 68-point face grid or the three-dimensional mouth key point subset is preferably used. By connecting the key point trajectory and calculating the inter-frame Euclidean change rate, the lip motion trajectory is obtained. The trajectory is reorganized by frame to form a second mode sequence that is time-aligned with the acoustic sequence, which is used to represent the state of the evolution of the lip shape change over time.

[0035] In this embodiment, the step S103 is used to extract the visual features of the face region captured in the continuous frame video into a structured second mode sequence, referring to Figure 2 , the specific implementation includes the following sub-steps: S1031, receiving a face key point sequence matrix containing multiple frames of video images, the matrix being two-dimensional spatial coordinate information of each key point in the face region identified from continuous frame images based on a visual acquisition module; The input video stream is received, and the visual acquisition module (such as Dlib, OpenFace, MediaPipe, or a self-developed key point detection model) is used to perform face detection and key point positioning operations on each frame.

[0036] The two-dimensional spatial coordinates of the K face key points identified for each frame form a frame-level key point matrix: ; The V frepresents a set of facial key points extracted from the f-th image in the video sequence. Wherein: f: video frame index, representing the f-th image currently processed; K: total number of facial key points detected in a single image frame; (x i , y i ): two-dimensional spatial coordinates of the i-th facial key point in the image coordinate system; i = 1, 2, …, K, K: key point index number.

[0037] S1032, based on a pre-defined set of lip key point indexes, extract the lip region motion trajectory corresponding to the target frame from the facial key point sequence matrix, and construct it as a lip region motion trajectory sequence, the set of lip key point indexes contains the coordinate indexes of key anatomical feature points such as upper lip, lower lip and corner of mouth; For a video image sequence containing T frames, a key point time sequence matrix is constructed: ; The VSKM represents a facial key point time sequence matrix composed of multiple consecutive video images.

[0038] Wherein: T represents the total number of frames contained in the video segment; V1, V2, …, V T respectively represent the frame-level key point matrix extracted from the 1st to the Tth image.

[0039] According to the pre-set lip key point index set (such as the points with index range [61-88] in MediaPipe), the coordinate points of the upper lip, lower lip, corner of mouth and other regions in each frame are extracted to form a lip region subset.

[0040] The lip key point set is extracted from each frame: ; Wherein is the lip region key point index set, (x i , y i ): two-dimensional spatial coordinates of the i-th key point belonging to the lip region; i ∈ I lip : indicates that the key point index belongs to the lip key point index set.

[0041] L f represents the set of lip key points.

[0042] The T-frame lip key point sequence is spliced into a lip motion trajectory sequence: ; In the formula, L represents the lip movement trajectory sequence, L1, L2, …, L T respectively represent the lip key point sets extracted from the 1st frame to the Tth frame.

[0043] In an optional embodiment, the lip point sequence can also be subjected to PCA dimension reduction or relative motion vector construction based on mesh to enhance the description capability of the lip contour change.

[0044] S1033, convert the lip region movement trajectory sequence into a second modality sequence.

[0045] The lip movement trajectory sequence is organized into a time-ordered visual feature sequence by frame. The sequence can be constructed in the following two ways: A. Directly use two-dimensional coordinate splicing vectors to form frame-level visual features; B. Extract key point movement increment vectors between adjacent frames to express lip dynamic features.

[0046] The final second modality sequence VMS is consistent with the number of audio frames, which is suitable for subsequent cross-modality alignment calculation.

[0047] S104, calculate the mutual information function of the first modality sequence and the word boundary probability vector sequence based on joint probability density; The joint probability estimation method is adopted to respectively calculate the joint distribution between the acoustic feature frames and the language boundary vector frames. The mutual information function (Mutual Information, MI) is used to measure the information sharing degree between the two modalities. The mutual information value can be estimated based on experience, kernel density estimation or based on a deep estimation model to reflect the time coupling strength between the acoustic signal and the language boundary.

[0048] This step S104 aims to calculate the mutual information function between the acoustic modality sequence and the language modality sequence to measure the correlation between the two in time sequence, thereby providing a structural basis for modality weight calculation and fusion alignment path. Referring to Figure 3 , the process can be realized through the following sub-steps: S1041, respectively perform time window sliding sampling on the first modality sequence and the word boundary probability vector sequence with a fixed step length, and extract the local feature vector pair in the corresponding time segment; The first modality sequence (acoustic feature flow) and the language modality sequence (word boundary probability vector sequence) are respectively subjected to fixed step length sliding window operation, and the corresponding local feature vector pair in the frame-level time segment is extracted.

[0049] Let the sliding window length be w (such as 15 frames) and the step length be s (such as 5 frames). For each time window: Extract the local feature vector in the window from the acoustic modality: ; wherein A wi represents the local feature vector extracted from the first modality sequence in the current time window; a t , a t+1 , …, a t+w−1 respectively represent the acoustic feature vector corresponding to the t-th frame to the t+w-1-th frame; Each ai is a high-dimensional vector, which can include MFCC, fundamental frequency, energy, and other acoustic description information.

[0050] Extract the language boundary probability vector of the corresponding window from the language modality: ; T win represents the language boundary probability vector extracted from the language modality sequence in the current time window; t i represents the language feature vector corresponding to the i-th frame, which is used to describe the probability distribution of the time position being the beginning of a word, the middle of a word, or a non-word state.

[0051] Pair them to form a feature pair sample set: .

[0052] X represents a feature pair sample set formed by pairing the acoustic features and the language features according to the time index in the same time window; (a i , t i ) represents a set of acoustic-language feature sample pairs observed simultaneously at the i-th frame; t≤i<t+w: indicates that the sample index range is limited to the current sliding time window.

[0053] S1042, a joint probability density function of each set of local feature vectors is constructed using the KDE method, wherein the kernel function is a Gaussian kernel, and the bandwidth parameter is adaptively adjusted according to the sampled local entropy value; The kernel density estimation (KDE) method is used to model the feature pairs sampled above, and a local joint probability density function is constructed.

[0054] The kernel function used can be a Gaussian kernel function: ; wherein: K(x) represents the kernel function used for kernel density estimation; In this embodiment, a Gaussian kernel function is selected, which has the form: x represents the distance vector between the sample point and the estimated point in the feature space; 2π is a normalization constant, which is used to ensure that the kernel function integral is 1; exp(·) is an exponential function, which is used to realize the smooth weight decay.

[0055] The kernel bandwidth parameter h can adopt a local entropy self-adaptive adjustment mechanism, that is, first calculate the entropy H(X) of the local window; set the minimum bandwidth h min , the maximum bandwidth h max , and the bandwidth is adaptively adjusted according to the following formula: ; Where H(X) represents the Shannon information entropy of the sample set X in the current sliding window, which is used to measure the complexity of the local sample distribution; H max represents the maximum entropy value for normalization, which can be pre-set according to the feature dimension or historical statistical results.

[0056] Estimate the joint density function P(a, t): ; Where P(a, t) represents the joint probability density function of the acoustic feature random variable a and the language feature random variable t; n represents the number of sample pairs in the current sliding window, which is generally equal to the length w of the sliding window mentioned above; (ai, ti) represents the i-th acoustic-linguistic feature sample pair; Kh(·) represents the kernel function with bandwidth h.

[0057] S1043, respectively estimate the marginal probability density function and the joint probability density function; On the basis of the above, one-sided KDE is performed on the acoustic modality and the language modality respectively to obtain the marginal probability distribution: ; .

[0058] In this embodiment, P(a) represents the marginal probability density function of the acoustic feature random variable a; P(t) represents the marginal probability density function of the language feature random variable t; The above marginal probability density is obtained by one-dimensional kernel density estimation or numerical integral approximation of the joint probability density; the calculation process is consistent with the joint probability density estimation in form, and only the distribution of a single modality is counted.

[0059] S1044, calculate the mutual information function based on the preset information entropy formula.

[0060] According to the definition formula of mutual information, the information entropy formula can be used for estimation: ; In this embodiment, MI(A, T) represents the value of the mutual information function of the acoustic modality sequence A and the language modality sequence T in the current sliding window; the value of the mutual information function is a non-negative scalar, which is used to measure the correlation degree of the two modalities in the statistical sense; when the two modalities are highly synchronized, the value of the mutual information function increases; when they are independent or weakly correlated, the value decreases.

[0061] In this embodiment, the mutual information value corresponding to each sliding window is obtained by integrating and approximately solving the mutual information of all windows; the mutual information value can be extended to a frame-level mutual information function, and the mutual information scalar between each frame is obtained through smooth interpolation.

[0062] For example, if an audio mutation (such as a burst sound) and a subtitle word boundary are observed to synchronously rise in a certain sliding window, the MI value in the window will be significantly higher than that of a semantic redundancy or a silent segment, reflecting the improvement of the time dependence between modalities.

[0063] In this embodiment, the mutual information value is first calculated in a sliding time window, and each sliding time window corresponds to a mutual information scalar, which is used to represent the statistical correlation between the first modality sequence and the language modality sequence in the time window.

[0064] To extend the sliding window-level mutual information value to a frame-level mutual information function, preferably, the mutual information value of each sliding time window is assigned to the center frame or a preset reference frame covered by the time window, and the time index of the reference frame is taken as the corresponding frame position of the mutual information value.

[0065] For the time frames between adjacent sliding time windows that do not directly correspond to mutual information values, the mutual information scalar of the corresponding time frame is obtained by smooth interpolation calculation of the mutual information values corresponding to adjacent sliding windows, so as to construct a continuous frame-level mutual information function sequence. The smooth interpolation method can be linear interpolation, weighted average interpolation or other interpolation methods that can maintain time continuity.

[0066] In the above manner, the mapping from the sliding window-level mutual information value to the frame-level mutual information function is realized, so that each time frame corresponds to a unique mutual information scalar, which is used for subsequent modality weight calculation and multi-target cost construction.

[0067] It should be noted that the information entropy calculation formula is used to quantify the statistical dependence between the joint probability distribution and the marginal probability distribution, and its calculation process takes the estimation result of the joint probability density function as a prerequisite, and obtains the mutual information value through algebraic operation in the form of information entropy of the joint probability density function and the marginal probability density function.

[0068] It should be understood that the "calculating mutual information function based on joint probability density" and "calculating mutual information function based on information entropy formula" in the specification are different levels of description of the same mutual information calculation process, where the former describes the probability modeling basis of mutual information, and the latter describes the implementation of numerical calculation based on the probability modeling basis, and the two do not constitute different or parallel mutual information calculation methods.

[0069] Therefore, the calculation method of the mutual information function in this embodiment is always based on the estimation result of the joint probability density, and the information entropy formula is only used as a mathematical expression form of mutual information calculation, and does not constitute an independent calculation path other than the joint probability density.

[0070] For the foregoing information entropy formula, in this embodiment, the calculation of the mutual information function takes the local feature vector pair obtained by time window sliding sampling as the basic processing unit. For the local feature vector pair in each sliding time window, first, according to the kernel density estimation result constructed in step S1042, the joint probability density function and the marginal probability density function in the corresponding time window are obtained.

[0071] The integral operation in the mutual information function is used to statistically sum the probability density function in its definition domain, and the integral interval is implicitly determined by the value range of the local feature vector in the current sliding time window. In the specific implementation process, the integral operation is approximated by summing the values of the probability density function at discrete sampling points, rather than using continuous analytical integration.

[0072] Therefore, the double integral expression shown in the specification is a theoretical representation of mutual information calculation, which corresponds to the discrete accumulation calculation of the joint probability density value and the marginal probability density value of all sampling points in the current sliding time window in engineering implementation. Through the above method, an independent mutual information value can be obtained for each sliding time window, thereby forming a mutual information function sequence that changes with time.

[0073] In this embodiment, the upper and lower limits of the integral interval are not explicitly given, and it does not affect the calculation result of the mutual information function, and its essence is automatically limited by the support set of the sample distribution in the current sliding time window.

[0074] S105, calculating a distance function of the first modality sequence and the second modality sequence based on weighted distance; In this step, the system calculates the distance function between the extracted second modality sequence and the acoustic modality sequence to quantify the similarity of their time sequence features in space, thereby providing a calculation basis for subsequent modality weight estimation and alignment path construction.

[0075] The weighted distance calculation is performed on the feature difference between each pair of acoustic frame and visual frame, preferably using Euclidean distance or Mahalanobis distance metric. To enhance dynamic adaptability, the fundamental frequency variation rate can be used as a weighting coefficient to weight and accumulate the position error between the lip key points. Finally, the matching distance function between the acoustic modality and the visual modality is generated as an important parameter for subsequent frame fusion.

[0076] Specifically, a preset key point subset can be selected from each normalized frame of lip key point set to construct a lip movement feature vector; the key point subset can include representative points of the upper lip center, lower lip center, and bilateral corners, etc., to ensure that the extracted features can reflect the main mouth shape changes during speech production. The features can be obtained by two-dimensional coordinate splicing, Euclidean distance coding, or principal component compression, for example, reducing the mouth region features of each frame to a fixed dimension vector to form a frame-level lip movement sequence. The acoustic feature vector of each frame in the first modality sequence is transformed to a feature space with the same dimension as the lip movement feature vector through linear mapping, and a comparison pair is obtained; specifically, the acoustic feature vector of each frame in the first modality sequence is projected to a feature space consistent with the lip movement feature vector through a linear transformation function or a neural network mapping function obtained by training. The goal of this mapping operation is to make the acoustic features comparable to the visual features in terms of spatial structure, thereby supporting inter-frame similarity calculation. To enhance the expression ability of the mapping, the projection function can be implemented by a single-layer or multi-layer perceptron, and its parameters are optimized by a small batch of paired data sets in the pre-training stage to ensure a reliable alignment basis between normal speech and visual mouth shape. Each pair of mapped acoustic feature vectors and lip movement feature vectors is compared at the frame level by a preset distance function to calculate the distance between them. The distance function can be Euclidean distance, cosine similarity inverse, or Manhattan distance, and the specific choice depends on the characteristics of the sequence to be aligned. In this embodiment, the normalized weighted Euclidean distance is preferably used to consider the physical position weight of the key points and the variation amplitude of the mapped acoustic features.

[0077] S106, dynamically calculating a modality weight coefficient based on the mutual information function and the distance function, and constructing a frame-level fusion feature vector according to the modality weight coefficient; The purpose of this step is to dynamically fuse acoustic, language and visual modalities at frame-level granularity to construct frame-level fusion feature vectors reflecting the semantic collaborative relationship of multi-modalities. This fusion process is based on the aforementioned mutual information function and modal distance function, and the modal weight coefficients are dynamically calculated to weight and combine the three types of modal features. Based on the aforementioned mutual information function and distance function, the modal weight coefficients are dynamically calculated according to the time frame. The modal weight coefficients are used to adjust the contribution degree of different modalities in the fusion features. Preferably, adaptive adjustment is made according to the confidence (such as speech activity detection result, text prediction probability, video key point recognition completeness) and local change rate of mutual information or distance. According to the weight coefficients, the acoustic, language and visual features of each frame are weighted and summed to obtain a sequence of frame-level fusion feature vectors.

[0078] In a specific implementation, the system first uses the mutual information function obtained in step S104 (such as the frame-level mutual information sequence between acoustic and language) and the cross-modal distance function obtained in step S105 (such as the frame-level distance sequence between acoustic and visual) to quantitatively evaluate the correlation between modalities corresponding to each time frame. In order to further enhance the adaptive ability, the system can also introduce a modal confidence evaluation mechanism to model the signal integrity and reliability of each modality.

[0079] Taking the t-th frame as an example, the system can estimate the confidence of the three modalities according to the following indicators: For the acoustic modality (audio), according to the voice activity detection (VAD) result, if it is detected as a silent frame, a lower confidence is given; For the language modality (text), the language model prediction probability (such as BERT softmax score) of the corresponding vocabulary of the frame can be used as the text confidence; For the visual modality (image), the visual integrity score can be given according to whether complete lip key points are detected in the frame (such as a missing rate <10%).

[0080] Combining the above confidence scores, the system normalizes the mutual information value or distance change rate between modalities at each frame to estimate the modal weight coefficient of the frame. For example, if the audio signal is clear, the text prediction confidence is high, but the lip key points are missing seriously, then the audio and text modal weight coefficients of the frame can be made larger, and the visual modal weight coefficient is relatively weakened.

[0081] Let the feature vectors corresponding to the audio, language and visual modalities be a t , l t , v t , and the modal weights be α t , β t , γ t , then the final frame-level fusion feature vector F tThe calculation is as follows: ; The weighting coefficients satisfy the following constraints:

[0082] The following is a specific example to further illustrate this: When the system processes an audio-visual segment of a child telling a story, the mutual information change rate of the audio increases and the confidence of VAD decreases in segments with sudden changes in speech rate or pauses. In this case, the system dynamically reduces the weight of the audio modality. Conversely, text prediction, due to its clear contextual structure, has higher confidence, so the system increases the contribution of the language modality. If the lip movements are clear and the keypoint integrity rate is high in the visual modality, then the visual modality is also given a higher weight. Ultimately, the fused feature vectors can more accurately capture the co-variations between pronunciation and semantics, which is beneficial for building a more robust alignment path. This dynamic modality fusion mechanism effectively avoids the problem of insufficient adaptability of fixed weighting strategies in multimodal temporal alignment, and improves the stability of multimodal alignment in scenarios with noise, occlusion, and speech rate changes.

[0083] S107. Construct a multi-objective cost matrix based on the frame-level fusion feature vectors. Each element of the multi-objective cost matrix includes the inter-frame difference between the frame-level fusion feature vectors, the mutual information loss term, and the key point matching error term. Using the frame-level fusion feature vectors, a three-modal joint multi-objective cost matrix is ​​constructed. Each cost element consists of three parts: (1) the inter-frame difference of the fusion vector between the current frame and the next frame; (2) the mutual information loss term between corresponding frame pairs, i.e., the negative logarithm of the mutual information value; and (3) the visual keypoint error term, i.e., the weighted squared error based on the estimated keypoint and the actual keypoint. The three terms are multiplied by adjustable weight coefficients λ1, λ2, and λ3, respectively, and normalized and combined into the total cost.

[0084] In this step, the system further generates a multi-objective cost matrix for modality alignment path search based on the constructed frame-level fused feature vector sequence. Each element in the cost matrix comprehensively considers multiple loss terms to reflect the matching cost between heterogeneous modalities in the time dimension. See also... Figure 4 This process can be implemented in the following specific ways: S1071. For any two frame-level fused feature vectors, calculate the inter-frame difference, where the inter-frame difference is the square of the Euclidean distance between the two frame-level fused feature vectors, used to measure the degree of feature change between the current time and the adjacent time. First, for any two frame-level fused feature vectors F i With F jThe square of the Euclidean distance is calculated and used as the inter-frame difference. This difference term is used to measure the magnitude of change in the fused features between adjacent time frames. The larger the value, the more significant the difference in modal information at the corresponding time point, indicating a sudden change or switching trend.

[0085] S1072. The negative logarithm of the calculated frame-level mutual information value is taken after adding a preset minimum constant and then constructing it as a mutual information loss term. The frame-level mutual information value MI(a) obtained from the aforementioned mutual information calculation steps is used as an example. i , l j Add a very small constant (e.g., ϵ1=10) -6 The negative logarithm of the equation is then used to construct a mutual information loss term. This loss term reflects the degree of decrease in semantic relevance between acoustic and linguistic modalities, thereby suppressing low mutual information intervals during alignment.

[0086] S1073. Calculate the key point matching error term based on the weighted squared difference between the estimated visual key point coordinates and the actual visual key point coordinates, where the weighting coefficients are set according to the importance of each key point. Furthermore, the system constructs a keypoint matching error term based on the difference between the visual keypoint coordinates (from the acoustic input) estimated in step S105 through linear mapping or a neural network and the actual visual modal keypoints. The error term is calculated by squaring the coordinate differences of all keypoints and then summing them using weighted averages. The weight w for each keypoint is... k The weights are pre-set based on their importance in lip movement expression. For example, the center point of the upper lip and the corner of the mouth may be given higher weights to ensure that points that are more sensitive to changes in pronunciation have higher weights in error assessment.

[0087] S1074. The inter-frame difference, mutual information loss term, and key point matching error term are linearly combined according to preset weighting coefficients to obtain the values ​​of the corresponding elements in the multi-objective cost matrix. The weighting coefficients are used to control the proportion of each loss term in the overall cost, and the sum of the three weighting coefficients is one. Subsequently, the inter-frame difference, mutual information loss term, and keypoint matching error term are linearly combined according to preset weighting coefficients λ1, λ2, and λ3 to obtain the value of each element in the multi-objective cost matrix, i.e.: ; The weighting coefficients satisfy λ1+λ2+λ3=1, which controls the proportion of different loss terms in the final generation value. This combination method provides an adjustable fusion strategy, allowing for flexible setting of weight parameters for different scenarios.

[0088] Among them, C i , jrepresents the comprehensive matching cost value between the first modality frame with time index i and the second modality frame with time index j, used to construct a multi-modal alignment path, a timing matching matrix or a dynamic programming search space; the cost value is a non-negative real number, and its numerical value is used to measure the inconsistency degree of the two frames in the multi-modal feature level, and the smaller the value is, the higher the matching degree is.

[0089] i represents the frame index in the first modality sequence (such as the acoustic modality or the visual lip movement modality), used to identify the i-th feature frame on the time axis; j represents the frame index in the second modality sequence (such as the language modality or the text boundary modality), used to identify the candidate time position for alignment with the first modality.

[0090] λ1 represents the first feature loss term L feat the weight coefficient of (i, j), used to adjust the contribution proportion of the feature similarity in the total cost function; λ2 represents the mutual information loss term L MI the weight coefficient of (i, j), used to control the influence degree of the modality statistical correlation on the alignment result; λ3 represents the visual consistency loss term L vis the weight coefficient of (i, j), used to adjust the importance of the visual modality constraint in the comprehensive cost; S1075, with the maximum value in each term as a reference, the numerical values of all elements in the multi-objective cost matrix are mapped to a preset interval.

[0091] To enhance the numerical stability and contrastiveness of subsequent path search, the system takes the maximum value in each term as a normalization reference, and maps all numerical values in the cost matrix C i，j to the interval [0, 1]. This normalization processing can adopt the maximum scaling method, that is, each element is divided by the maximum item in its corresponding channel, to ensure that the losses in different dimensions have comparability in numerical values, and to suppress the interference of abnormal points on path calculation.

[0092] The following is a specific, perceptible and exemplary description of step S107: Taking a "digital human news broadcast segment" as an example, the segment includes a 10-second voice explanation, which corresponds to 300 frames of acoustic features (each frame interval is 33 ms), 300 frames of lip movement visual key point sequence, and word boundary probability associated with the text content.

[0093] After frame-level fusion, assuming that the current time i = 105, j = 106, that is, the alignment cost between frames 105 and 106 is examined, the following is the specific calculation process: Assuming the fused feature vectors for frames 105 and 106 are F105 and F106 respectively, with a vector dimension of 256, the system calculates the squared Euclidean distance between them as the inter-frame difference: ; This value reflects the overall degree of modal change between current frames.

[0094] In step S104, the acoustic feature a corresponding to frame 105 is estimated. 105 With the boundary probability vector t of the 106th word in the language modality 106 The mutual information between them is 0.72. After adding a minimal constant and taking the negative logarithm, we obtain the mutual information loss: ; Through acoustic features a 105 The lip keypoints predicted by the pre-trained mapping function are v^ 105 and with real visual key points v 106 Comparison is made. Taking 20 key lip points as an example, the weighted average difference calculation yields the following results: ; in, This represents the position vector or its scalar representation (e.g., two-dimensional coordinates or normalized displacement value) of the k-th lip keypoint in the real visual modality of frame 106. This represents the estimated position of the k-th lip keypoint obtained based on acoustic feature prediction.

[0095] wk represents the weight coefficient corresponding to the k-th lip keypoint, which is used to reflect the importance of different keypoints in visual consistency assessment.

[0096] In this embodiment, L vis (105, 106) represents the visual consistency loss between the acoustic features of frame 105 and the visual keypoints of frame 106; the value 0.57 is a specific example result calculated in this embodiment, used to illustrate the calculation process, and its magnitude reflects the degree of matching of the frame pair at the visual synchronization level; the smaller the loss value, the higher the synchronization between the sound and the lip movement at that time position. This measure measures the degree of synchronization between the sound and the lip movement in that frame.

[0097] Given weighting coefficients λ1=0.4, λ2=0.3, and λ3=0.3, the three linear combinations are: ; Assuming the maximum element in the current cost matrix is ​​1.45, perform maximum value normalization: .

[0098] in, The multi-objective cost value after normalization processing is used to measure the comprehensive matching cost between the acoustic frame 105 and the visual frame 106. S108, based on the preset maximum slope constraint parameter, performing path search on the multi-objective cost matrix, and obtaining at least one minimum cost path satisfying continuity and monotonicity; On the basis of the constructed cost matrix, the maximum slope constraint of path search is set (for example, the maximum slope is 1.5), to ensure that the generated path satisfies the monotonicity and continuity constraints. Through dynamic programming or heuristic path optimization algorithm, the path with the minimum total cost is searched as the alignment path between the three modalities. The path represents the optimal time frame correspondence between modalities.

[0099] Specifically, a specific implementation of step S108 is provided below, please refer to Figure 5 : S1081, determining the maximum slope constraint parameter in path search, to limit the maximum moving step length of the path in the time dimension and the modality dimension; S1082, based on the maximum slope constraint parameter, performing dynamic path search under the monotonicity and continuity constraints on the multi-objective cost matrix, to obtain a path sequence with the minimum cumulative cost, the path sequence being composed of multiple nodes, each node corresponding to a multi-modality frame combination; Based on the multi-objective cost matrix constructed in step S107, a path search operation is performed on the matrix. Specifically, a path satisfying the time sequence monotonicity and continuity constraints is searched from the top left corner (the starting boundary) to the bottom right corner (the terminal boundary) on the two-dimensional cost matrix. Each node in the path corresponds to a group of matching combinations between different modality frames; in the path construction process, the growth direction of the path is not allowed to back off, and the step length of the growth per unit time satisfies the maximum slope constraint parameter (such as the maximum slope value is 1.5), to tolerate the speed variation or short silent fragments; among all the feasible paths, the path with the minimum cumulative cost is selected as the optimal path, denoted as the minimum cost path P*.

[0100] The minimum cost path P* is composed of multiple nodes, each node being in the form of a triple (i, j, k), where i, j, k represent the frame indexes of the acoustic modality, the language modality, and the visual modality, respectively.

[0101] The unified time alignment sequence is output according to the minimum cost path, including: backtracking from the starting boundary to the terminal boundary of the multi-objective cost matrix, and extracting the modality time index combination corresponding to the minimum cost path; In this embodiment, starting from the terminal boundary (lower right corner) of the multi-target cost matrix, the minimum cost path P* is backtracked along, and the index combination of each modality in the path is extracted node by node; Each extracted node contains a multi-modal matching pair, that is, the correspondence between an acoustic frame, a corresponding language boundary frame and a video frame index; all nodes in the path are organized in order into a time alignment index table, and the record form is as follows: ; Where t1, t2, …, tn represent time stamps on a unified time axis;(a tn , l tn , v tn ) represents the frame-level index or frame-level feature of the three modalities aligned at the unified time point tn, wherein: a tn represents the acoustic frame index or acoustic feature vector corresponding to the acoustic modality at time point tn; l tn represents the language unit index or word boundary probability feature corresponding to the language modality at time point tn; v tn represents the visual frame index or lip movement feature corresponding to the visual modality at time point tn; t1, t2, …, tn: represent a sequence of discrete time stamps on a unified reference time axis.

[0102] According to the index combination of each modality in the minimum cost path, a unified reference time axis is established, and the original data sequences of each heterogeneous modality are synchronously mapped to the unified reference time axis, and a unified time alignment sequence is output.

[0103] A unified reference time axis is constructed, and the length n of the minimum cost path is used as a reference to divide the time period, which is equally divided into n synchronous time points; the time indexes of the original sequences in the acoustic modality, the language modality and the visual modality are respectively mapped according to the triple index in the minimum cost path, so that the three modalities have unique corresponding frames at each reference time point; the aligned three-modality data sequence is output on the unified time axis, and the specific output structure is: ; Where each group of triple content is at the same time point tn, and is used to drive multi-modal tasks such as subtitle display, lip animation, speech synthesis, etc. in actual application.

[0104] AlignedSequence represents the unified output sequence obtained after multi-modal time alignment processing; the sequence maps the data of different modalities to the same time coordinate system based on the unified reference time axis.

[0105] represents the aligned three-modal data unit at the nth unified time point tn, wherein: represents the aligned acoustic feature or speech segment of the acoustic modality at the time point tn; represents the aligned language feature of the language modality at the time point tn, such as word boundary probability or text label; represents the aligned visual feature of the visual modality at the time point t, such as lip keypoint or mouth animation parameter.

[0106] S109, outputting a unified time alignment sequence according to the minimum cost path.

[0107] According to the pairing relationship between each time frame in the minimum cost path, the acoustic, language and visual three modalities are mapped to a unified time axis, and a fused multi-modal alignment sequence is output. This sequence can be directly used for digital human driving, synchronous subtitle rendering, speech broadcasting and other multi-modal tasks, and has the advantages of frame-level synchronization, fast response, fault tolerance to abnormal modalities, etc.

[0108] In an optional embodiment, in order to further improve the alignment robustness of the frame-level fusion feature in the scene with severe timing fluctuations, the system performs timing curvature-aware enhancement processing on the frame-level fusion feature vector. This method can further include the following steps: First and second order difference calculations are performed on the frame-level fusion feature vectors of the continuous multiple time frames in the frame-level fusion feature vector, to obtain the local change rate and local change curvature of the frame-level fusion feature vector; The local change curvature is compared with a preset curvature threshold, and a target time segment with severe local change is identified according to the comparison result; Based on the target time segment, the matching penalty weight of the corresponding target time segment in the multi-objective cost matrix is adjusted; The adjusted multi-objective cost matrix is smoothed.

[0109] Specifically, this optional embodiment first obtains the fused frame-level feature vector sequence, and performs first and second order difference operations on the sequence to obtain the local change rate (i.e. speed) and local change curvature (i.e. acceleration) of the sequence at each time frame. Preferably, the following form is used for calculation: ; Where t represents the time index, which is used to identify the position of the frame-level fusion feature vector in the time sequence; F tdenotes a frame-level fusion feature vector corresponding to a time index t, which is obtained by fusing acoustic modality, language modality and visual modality with weights; F t+1 denotes a frame-level fusion feature vector at the next time index t+1.

[0110] wherein, D t (1) denotes a first-order difference value calculated at the time index t; D t+1 (1) denotes a first-order difference value calculated at the time index t+1; D t (2) denotes a second-order difference result calculated at the time index t, which is used to describe the change degree of the change rate of the fusion feature, i.e., the local change curvature.

[0111] Then, the curvature amplitude at each time point is calculated and compared with a preset curvature threshold, so as to identify a set of time segments with local drastic changes. The set can correspond to a situation of sudden change of speech rate, jump of mouth shape, or rearrangement of text, etc.

[0112] For the target time segments identified above, an enhanced penalty is imposed on the matching cost items in the corresponding multi-target cost matrix, i.e., the cost of the non-optimal matching path is increased at these time points, so as to enhance the alignment sensitivity of the system to these drastic change areas.

[0113] In order to avoid too large gradient fluctuation introduced by the penalty factor, a sliding window can be used to smooth the adjusted cost matrix, so that the cost surface is more continuous and stable, and the stability of subsequent path search is improved.

[0114] In another optional embodiment, in order to improve the dynamic adaptability of the contribution of each modality in the fusion feature, a modality confidence adjustment mechanism is further introduced, and the specific steps are as follows: For each single-modality sequence in the frame-level fusion feature vector, the modality sequence confidence between the current frame and the previous and next frames is calculated. When the modality sequence confidence of the target single-modality sequence is lower than a preset confidence threshold, the contribution of the target single-modality sequence to the frame-level fusion feature vector is reduced through a scaling coefficient.

[0115] In this embodiment, for the frame-level fusion feature vector of each frame, three modality sub-vectors of the frame are obtained respectively: an acoustic modality sub-vector A t , a language modality sub-vector L t , and a visual modality sub-vector V t .

[0116] For each modality, its confidence value between the current frame and the adjacent frame is calculated respectively. For example: For acoustic modality, whether the current frame is in the speech segment can be judged based on the voice activity detection (VAD) result; For language modality, the language boundary confidence of the current frame can be measured according to the mean or variance of the word boundary prediction probability value; For visual modality: the visual confidence can be calculated according to the face key point recognition integrity score of the current frame, or whether the lip region is missing, etc.

[0117] When the confidence of any one modality in the current frame is lower than a preset threshold (such as 0.3), the system will apply scaling operation to the modality feature, reducing its influence on the fusion feature. For example, the original weight can be set as α=0.5, β=0.3, γ=0.2, if the visual modality confidence is insufficient, then γ is multiplied by a decay coefficient (such as 0.2), and the overall weight is re-normalized.

[0118] The embodiments of the device and system involved in the present application are described below: Referring to Figure 6 The present application also provides an embodiment of an alignment device for a heterogeneous modality sequence, which comprises: A first modality sequence generation unit 601 is configured to extract acoustic feature vectors and fundamental frequency features containing each frame based on a window function and an overlap rate from an input acoustic feature stream AFS, to form a first modality sequence; A second modality sequence generation unit 602 is configured to perform word-level probability coding on an input language modality sequence LBPV to generate a word boundary probability vector sequence; A third modality sequence generation unit 603 is configured to extract lip motion trajectories in an input visual space key point matrix VSKM, and to construct a second modality sequence based on a target key point set; A first calculation unit 604 is configured to calculate a mutual information function of the first modality sequence and the word boundary probability vector sequence based on a joint probability density; A second calculation unit 605 is configured to calculate a distance function of the first modality sequence and the second modality sequence based on a weighted distance; A third calculation unit 606 is configured to dynamically calculate a modality weight coefficient based on the mutual information function and the distance function, and to construct a frame-level fusion feature vector according to the modality weight coefficient; A matrix construction unit 607 is configured to construct a multi-objective cost matrix based on the frame-level fusion feature vector, each element of the multi-objective cost matrix containing an inter-frame difference between the frame-level fusion feature vectors, a mutual information loss term and a key point matching error term; The path searching unit 608 is configured to perform path searching on the multi-target cost matrix based on a preset maximum slope constraint parameter, and obtain at least one minimum cost path satisfying continuity and monotonicity. The output unit 609 is configured to output a unified time alignment sequence according to the minimum cost path.

[0119] Optionally, the first calculation unit 604 is specifically configured to: perform time window sliding sampling on the first modality sequence and the word boundary probability vector sequence respectively at a fixed step length, and extract a local feature vector pair in a corresponding time segment; adopt a KDE method to construct a joint probability density function of each group of the local feature vector pair, wherein a kernel function is a Gaussian kernel, and a bandwidth parameter is adaptively adjusted according to a local entropy value of sampling; respectively estimate an edge probability density function and the joint probability density function; based on a preset information entropy, calculate a mutual information function.

[0120] Optionally, the second calculation unit 605 is specifically configured to: select a preset key point subset from each frame of the normalized lip key point set to construct a lip movement feature vector; linearly map each acoustic feature vector in the first modality sequence to a feature space with the same dimension as the lip movement feature vector, and obtain a comparison pair; calculate a distance function between two frames in the comparison pair through a preset distance function.

[0121] Optionally, the matrix construction unit 607 is specifically configured to: for any two frame-level fusion feature vectors, calculate a frame difference value, the frame difference value being a square of a Euclidean distance between the two frame-level fusion feature vectors, for measuring a degree of change of features between a current time and an adjacent time; take a negative value of a logarithm of the calculated frame-level mutual information value plus a preset minimum constant, and construct as a mutual information loss term; based on a weighted squared difference between the visual key point coordinates estimated from the acoustic feature flow and the real visual key point coordinates, calculate a key point matching error term, wherein a weighting coefficient is set according to an importance of each key point; linearly combine the frame difference value, the mutual information loss term and the key point matching error term according to preset weighting coefficients to obtain a value of a corresponding element in the multi-target cost matrix, the weighting coefficients being used to control a proportion of each loss term in the overall cost, and a sum of the three weighting coefficients being one; map numerical values of all elements in the multi-target cost matrix to a preset interval with reference to a maximum value in each term.

[0122] Optionally, the path searching unit 608 is specifically configured to: determine a maximum slope constraint parameter in path searching, to limit the maximum moving step of the path in the time dimension and the modal dimension; based on the maximum slope constraint parameter, perform dynamic path searching under monotonicity and continuity constraints on the multi-objective cost matrix to obtain a path sequence with minimum cumulative cost, the path sequence being composed of a plurality of nodes, each node corresponding to a multi-modal frame combination; Optionally, the output unit 609 is specifically configured to: backtrack from the starting boundary to the terminal boundary of the multi-objective cost matrix, and extract a modal time index combination corresponding to the minimum cost path; establish a unified reference time axis according to each modal time index combination contained in the minimum cost path, synchronously map the original data sequences of each heterogeneous modality to the unified reference time axis, and output a unified time-aligned sequence.

[0123] Optionally, the third modality sequence generation unit 603 is specifically configured to: receive a face key point sequence matrix containing a plurality of video images, the matrix being two-dimensional spatial coordinate information of each key point in a face region identified from continuous frame images based on a visual acquisition module; extract a lip region motion trajectory corresponding to a target frame from the face key point sequence matrix based on a predefined lip key point index set, and construct a lip region motion trajectory sequence, the lip key point index set containing coordinate indexes of key anatomical feature points including at least upper lip, lower lip, and corner of the mouth; convert the lip region motion trajectory sequence into a second modality sequence.

[0124] Optionally, further comprising: an adjustment unit 610, specifically configured to: perform first-order and second-order difference calculations on frame-level fusion feature vectors of a plurality of consecutive time frames in the frame-level fusion feature vector to obtain local change rates and local change curvatures of the frame-level fusion feature vectors; compare the local change curvatures with a preset curvature threshold, and identify a target time segment with a local change that is violent according to a comparison result; based on the target time segment, adjust a matching penalty weight of a corresponding target time segment in the multi-objective cost matrix; perform smoothing processing on the adjusted multi-objective cost matrix.

[0125] Optionally, further comprising: a coefficient scaling unit 611, specifically configured to: For each single-modal sequence in the frame-level fusion feature vector, calculate the modal sequence confidence between the current frame and the previous and next frames; When the modal sequence confidence of the target single-modal sequence is lower than the preset confidence threshold, then reduce the contribution of the target single-modal sequence to the frame-level fusion feature vector through the scaling coefficient.

[0126] Referring to Figure 7 The application further provides an alignment system for heterogeneous modal sequences, comprising: The processor 701, the memory 702, the input / output unit 703, and the bus 704; The processor 701 is connected with the memory 702, the input / output unit 703, and the bus 704; The memory 702 stores a program, and the processor 701 invokes the program to perform any of the above methods.

[0127] The application further relates to a computer readable storage medium, which stores a program. When the program runs on a computer, the computer performs any of the above methods.

[0128] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described system, device and unit can refer to the corresponding processes in the foregoing method embodiments, which will not be repeated here.

[0129] In the several embodiments provided in the present application, it should be understood that the disclosed system, device and method can be implemented in other ways. For example, the device embodiments described above are only schematic. The division of the units is only a logical function division. There can be another division manner in actual implementation. For example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the displayed or discussed mutual couplings or direct couplings or communication connections between the units can be indirect couplings or communication connections through some interfaces, devices or units, and can be electrical, mechanical or in other forms.

[0130] The units described as separate components can or can not be physically separate, and the components displayed as units can or can not be physical units, i.e., can be located in one place, or can be distributed on a plurality of network units. According to actual needs, some or all of the units can be selected to achieve the purpose of the embodiment scheme.

[0131] In addition, each function unit in each embodiment of the present application can be integrated in one processing unit, or each unit can be physically present separately, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of a software function unit.

[0132] When the integrated unit is realized in the form of a software function unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of the present application, essentially or in the form of a contribution to the prior art, or all or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium, and includes several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in each embodiment of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, read-only memory), a random access memory (RAM, random access memory), a magnetic disk or an optical disk, and various media that can store program codes.

Claims

1. An alignment method for heterogeneous modal sequences, characterized in that, The method includes: For the input acoustic feature stream AFS, the acoustic feature vectors and fundamental frequency features of each frame are extracted based on the window function and overlap rate to form the first mode sequence; Word-level probabilistic encoding is performed on the input language modality sequence LBPV to generate a word boundary probability vector sequence; The lip motion trajectory is extracted from the input visual spatial keypoint matrix (VSKM), and a second modality sequence is constructed based on the target keypoint set. The mutual information function between the first modality sequence and the word boundary probability vector sequence is calculated based on the joint probability density. The distance function between the first modal sequence and the second modal sequence is calculated based on the weighted distance. The modality weight coefficients are dynamically calculated based on the mutual information function and the distance function, and a frame-level fusion feature vector is constructed based on the modality weight coefficients. A multi-objective cost matrix is ​​constructed based on the frame-level fusion feature vectors. Each element of the multi-objective cost matrix includes the inter-frame difference between the frame-level fusion feature vectors, the mutual information loss term, and the key point matching error term. Based on the preset maximum slope constraint parameter, a path search is performed on the multi-objective cost matrix to obtain at least one minimum cost path that satisfies continuity and monotonicity. Output a unified time-aligned sequence based on the minimum cost path.

2. The alignment method for heterogeneous modal sequences according to claim 1, characterized in that, The mutual information function calculated based on the joint probability density of the first modality sequence and the word boundary probability vector sequence includes: The first modality sequence and the word boundary probability vector sequence are subjected to time window sliding sampling with a fixed step size, and local feature vector pairs within the corresponding time segments are extracted. The KDE method is used to construct the joint probability density function of each pair of local feature vectors, where the kernel function is a Gaussian kernel and the bandwidth parameter is adaptively adjusted according to the sampled local entropy value. Estimate the marginal probability density function and the joint probability density function separately; The mutual information function is calculated based on a pre-defined information entropy formula.

3. The alignment method for heterogeneous modal sequences according to claim 1, characterized in that, The method for calculating the distance between the first modal sequence and the second modal sequence based on weighted distance includes: Select a subset of key points from the normalized set of lip key points in each frame to construct a lip movement feature vector; The acoustic feature vectors of each frame in the first modal sequence are transformed into a feature space with the same dimension as the lip movement feature vectors through linear mapping, and a comparison pair is obtained. The distance function between the two frames in the comparison pair is calculated using a preset distance function.

4. The alignment method for heterogeneous modal sequences according to claim 1, characterized in that, The construction of a multi-objective cost matrix based on the frame-level fused feature vectors, wherein each element of the multi-objective cost matrix includes the inter-frame difference between the frame-level fused feature vectors, a mutual information loss term, and a keypoint matching error term, including: For any two frame-level fused feature vectors, calculate the inter-frame difference, which is the square of the Euclidean distance between the two frame-level fused feature vectors, and is used to measure the degree of feature change between the current time and the adjacent time. The negative logarithm of the calculated frame-level mutual information value is then taken after adding a preset minimum constant and constructing the mutual information loss term. The keypoint matching error term is calculated based on the weighted squared difference between the visual keypoint coordinates estimated by the acoustic feature flow and the actual visual keypoint coordinates, where the weighting coefficients are set according to the importance of each keypoint. The inter-frame difference, mutual information loss term, and key point matching error term are linearly combined according to preset weighting coefficients to obtain the values ​​of the corresponding elements in the multi-objective cost matrix. The weighting coefficients are used to control the proportion of each loss term in the overall cost, and the sum of the three weighting coefficients is one. Using the maximum value in each item as a reference, the values ​​of all elements in the multi-objective cost matrix are mapped to a preset range.

5. The alignment method for heterogeneous modal sequences according to claim 1, characterized in that, The method involves performing a path search on the multi-objective cost matrix based on a preset maximum slope constraint parameter, and obtaining at least one minimum cost path that satisfies both continuity and monotonicity, including: Predetermine the maximum slope constraint parameter in the path search to limit the maximum step size of the path in the time and modal dimensions; Based on the maximum slope constraint parameter, a dynamic path search is performed on the multi-objective cost matrix under monotonicity and continuity constraints to obtain the path sequence that minimizes the cumulative cost. The path sequence consists of multiple nodes, and each node corresponds to a multimodal frame combination. The step of outputting a unified time-aligned sequence based on the minimum cost path includes: Backtrack from the starting boundary to the ending boundary of the multi-objective cost matrix and extract the modal time index combination corresponding to the minimum cost path; Based on the combination of time indices of each modality contained in the minimum cost path, a unified reference time axis is established, the original data sequences of each heterogeneous modality are synchronously mapped onto the unified reference time axis, and a unified time-aligned sequence is output.

6. The alignment method for heterogeneous modal sequences according to claim 1, characterized in that, The step of extracting the lip motion trajectory from the input visual spatial keypoint matrix (VSKM) and constructing a second modality sequence based on the target keypoint set includes: Receive a facial key point sequence matrix containing multiple frames of video images, wherein the facial key point sequence matrix is ​​the two-dimensional spatial coordinate information of each key point in the facial region identified from consecutive frame images by a visual acquisition module; Based on a predefined set of lip keypoint indexes, the motion trajectory of the lip region corresponding to the target frame is extracted from the face keypoint sequence matrix and constructed as a sequence of lip region motion trajectories. The set of lip keypoint indexes includes the coordinate indexes of key anatomical feature points, including at least the upper lip, lower lip, and corners of the mouth. The motion trajectory sequence of the lip region is converted into a second modal sequence.

7. The alignment method for heterogeneous modal sequences according to claim 1, characterized in that, The method further includes: The frame-level fusion feature vectors of multiple consecutive time frames in the frame-level fusion feature vector are subjected to first-order and second-order difference calculations to obtain the local change rate and local change curvature of the frame-level fusion feature vector. The local curvature is compared with a preset curvature threshold, and the target time segment with drastic local changes is identified based on the comparison result. Based on the target time segment, adjust the matching penalty weights of the corresponding target time segments in the multi-objective cost matrix; The adjusted multi-objective cost matrix is ​​then smoothed.

8. The alignment method for heterogeneous modal sequences according to claim 1, characterized in that, The method further includes: For each frame-level fused feature vector containing a single modal sequence, calculate the modal sequence confidence between the current frame and the frames before and after it. When the modality sequence confidence of the target single modality sequence is lower than the preset confidence threshold, the contribution of the target single modality sequence to the frame-level fusion feature vector is reduced by the scaling factor.

9. An alignment device for heterogeneous modal sequences, characterized in that, The device includes: The first modal sequence generation unit is used to extract acoustic feature vectors and fundamental frequency features of each frame from the input acoustic feature stream (AFS) based on the window function and overlap rate, and to form the first modal sequence. The second modality sequence generation unit is used to perform word-level probabilistic encoding on the input language modality sequence LBPV to generate a word boundary probability vector sequence. The third modality sequence generation unit is used to extract the lip motion trajectory from the input visual spatial keypoint matrix (VSKM) and construct the second modality sequence based on the target keypoint set. The first computing unit is used to calculate the mutual information function between the first modality sequence and the word boundary probability vector sequence based on the joint probability density; The second calculation unit is used to calculate the distance function between the first modality sequence and the second modality sequence based on the weighted distance; The third calculation unit is used to dynamically calculate the modality weight coefficients based on the mutual information function and the distance function, and to construct a frame-level fusion feature vector based on the modality weight coefficients; A matrix construction unit is used to construct a multi-objective cost matrix based on the frame-level fusion feature vectors. Each element of the multi-objective cost matrix includes the inter-frame difference between the frame-level fusion feature vectors, a mutual information loss term, and a key point matching error term. The path search unit is used to perform path search on the multi-objective cost matrix based on a preset maximum slope constraint parameter, and obtain at least one minimum cost path that satisfies continuity and monotonicity. The output unit is used to output a uniform time-aligned sequence based on the minimum cost path.

10. An alignment system for heterogeneous modal sequences, characterized in that, The system includes: Processor, memory, input / output units, and bus; The processor is connected to the memory, the input / output unit, and the bus; The memory stores a program, which the processor invokes to perform the method as described in any one of claims 1 to 8.

11. A computer-readable storage medium, characterized in that, The computer-readable storage medium contains a program that, when executed on a computer, performs the method as described in any one of claims 1 to 8.

Images