An alignment method, device, system and storage medium for heterogeneous modal sequences

Abstract

The application discloses a method, device and system for alignment of heterogeneous modal sequences and a storage medium. The method comprises: extracting frame-level acoustic vectors and fundamental frequencies of an acoustic feature stream to generate a first modal sequence; performing word-level probability coding on a language modal to obtain a word boundary probability vector; extracting a lip movement trajectory from a visual key point matrix to construct a second modal sequence. Further, a mutual information function of the acoustic and language modals is calculated based on a joint probability density, a distance function of the acoustic and visual modals is calculated based on a weighted distance, and a modal weight is dynamically determined based on the distance function to generate a frame-level fusion feature vector. A multi-objective cost matrix containing an inter-frame difference, mutual information loss and key point matching error is constructed by the fusion feature, path searching is performed based on a maximum slope constraint to obtain a minimum cost path, a unified time alignment sequence is output, and accurate timing synchronization of multi-modal information is achieved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of big data, and particularly relates to a method, device and system for aligning heterogeneous modal sequences and a storage medium. BACKGROUND

[0002] In the field of multi-modal audio-video processing, the time alignment between heterogeneous modalities such as speech, image and text is a key technical link to realize content synchronization generation and multi-modal interaction. For example, in virtual digital human driving, speech subtitle synchronization, intelligent customer service question and answer generation, short video explanation, emotional broadcasting and other applications, the system usually needs to match the acoustic signal with the language and text information, the modal data such as the lip movement or facial expression track in the video for time consistency, so as to ensure the coordination and fluency of speech, subtitles and picture content at the perception level.

[0003] In the prior art, in order to reduce the implementation complexity, most schemes adopt a phased and pair-by-pair alignment strategy, that is, the speech signal and the text are matched first, and then the speech and the video frame are registered, so as to indirectly establish the time mapping relationship between the multi-modalities. Although this kind of scheme is relatively simple in structure, it has obvious limitations in actual application. Due to the essential differences in the acquisition, processing and expression of each modality, the step-by-step alignment strategy is prone to amplify the alignment error layer by layer, especially in unstable scenes such as fast pronunciation speed change, sudden change of spoken language expression, strong dialect interference or occlusion of mouth shape in the video, the generated subtitles, speech and pictures often have obvious time dislocation, which affects the synchronization experience of the final content.

[0004] In addition, some existing methods lack sufficient modeling capability when dealing with nonlinear changes between different modalities or dynamic modality weight adjustment, and cannot judge the stability and reliability of the current modality in real time during the alignment process, so they still rely heavily on the specific modality when the interference of the specific modality is intensified, further aggravating the alignment deviation.

[0005] In summary, the existing multi-modal alignment method still has obvious room for improvement in precision robustness, dynamic adaptability and perceptual consistency, and an alignment optimization method that can fuse multi-modal features, has high time sequence analysis capability and adapts to speech speed and modality changes is urgently needed. SUMMARY

[0006] To solve the above technical problems, the present application provides a method, device and system for aligning heterogeneous modal sequences and a storage medium.

[0007] The technical solutions provided in the present application are described as follows:

[0008] The first aspect of the present application provides a method for aligning heterogeneous modal sequences, comprising:

[0009] extracting acoustic feature vectors and fundamental frequency features comprising each frame based on a window function and an overlap rate, to form a first modality sequence, from an input acoustic feature stream AFS;

[0010] performing word-level probability encoding on the input language modality sequence LBPV to generate a word boundary probability vector sequence;

[0011] extracting a lip motion trajectory in the input visual spatial key point matrix VSKM, and constructing a second modality sequence based on a target key point set;

[0012] calculating a mutual information function of the first modality sequence and the word boundary probability vector sequence based on a joint probability density;

[0013] calculating a distance function of the first modality sequence and the second modality sequence based on a weighted distance;

[0014] dynamically calculating a modality weight coefficient based on the mutual information function and the distance function, and constructing a frame-level fusion feature vector according to the modality weight coefficient;

[0015] constructing a multi-objective cost matrix based on the frame-level fusion feature vector, each element of the multi-objective cost matrix comprising an inter-frame difference between the frame-level fusion feature vectors, a mutual information loss term, and a key point matching error term;

[0016] performing path search on the multi-objective cost matrix based on a preset maximum slope constraint parameter, and obtaining at least one minimum cost path satisfying continuity and monotonicity;

[0017] outputting a unified time alignment sequence according to the minimum cost path.

[0018] Optionally, the mutual information function of the first modality sequence and the word boundary probability vector sequence based on the joint probability density comprises:

[0019] performing time window sliding sampling on the first modality sequence and the word boundary probability vector sequence at a fixed step length, respectively, and extracting a local feature vector pair in a corresponding time segment;

[0020] constructing a joint probability density function of each group of the local feature vector pair using a KDE method, wherein the kernel function is a Gaussian kernel, and the bandwidth parameter is adaptively adjusted according to the sampled local entropy value;

[0021] respectively estimating an edge probability density function and a joint probability density function;

[0022] calculating the mutual information function based on a preset information entropy formula.

[0023] Optionally, the distance function between the first modality sequence and the second modality sequence is calculated based on a weighted distance, including:

[0024] A preset key point subset is selected from each normalized frame of the lip key point set, and a lip movement feature vector is constructed.

[0025] The acoustic feature vector of each frame in the first modality sequence is transformed into a feature space with the same dimension as the lip movement feature vector through linear mapping, and a comparison pair is obtained.

[0026] The distance function between the two frames in the comparison pair is calculated through a preset distance function.

[0027] Optionally, a multi-objective cost matrix is constructed based on the frame-level fusion feature vector, and each element of the multi-objective cost matrix includes an inter-frame difference value, an mutual information loss term, and a key point matching error term between the frame-level fusion feature vectors, including:

[0028] For any two frame-level fusion feature vectors, the inter-frame difference value is calculated, which is the square of the Euclidean distance between the two frame-level fusion feature vectors, used to measure the degree of change of the features between the current time and the adjacent time.

[0029] The mutual information value calculated is added to a preset minimum constant, and the negative value of the logarithm is taken to construct the mutual information loss term.

[0030] Based on the weighted squared difference between the visual key point coordinates estimated based on the acoustic feature flow and the real visual key point coordinates, the key point matching error term is calculated, wherein the weighting coefficient is set according to the importance of each key point.

[0031] The inter-frame difference value, the mutual information loss term, and the key point matching error term are linearly combined according to the preset weighting coefficients to obtain the value of the corresponding element in the multi-objective cost matrix, and the weighting coefficients are used to control the proportion of each loss term in the overall cost, and the sum of the three weighting coefficients is one.

[0032] Taking the maximum value in each term as a reference, the numerical values of all elements in the multi-objective cost matrix are mapped to a preset interval.

[0033] Optionally, based on a preset maximum slope constraint parameter, path search is performed on the multi-objective cost matrix, and at least one minimum cost path satisfying continuity and monotonicity is obtained, including:

[0034] The maximum slope constraint parameter in the path search is determined in advance to limit the maximum movement step of the path in the time dimension and the modality dimension.

[0035] perform a dynamic path search under monotonicity and continuity constraints on the multi-objective cost matrix based on the maximum slope constraint parameter, to obtain a path sequence with minimum cumulative cost, the path sequence being composed of multiple nodes, each node corresponding to a multi-modal frame combination;

[0036] outputting a unified time-aligned sequence according to the minimum cost path, includes:

[0037] backtracking from a start boundary to a termination boundary of the multi-objective cost matrix, and extracting a modal time index combination corresponding to the minimum cost path;

[0038] establishing a unified reference time axis according to each modal time index combination contained in the minimum cost path, synchronously mapping original data sequences of each heterogeneous modality to the unified reference time axis, and outputting a unified time-aligned sequence.

[0039] Optionally, the method further includes:

[0040] receiving a face key point sequence matrix containing multiple frames of video images, the matrix being two-dimensional spatial coordinate information of each key point in a face region identified from continuous frame images based on a visual acquisition module;

[0041] extracting a lip region motion trajectory corresponding to a target frame from the face key point sequence matrix based on a predefined lip key point index set, and constructing a lip region motion trajectory sequence, the lip key point index set containing coordinate indexes of key anatomical feature points including at least upper lip, lower lip, and corner of mouth;

[0042] converting the lip region motion trajectory sequence into a second modality sequence.

[0043] Optionally, the method further includes:

[0044] performing first-order and second-order difference calculations on frame-level fusion feature vectors of consecutive multiple time frames in the frame-level fusion feature vector to obtain local change rates and local change curvatures of the frame-level fusion feature vectors;

[0045] comparing the local change curvatures with a preset curvature threshold, and identifying a target time segment with a local change that is sharp according to a comparison result;

[0046] adjusting a matching penalty weight of a corresponding target time segment in the multi-objective cost matrix based on the target time segment;

[0047] performing smoothing processing on the multi-objective cost matrix after the adjustment.

[0048] Optionally, the method further comprises:

[0049] For each single-modality sequence in the frame-level fusion feature vector, a modality sequence confidence between the current frame and the previous and next frames is calculated.

[0050] When the modality sequence confidence of a target single-modality sequence is lower than a preset confidence threshold, the contribution of the target single-modality sequence to the frame-level fusion feature vector is reduced by a scaling coefficient.

[0051] The second aspect of the present application provides an alignment device for heterogeneous modality sequences, the device comprising:

[0052] A first modality sequence generation unit is configured to extract acoustic feature vectors and fundamental frequency features comprising frames based on a window function and an overlap rate from an input acoustic feature stream AFS, and to form a first modality sequence.

[0053] A second modality sequence generation unit is configured to perform word-level probability encoding on an input language modality sequence LBPV to generate a word boundary probability vector sequence.

[0054] A third modality sequence generation unit is configured to extract lip motion trajectories from an input visual spatial key point matrix VSKM, and to construct a second modality sequence based on a target key point set.

[0055] A first calculation unit is configured to calculate a mutual information function of the first modality sequence and the word boundary probability vector sequence based on a joint probability density.

[0056] A second calculation unit is configured to calculate a distance function of the first modality sequence and the second modality sequence based on a weighted distance.

[0057] A third calculation unit is configured to dynamically calculate a modality weight coefficient based on the mutual information function and the distance function, and to construct a frame-level fusion feature vector according to the modality weight coefficient.

[0058] A matrix construction unit is configured to construct a multi-objective cost matrix based on the frame-level fusion feature vector, each element of the multi-objective cost matrix comprising an inter-frame difference between the frame-level fusion feature vectors, a mutual information loss term, and a key point matching error term.

[0059] A path search unit is configured to perform path search on the multi-objective cost matrix based on a preset maximum slope constraint parameter, and to obtain at least one minimum cost path satisfying continuity and monotonicity.

[0060] An output unit is configured to output a unified time alignment sequence according to the minimum cost path.

[0061] The third aspect of the present application provides an alignment system for heterogeneous modality sequences, the system comprising:

[0062] a processor, a memory, an input / output unit and a bus;

[0063] the processor is connected with the memory, the input / output unit and the bus;

[0064] the memory stores a program, and the processor invokes the program to execute the method of the first aspect and any optional one of the first aspect.

[0065] The fourth aspect of the present application provides a computer readable storage medium, which stores a program, and the program executes the method of the first aspect and any optional one of the first aspect when executed on a computer.

[0066] From the above technical solutions, the present application has the following beneficial effects:

[0067] The present application breaks through the problem of error accumulation caused by "alignment of each pair of modalities" in the prior art by constructing a frame-level fusion feature vector and introducing the coupling relationship among the three types of modalities of speech, text and vision, and performing time series modeling and alignment in a unified feature space. Compared with the traditional dual-modality phased processing method, the present application realizes global joint optimization of multi-modality time sequence synchronization by constructing a fusion cost matrix at the frame level and uniformly performing optimal path search, so that the output subtitle text, speech broadcast and lip movement have higher consistency and synchronization accuracy, and the interactive naturalness and visual quality of the multi-modality content generation system are significantly improved. BRIEF DESCRIPTION OF DRAWINGS

[0068] In order to more clearly illustrate the technical solutions in the present application, the drawings needed in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0069] Figure 1 An embodiment flowchart of the alignment method for heterogeneous modal sequence provided in the present application;

[0070] Figure 2 An embodiment flowchart of step S103 in the alignment method for heterogeneous modal sequence provided in the present application;

[0071] Figure 3 An embodiment flowchart of step S104 in the alignment method for heterogeneous modal sequence provided in the present application;

[0072] Figure 4An embodiment flowchart of step S107 in the alignment method for heterogeneous modality sequences provided in the present application;

[0073] Figure 5 An embodiment flowchart of step S108 in the alignment method for heterogeneous modality sequences provided in the present application;

[0074] Figure 6 An embodiment structure diagram of the alignment device for heterogeneous modality sequences provided in the present application;

[0075] Figure 7 An embodiment structure diagram of the alignment system for heterogeneous modality sequences provided in the present application. DETAILED DESCRIPTION

[0076] Referring to Figure 1 The present application first provides an embodiment of an alignment method for heterogeneous modality sequences, which comprises:

[0077] S101, based on a window function and an overlap rate, extracting acoustic feature vectors and pitch features containing each frame from an input acoustic feature stream AFS to form a first modality sequence;

[0078] In the present embodiment, the "heterogeneous modality sequence" includes but is not limited to an acoustic modality sequence (e.g., MFCC features, pitch features, etc.), a language modality sequence (e.g., word boundary encoding vectors corresponding to text content), and a second modality sequence (e.g., lip movement key point trajectories, etc.). The method is applicable to scenarios such as multi-modal speech broadcasting, virtual digital human driving, and subtitle alignment, and can achieve high-precision alignment of three modalities in the time dimension.

[0079] In this step, the input acoustic feature stream (Acoustic Feature Stream, AFS) is subjected to sliding window sampling processing, and the sampling window size can be set to 25 ms and the frame shift to 10 ms. Acoustic features are extracted for each time window, including but not limited to 13-dimensional Mel frequency cepstral coefficients (MFCC), their first and second derivatives, which are combined into a 39-dimensional feature vector; further, pitch and pitch contour are extracted to form a one-dimensional or two-dimensional pitch feature sequence. Finally, a first modality sequence composed of multiple frames and a corresponding pitch sequence are obtained.

[0080] In the present embodiment, it is assumed that the input acoustic feature stream AFS is a 10-second speech signal with a sampling rate of 16 kHz. In order to extract frame-level features, a conventional speech feature extraction strategy is used for preprocessing.

[0081] Frame division is performed on the original speech signal, with a window length of 25 milliseconds (400 sampling points) and a frame shift of 10 milliseconds (160 sampling points), and about 1000 time frames are obtained in 10 seconds of audio.

[0082] The following feature extraction operation is performed on each frame: using the Mel frequency cepstral coefficient (MFCC) method, a 13-dimensional MFCC vector is extracted; further, the first-order difference (Δ) and the second-order difference (ΔΔ) are calculated, totaling 26 dimensions; and the static state is merged to obtain a 39-dimensional MFCC feature vector.

[0083] The pitch of the frame, its log-F0, and the pitch contour (if any) are extracted in parallel to form a pitch feature (e.g., 2-4 dimensions); finally, each frame forms an acoustic feature vector with a length of about 41-43 dimensions.

[0084] Next, the first modal sequence and the pitch sequence are constructed, and the MFCC features of each frame are combined in time sequence to form a first modal feature sequence with a length of about 1000; correspondingly, the pitch features of each frame are combined to form a pitch sequence; both share the time axis as the acoustic modal input in the method of the present application.

[0085] For example, at the 300th frame, the MFCC feature vector is [12.1, -4.3, 0.7,..., -1.2] (39 dimensions in total), and the corresponding pitch is F0=110 Hz, so the complete acoustic description of this frame is:

[0086] Frame_300={

[0087] 'mfcc':[12.1, -4.3, 0.7,..., -1.2],

[0088] 'pitch':110.0

[0089] }

[0090] S102, word-level probability encoding is performed on the input language modal sequence LBPV to generate a word boundary probability vector sequence.

[0091] The input language modal sequence LBPV is processed at the word level, and the BIO (Begin-Inside-Outside) labeling method is performed on each word boundary to generate a boundary probability vector that is time-aligned. Each time corresponds to a real-valued probability vector that describes the probability of being at the beginning of a word, in the middle of a word, or not in a word at the current time point, serving as a structured representation of the language modal and forming a language modal vector sequence with the same length as the acoustic modal.

[0092] In this embodiment, it is assumed that the language modal input corresponding to the 10-second speech described in S101 is a text sequence, and the transcription result thereof is:

[0093] "Welcome to use the multi-modal alignment system."

[0094] To realize the structured processing of the text, the input language modal sequence (Language Boundary Probability Vector, LBPV) is processed by word-level probability coding in this step, mainly through the following specific ways.

[0095] First, the original text is processed by word segmentation to obtain a word sequence, for example:

[0096] ["Welcome", "use", "multi-modal", "alignment", "system", "."]

[0097] It is assumed that there are 1000 frames corresponding to the audio, and the start and end time of each word is estimated based on the preliminary alignment result of the speech recognition system (ASR) or the acoustic model prediction result.

[0098] For example, "Welcome" corresponds to frames [0-80], "use" corresponds to frames [81-160], "multi-modal" corresponds to frames [161-350], and so on.

[0099] At the position of each frame, it is marked whether the frame is the beginning (B), middle (I), or does not belong to any word (O).

[0100] Introduce a word boundary probability estimator (BiLSTM-CRF, Transformer, or attention-based model can be used), which outputs a 3-dimensional probability vector for each frame:

[0101] P_frame_t=[P_B(t), P_I(t), P_O(t)]

[0102] For example:

[0103] Frame 162 (belongs to the first word of "multi-modal") -> [0.89, 0.08, 0.03]

[0104] Frame 163 (belongs to the middle word of "multi-modal") -> [0.03, 0.95, 0.02]

[0105] The above frame-level probability vectors are concatenated in time order to form a language modal sequence with the same length as the acoustic modal:

[0106] LBPV=[

[0107] [0.82, 0.13, 0.05], #Frame0

[0108] [0.91, 0.06, 0.03], #Frame1 ...

[0109] [0.03, 0.95, 0.02], #Frame163 ... ]

[0111] Each frame is represented by a three-dimensional probability vector to characterize its semantic boundary features in the word structure, which can be directly used for acoustic modality mutual information estimation.

[0112] Finally, the structure mapping of text to frame-level probability is completed, which introduces discrete language units into the continuous time domain, making them comparable with the acoustic modality, and constitutes the basis for subsequent modality alignment.

[0113] S103, extracting lip motion trajectory in the input visual space key point matrix VSKM, and constructing a second modality sequence based on the target key point set;

[0114] Based on the pre-acquired video key point matrix (VSKM), the key point coordinates of the facial regions including the lips, the lower jaw, etc. are extracted, and the 68-point face grid or the three-dimensional mouth key point subset is preferably used. By connecting the key point trajectories and calculating their inter-frame Euclidean change rate, the lip motion trajectory is obtained. The trajectory is reorganized by frame to form a second modality sequence that is time-aligned with the acoustic sequence, which is used to represent the state of the evolution of the lip shape change over time.

[0115] In this embodiment, the step S103 is used to extract the visual features of the face region captured in the continuous frame video into a structured second modality sequence, referring to Figure 2 , the specific implementation includes the following sub-steps:

[0116] S1031, receiving a face key point sequence matrix containing multiple frames of video images, the matrix being two-dimensional spatial coordinate information of each key point in the face region identified from continuous frame images based on a visual acquisition module;

[0117] An input video stream is received, and a face detection and key point positioning operation is performed on each frame based on a visual acquisition module (such as Dlib, OpenFace, MediaPipe, or a self-developed key point detection model).

[0118] Two-dimensional spatial coordinates containing K facial key points are identified for each frame of image to form a frame-level key point matrix:

[0119] ;

[0120] The V f represents a set of face key points extracted from the fth frame of image in the video sequence.

[0121] f: video frame index, indicating the fth frame image currently processed;

[0122] K: total number of face key points detected in a single frame image;

[0123] (x i , y i ): two-dimensional spatial coordinates of the ith face key point in the image coordinate system; i = 1, 2, …, K, K: key point index number.

[0124] S1032, based on a predefined set of lip key point indexes, extracting the lip region motion trajectory corresponding to the target frame from the face key point sequence matrix, and constructing it as a lip region motion trajectory sequence, the set of lip key point indexes contains the coordinate indexes of key anatomical feature points such as upper lip, lower lip, and corner of mouth;

[0125] For a video image sequence containing T frames, a key point time sequence matrix is constructed:

[0126] ;

[0127] The VSKM represents a face key point time sequence matrix composed of multiple consecutive video images.

[0128] Wherein:

[0129] T represents the total number of frames contained in the video segment;

[0130] V1, V2, …, V T respectively represent the frame-level key point matrix extracted from the 1st to the Tth frame image.

[0131] According to the preset set of lip key point indexes (such as the points with index range [61-88] in MediaPipe), the coordinate points of the upper lip, lower lip, and corner of mouth region are extracted from each frame to form a lip region subset.

[0132] The lip key point set is extracted from each frame:

[0133] ;

[0134] Wherein is the set of lip region key point indexes, (x i , y i ): two-dimensional spatial coordinates of the ith key point belonging to the lip region; i ∈ I lip : indicates that the key point index belongs to the set of lip key point indexes.

[0135] L f represents the set of lip key points.

[0136] Concatenate the T-frame lip keypoint sequence into a lip motion trajectory sequence:

[0137]

[0138] In the formula, L represents the lip motion trajectory sequence, L1, L2, …, LT represent the lip keypoint sets extracted from the 1st frame to the Tth frame. T

[0139] In an optional embodiment, PCA dimensionality reduction can also be performed on the lip point sequence or a relative motion vector can be constructed based on the mesh to enhance the description capability of the lip contour changes.

[0140] S1033, convert the lip region motion trajectory sequence into a second modality sequence.

[0141] Organize the lip motion trajectory sequence by frame into a time-ordered visual feature sequence. The sequence can be constructed in the following two ways:

[0142] A. Directly use two-dimensional coordinate concatenation vectors to form frame-level visual features;

[0143] B. Extract the keypoint movement increment vector between adjacent frames to express the lip dynamic features.

[0144] The final second modality sequence VMS is consistent with the number of audio frames, which is suitable for subsequent cross-modality alignment calculation.

[0145] S104, calculate the mutual information function of the first modality sequence and the word boundary probability vector sequence based on the joint probability density;

[0146] Use joint probability estimation method to respectively count the joint distribution between acoustic feature frames and language boundary vector frames. The mutual information function (Mutual Information, MI) is used to measure the information sharing degree between the two modalities. The mutual information value can be estimated based on experience, kernel density estimation or based on a deep estimation model to reflect the time coupling strength between the acoustic signal and the language boundary.

[0147] This step S104 aims to calculate the mutual information function between the acoustic modality sequence and the language modality sequence to measure the correlation between the two in time sequence, thereby providing a structural basis for modality weight calculation and fusion alignment path. Referring to Figure 3 , the process can be implemented through the following sub-steps:

[0148] S1041, respectively perform time window sliding sampling on the first modality sequence and the word boundary probability vector sequence with a fixed step length, and extract the local feature vector pair in the corresponding time segment.​​

[0149] Perform fixed step sliding window operation on the first modal sequence (acoustic feature stream) and the language modal sequence (word boundary probability vector sequence) respectively, and extract the corresponding local feature vector pair in the frame-level time segment.

[0150] Let the sliding window length be w (such as 15 frames) and the step size be s (such as 5 frames). For each time window:

[0151] Extract the local feature vector within the window from the acoustic modal:

[0152] ;

[0153] Wherein, A wi represents the local feature vector extracted from the first modal sequence in the current time window;

[0154] a t , a t+1 , …, a t+w−1 respectively represent the acoustic feature vectors corresponding to the t-th frame to the t+w-1-th frame;

[0155] Each ai is a high-dimensional vector, which can include MFCC, fundamental frequency, energy, and other acoustic description information.

[0156] Extract the language boundary probability vector of the corresponding window from the language modal:

[0157] ;

[0158] T win represents the language boundary probability vector extracted from the language modal sequence in the current time window;

[0159] t i represents the language feature vector corresponding to the i-th frame, which is used to describe the probability distribution of the time position being the beginning of a word, the middle of a word, or a non-word state.

[0160] Pair them to form a feature pair sample set:

[0161] .

[0162] X represents a feature pair sample set formed by pairing acoustic features and language features according to time index in the same time window;

[0163] (a i , t i ) represents a set of acoustic- language feature sample pairs observed simultaneously at the i-th frame;

[0164] t≤i<t+w: indicates that the sample index range is limited to the current sliding time window.

[0165] S1042, a joint probability density function of each group of the local feature vector pairs is constructed by using a KDE method, wherein a kernel function is a Gaussian kernel, and a bandwidth parameter is adaptively adjusted according to a sampled local entropy value;

[0166] The KDE (Kernel Density Estimation) method is used to model the feature pairs sampled above, and a local joint probability density function is constructed.

[0167] The kernel function used can be a Gaussian kernel function:

[0168] ;

[0169] Wherein:

[0170] K(x) represents a kernel function used for kernel density estimation;

[0171] In this embodiment, a Gaussian kernel function is selected, which has the form:

[0172] x represents a distance vector between a sample point and an estimation point in a feature space;

[0173] 2π is a normalization constant, used to ensure that the integral of the kernel function is 1;

[0174] exp(·) is an exponential function, used to realize smooth weight decay.

[0175] The kernel bandwidth parameter h can use a local entropy adaptive adjustment mechanism, that is, first calculate the entropy H(X) of the local window; set the minimum bandwidth h min , the maximum bandwidth h max , and the bandwidth is adaptively adjusted according to the following formula:

[0176] ;

[0177] Wherein, H(X) represents the Shannon information entropy of the sample set X in the current sliding window, used to measure the complexity of the local sample distribution; H max represents the maximum entropy value for normalization, which can be pre-set according to the feature dimension or historical statistical results.

[0178] Estimate the joint density function P(a, t):

[0179] ;

[0180] Wherein, P(a, t) represents the joint probability density function of the acoustic feature random variable a and the language feature random variable t; n represents the number of sample pairs in the current sliding window, generally equal to the length w of the sliding window.

[0181] (ai, ti) represents the i-th acoustic-linguistic feature sample pair; Kh(·) represents a kernel function with bandwidth h.

[0182] S1043, respectively estimate the marginal probability density function and the joint probability density function;

[0183] On the basis of the above, one-sided KDE is performed on the acoustic mode and the linguistic mode respectively to obtain the marginal probability distribution:

[0184] ;

[0185] .

[0186] In this embodiment, P(a) represents the marginal probability density function of the acoustic feature random variable a; P(t) represents the marginal probability density function of the linguistic feature random variable t.

[0187] The above marginal probability density is obtained by one-dimensional kernel density estimation or numerical integral approximation on the joint probability density; the calculation process is consistent with the joint probability density estimation in form, and only the distribution of a single mode is counted.

[0188] S1044, calculate the mutual information function based on the preset information entropy formula.

[0189] According to the definition formula of mutual information, the information entropy formula can be used for estimation:

[0190] ;

[0191] In this embodiment, MI(A, T) represents the value of the mutual information function of the acoustic mode sequence A and the linguistic mode sequence T in the current sliding window; the value of the mutual information function is a non-negative scalar, which is used to measure the correlation degree of the two modes in the statistical sense; when the two modes are highly synchronized, the value of the mutual information function increases; when they are independent or weakly correlated, the value of the mutual information function decreases.

[0192] In this embodiment, the mutual information is approximately solved by integrating all windows to obtain the mutual information value corresponding to each sliding window; the mutual information value can be extended to a frame-level mutual information function, and the mutual information scalar between each frame is obtained by smoothing interpolation.

[0193] For example, if an audio mutation (such as a popping sound) and a subtitle word boundary are observed to rise synchronously in a certain sliding window, the MI value in the window will be significantly higher than that in a semantic redundant or silent segment, reflecting the improvement of the time dependence between the modes.

[0194] In the embodiment, the mutual information value is first calculated in a sliding time window unit, each sliding time window corresponds to a mutual information scalar, which is used to represent the statistical correlation degree between the first modal sequence and the language modal sequence in the time window.

[0195] To extend the sliding window level mutual information value to the frame level mutual information function, preferably, the mutual information value of each sliding time window is assigned to the central frame or the preset reference frame covered by the time window, and the time index of the reference frame is taken as the corresponding frame position of the mutual information value.

[0196] For the time frames between adjacent sliding time windows that do not directly correspond to mutual information values, the mutual information scalar of the corresponding time frame is obtained by smooth interpolation calculation of the mutual information values corresponding to adjacent sliding windows, so as to construct a continuous frame level mutual information function sequence. The smooth interpolation method can be linear interpolation, weighted average interpolation or other interpolation methods that can maintain time continuity.

[0197] In the above manner, the mapping from the sliding window level mutual information value to the frame level mutual information function is realized, so that each time frame corresponds to a unique mutual information scalar, which is used for subsequent modal weight calculation and multi-target cost construction.

[0198] It should be noted that the information entropy calculation formula is used to quantify the statistical dependence between the joint probability distribution and the marginal probability distribution, and the calculation process takes the estimation result of the joint probability density function as a prerequisite, and obtains the mutual information value by performing algebraic operation in the form of information entropy on the joint probability density function and the marginal probability density function.

[0199] It should be understood that the "calculating mutual information function based on joint probability density" and "calculating mutual information function based on information entropy" described in the specification belong to different levels of description of the same mutual information calculation process, wherein the former describes the probability modeling basis of mutual information, and the latter describes the implementation method of numerical calculation based on the probability modeling basis, and the two do not constitute different or parallel mutual information calculation methods.

[0200] Therefore, the calculation method of the mutual information function in the embodiment is always based on the estimation result of the joint probability density, and the information entropy is only used as a mathematical expression form of mutual information calculation, and does not constitute an independent calculation path other than the joint probability density.

[0201] For the foregoing information entropy, in the embodiment, the calculation of the mutual information function is performed on the local feature vector pairs obtained by time window sliding sampling. For the local feature vector pairs in each sliding time window, first, the joint probability density function and the marginal probability density function in the corresponding time window are obtained according to the kernel density estimation result constructed in step S1042.

[0202] The integral operation in the mutual information function is used to statistically sum the probability density function within its definition domain, and the integral interval is implicitly determined by the value range of the local feature vector in the current sliding time window. In the specific implementation process, the integral operation is approximated by summing the numerical values of the probability density function at discrete sampling points, rather than using continuous analytical integration.

[0203] Therefore, the double integral expression shown in the specification is the theoretical representation of mutual information calculation, which corresponds to the discrete accumulation calculation of the joint probability density value and the edge probability density value of all sampling points in the current sliding time window in engineering implementation. Through the above method, a mutual information value can be obtained independently for each sliding time window, thereby forming a mutual information function sequence varying with time.

[0204] In this embodiment, the upper and lower limits of the integral interval are not explicitly given, which does not affect the calculation result of the mutual information function, and the essence is automatically limited by the support set of the sample distribution in the current sliding time window.

[0205] S105, calculating a distance function of the first modality sequence and the second modality sequence based on the weighted distance;

[0206] In this step, the system calculates the distance function between the extracted second modality sequence and the acoustic modality sequence to quantify the similarity of their time sequence features in space, thereby providing a calculation basis for subsequent modality weight estimation and alignment path construction.

[0207] The weighted distance calculation is performed on the feature difference between each pair of acoustic frame and visual frame, preferably using Euclidean distance or Mahalanobis distance measurement. To enhance dynamic adaptability, the fundamental frequency variation rate can be used as a weighting coefficient to weight and accumulate the position error between the lip key points. Finally, the matching distance function between the acoustic modality and the visual modality is generated, which is an important parameter for subsequent frame fusion.

[0208] Specifically, a preset key point subset can be selected from each normalized frame of lip key point set to construct a lip movement feature vector. The key point subset can include representative points of the upper lip center, lower lip center, and bilateral corners of the mouth, etc., to ensure that the extracted features can reflect the main mouth shape changes during speech production. The features can be obtained by two-dimensional coordinate splicing, Euclidean distance coding, or principal component compression, for example, reducing the mouth area features of each frame to a fixed dimension vector to form a frame-level lip movement sequence. The acoustic feature vector of each frame in the first modal sequence is transformed into a feature space with the same dimension as the lip movement feature vector through linear mapping, and a comparison pair is obtained; specifically, the acoustic feature vector of each frame in the first modal sequence is projected into a feature space consistent with the lip movement feature vector through a linear transformation function or a neural network mapping function obtained by training. The goal of this mapping operation is to make the acoustic features comparable to the visual features in terms of spatial structure, thereby supporting similarity calculation between frames. To enhance the expression ability of the mapping, the projection function can be implemented by a single-layer or multi-layer perceptron, and the parameters thereof are optimized by a small batch of paired data sets in the pre-training stage to ensure that a reliable alignment basis is formed between normal speech and visual mouth shapes. The distance between each pair of mapped acoustic feature vectors and lip movement feature vectors is calculated by a preset distance function. The distance function can be Euclidean distance, cosine similarity inverse, or Manhattan distance, and the specific choice depends on the characteristics of the sequences to be aligned. In this embodiment, the normalized weighted Euclidean distance is preferably used to consider the physical position weight of the key points and the change amplitude of the mapped acoustic features.

[0209] S106, dynamically calculating a modal weight coefficient based on the mutual information function and the distance function, and constructing a frame-level fusion feature vector according to the modal weight coefficient;

[0210] The purpose of this step is to dynamically fuse acoustic, language, and visual modalities at the frame level to construct a frame-level fusion feature vector that reflects the multi-modal semantic collaborative relationship. This fusion process is based on the aforementioned mutual information function and modal distance function, and the three types of modal features are weighted and combined by dynamically calculating a modal weight coefficient. Based on the aforementioned mutual information function and distance function, the modal weight coefficient is dynamically calculated according to the time frame. The modal weight coefficient is used to adjust the contribution degree of different modalities in the fusion feature. It is preferably adaptively adjusted according to the confidence (such as the speech activity detection result, the text prediction probability, and the video key point recognition completeness rate) and the local change rate of mutual information or distance. The acoustic, language, and visual features of each frame are weighted and summed according to the weight coefficient to obtain a frame-level fusion feature vector sequence.

[0211] In a specific implementation, the system first quantitatively evaluates the inter-modality correlation corresponding to each time frame by using the mutual information function (such as the frame-level mutual information sequence between acoustics and language) obtained in step S104 and the cross-modality distance function (such as the frame-level distance sequence between acoustics and vision) obtained in step S105. In order to further enhance the adaptive ability, the system can also introduce a modality confidence evaluation mechanism to model the signal integrity and reliability of each modality respectively.

[0212] Taking the t-th frame as an example, the system can estimate the confidence of the three modalities according to the following indicators:

[0213] For the acoustic modality (audio), according to the voice activity detection (VAD) result, if it is detected as a silent frame, a lower confidence is given;

[0214] For the language modality (text), the language model prediction probability (such as BERT softmax score) of the corresponding vocabulary of the frame can be used as the text confidence;

[0215] For the visual modality (image), the visual integrity score can be given according to whether complete lip key points are detected in the frame (such as a missing ratio <10%).

[0216] Combining the above confidence scores, the system normalizes the inter-modality mutual information value or distance change rate at each frame to estimate the modality weight coefficient of the frame. For example, if the audio signal of a certain frame is clear, the text prediction confidence is high, but the lip key points are missing seriously, then the audio and text modality weight coefficients of the frame can be made larger, and the visual modality weight coefficient is relatively weakened.

[0217] Suppose the feature vectors corresponding to the audio, language and visual modalities are a t , l t , v t , the modality weights are α t , β t , γ t , and the final frame-level fusion feature vector F t is calculated as follows:

[0218] ;

[0219] Where the weight coefficients satisfy the constraint:

[0220]

[0221] The following will be further illustrated by a specific example:

[0222] When the system processes an audio-visual segment of a child telling a story, the mutual information change rate of the audio increases and the confidence of VAD decreases in segments with sudden changes in speech rate or pauses. In this case, the system dynamically reduces the weight of the audio modality. Conversely, text prediction, due to its clear contextual structure, has higher confidence, so the system increases the contribution of the language modality. If the lip movements are clear and the keypoint integrity rate is high in the visual modality, then the visual modality is also given a higher weight. Ultimately, the fused feature vectors can more accurately capture the co-variations between pronunciation and semantics, which is beneficial for building a more robust alignment path. This dynamic modality fusion mechanism effectively avoids the problem of insufficient adaptability of fixed weighting strategies in multimodal temporal alignment, and improves the stability of multimodal alignment in scenarios with noise, occlusion, and speech rate changes.

[0223] S107. Construct a multi-objective cost matrix based on the frame-level fusion feature vectors. Each element of the multi-objective cost matrix includes the inter-frame difference between the frame-level fusion feature vectors, the mutual information loss term, and the key point matching error term.

[0224] Using the frame-level fusion feature vectors, a three-modal joint multi-objective cost matrix is ​​constructed. Each cost element consists of three parts: (1) the inter-frame difference of the fusion vector between the current frame and the next frame; (2) the mutual information loss term between corresponding frame pairs, i.e., the negative logarithm of the mutual information value; and (3) the visual keypoint error term, i.e., the weighted squared error based on the estimated keypoint and the actual keypoint. The three terms are multiplied by adjustable weight coefficients λ1, λ2, and λ3, respectively, and normalized and combined into the total cost.

[0225] In this step, the system further generates a multi-objective cost matrix for modality alignment path search based on the constructed frame-level fused feature vector sequence. Each element in the cost matrix comprehensively considers multiple loss terms to reflect the matching cost between heterogeneous modalities in the time dimension. See also... Figure 4 This process can be implemented in the following specific ways:

[0226] S1071. For any two frame-level fused feature vectors, calculate the inter-frame difference, where the inter-frame difference is the square of the Euclidean distance between the two frame-level fused feature vectors, used to measure the degree of feature change between the current time and the adjacent time.

[0227] First, for any two frame-level fused feature vectors F i With F j The square of the Euclidean distance is calculated and used as the inter-frame difference. This difference term is used to measure the magnitude of change in the fused features between adjacent time frames. The larger the value, the more significant the difference in modal information at the corresponding time point, indicating a sudden change or switching trend.

[0228] S1072. The negative logarithm of the calculated frame-level mutual information value is taken after adding a preset minimum constant and then constructing it as a mutual information loss term.

[0229] The frame-level mutual information value MI(a) obtained from the aforementioned mutual information calculation steps is used as an example. i , l j Add a very small constant (e.g., ϵ1=10) -6 The negative logarithm of the equation is then used to construct a mutual information loss term. This loss term reflects the degree of decrease in semantic relevance between acoustic and linguistic modalities, thereby suppressing low mutual information intervals during alignment.

[0230] S1073. Calculate the key point matching error term based on the weighted squared difference between the estimated visual key point coordinates and the actual visual key point coordinates, where the weighting coefficients are set according to the importance of each key point.

[0231] Furthermore, the system constructs a keypoint matching error term based on the difference between the visual keypoint coordinates (from the acoustic input) estimated in step S105 through linear mapping or a neural network and the actual visual modal keypoints. The error term is calculated by squaring the coordinate differences of all keypoints and then summing them using weighted averages. The weight w for each keypoint is... k The weights are pre-set based on their importance in lip movement expression. For example, the center point of the upper lip and the corner of the mouth may be given higher weights to ensure that points that are more sensitive to changes in pronunciation have higher weights in error assessment.

[0232] S1074. The inter-frame difference, mutual information loss term, and key point matching error term are linearly combined according to preset weighting coefficients to obtain the values ​​of the corresponding elements in the multi-objective cost matrix. The weighting coefficients are used to control the proportion of each loss term in the overall cost, and the sum of the three weighting coefficients is one.

[0233] Subsequently, the inter-frame difference, mutual information loss term, and keypoint matching error term are linearly combined according to preset weighting coefficients λ1, λ2, and λ3 to obtain the value of each element in the multi-objective cost matrix, i.e.:

[0234] ;

[0235] The weighting coefficients satisfy λ1+λ2+λ3=1, which controls the proportion of different loss terms in the final generation value. This combination method provides an adjustable fusion strategy, allowing for flexible setting of weight parameters for different scenarios.

[0236] Among them, C i , jrepresents the comprehensive matching cost value between the first modality frame with time index i and the second modality frame with time index j, used to construct a multi-modal alignment path, a timing matching matrix or a dynamic programming search space; the cost value is a non-negative real number, and the numerical value is used to measure the inconsistency degree of the two frames in the multi-modal feature level, and the smaller the numerical value is, the higher the matching degree is.

[0237] i represents the frame index in the first modality sequence (such as the acoustic modality or the visual lip movement modality), used to identify the i-th feature frame on the time axis;

[0238] j represents the frame index in the second modality sequence (such as the language modality or the text boundary modality), used to identify the candidate time position for alignment with the first modality.

[0239] λ1 represents the first feature loss term L feat the weight coefficient of (i, j), used to adjust the contribution proportion of the feature similarity in the total cost function;

[0240] λ2 represents the mutual information loss term L MI the weight coefficient of (i, j), used to control the influence degree of the modality statistical correlation on the alignment result;

[0241] λ3 represents the visual consistency loss term L vis the weight coefficient of (i, j), used to adjust the importance of the visual modality constraint in the comprehensive cost;

[0242] S1075, taking the maximum value in each term as a reference, the numerical values of all elements in the multi-objective cost matrix are mapped to a preset interval.

[0243] In order to enhance the numerical stability and contrastiveness of subsequent path search, the system takes the maximum value in each term as a normalization reference, and maps all numerical values in the cost matrix C i，j to the interval [0, 1]. This normalization processing can adopt the maximum scaling method, that is, each element is divided by the maximum item in the corresponding channel, so as to ensure that the losses in different dimensions have comparability in numerical value, and to suppress the interference of abnormal points on path calculation.

[0244] The following is a specific, perceptible and exemplary description of step S107:

[0245] Taking a "digital human news broadcast segment" as an example, the segment includes a 10-second voice explanation, which corresponds to 300 frames of acoustic features (each frame interval is 33 ms), 300 frames of lip movement visual key point sequence, and word boundary probability associated with text content.

[0246] After performing frame-level fusion, assuming the current time i=105 and j=106, we examine the alignment cost between frames 105 and 106. The specific calculation process is as follows:

[0247] Assuming the fused feature vectors for frames 105 and 106 are F105 and F106 respectively, with a vector dimension of 256, the system calculates the squared Euclidean distance between them as the inter-frame difference:

[0248] ;

[0249] This value reflects the overall degree of modal change between current frames.

[0250] In step S104, the acoustic feature a corresponding to frame 105 is estimated. 105 With the boundary probability vector t of the 106th word in the language modality 106 The mutual information between them is 0.72. After adding a minimal constant and taking the negative logarithm, we obtain the mutual information loss:

[0251] ;

[0252] Through acoustic features a 105 The lip keypoints predicted by the pre-trained mapping function are v^ 105 and with real visual key points v 106 Comparison is made. Taking 20 key lip points as an example, the weighted average difference calculation yields the following results:

[0253] ;

[0254] in, This represents the position vector or its scalar representation (e.g., two-dimensional coordinates or normalized displacement value) of the k-th lip keypoint in the real visual modality of frame 106.

[0255] This represents the estimated position of the k-th lip keypoint obtained based on acoustic feature prediction.

[0256] wk represents the weight coefficient corresponding to the k-th lip keypoint, which is used to reflect the importance of different keypoints in visual consistency assessment.

[0257] In this embodiment, L vis (105, 106) represents the visual consistency loss between the acoustic features of frame 105 and the visual keypoints of frame 106; the value 0.57 is a specific example result calculated in this embodiment, used to illustrate the calculation process, and its magnitude reflects the degree of matching of the frame pair at the visual synchronization level; the smaller the loss value, the higher the synchronization between the sound and the lip movement at that time position. This measure measures the degree of synchronization between the sound and the lip movement in that frame.

[0258] Set the weighting coefficients as λ1=0.4, λ2=0.3, λ3=0.3, and the three linear combinations:

[0259] ;

[0260] Suppose the maximum element in the current cost matrix is 1.45, perform maximum value normalization:

[0261] .

[0262] wherein, indicates the normalized multi-objective cost value, used to measure the comprehensive matching cost between the acoustic frame 105 and the visual frame 106;

[0263] S108, based on the preset maximum slope constraint parameter, performing path search on the multi-objective cost matrix, and obtaining at least one minimum cost path satisfying continuity and monotonicity;

[0264] On the basis of the constructed cost matrix, set the maximum slope constraint (for example, the maximum slope is 1.5) of the path search, and ensure that the generated path satisfies the monotonicity and continuity constraints. By dynamic programming or heuristic path optimization algorithm, the path with the minimum total cost is searched as the alignment path between the three modalities. The path represents the optimal time frame correspondence relationship between the modalities.

[0265] Specifically, a specific implementation of step S108 is provided below, please refer to Figure 5 :

[0266] S1081, predetermine the maximum slope constraint parameter in path search, used to limit the maximum moving step length of the path in the time dimension and the modality dimension;

[0267] S1082, based on the maximum slope constraint parameter, perform dynamic path search under the monotonicity and continuity constraints on the multi-objective cost matrix, obtain the path sequence with the minimum cumulative cost, and the path sequence is composed of multiple nodes, each node corresponds to a multi-modality frame combination;

[0268] Based on the multi-objective cost matrix constructed in step S107, a path search operation is performed on the matrix. Specifically, a path satisfying the time sequence monotonicity and continuity constraints is searched from the top-left corner (the starting boundary) to the bottom-right corner (the ending boundary) on the two-dimensional cost matrix. Each node in the path corresponds to a set of matching combinations between different modal frames; during the path construction process, the growth direction of the path is not allowed to back up, and the step length grown in a unit time satisfies the maximum slope constraint parameter (such as a maximum slope of 1.5), which is used to tolerate the speed variation or short silent segments; among all the feasible paths, the path with the minimum cumulative cost is selected as the optimal path, denoted as the minimum cost path P*.

[0269] The minimum cost path P* is composed of multiple nodes, each node being in the form of a triple (i, j, k), where i, j, and k represent the frame indexes of the acoustic modality, the language modality, and the visual modality, respectively.

[0270] The outputting of the unified time-aligned sequence according to the minimum cost path includes: backtracking from the starting boundary to the ending boundary of the multi-objective cost matrix, and extracting the modality time index combinations corresponding to the minimum cost path;

[0271] In this embodiment, the minimum cost path P* is backtracked from the ending boundary (the bottom-right corner) of the multi-objective cost matrix, and the index combinations of each modality in the path are extracted node by node;

[0272] Each extracted node contains a multi-modal matching pair, i.e., the correspondence among an acoustic frame, a corresponding language boundary frame, and a video frame index; all the nodes in the path are sequentially organized into a time-aligned index table, and the record form is as follows:

[0273] ;

[0274] Where t1, t2, …, tn represent the timestamps on the unified time axis; (a tn , l tn , v tn ) represent the frame-level indexes or frame-level features of the three modalities after alignment at the unified time point tn, where:

[0275] a tn represents the acoustic frame index or acoustic feature vector corresponding to the acoustic modality at the time point tn;

[0276] l tn represents the language unit index or word boundary probability feature corresponding to the language modality at the time point tn;

[0277] v tntn: represents the visual frame index or lip movement feature corresponding to the visual modality at time point tn;

[0278] t1, t2, …, tn: represent a sequence of discrete timestamps on a unified reference time axis.

[0279] According to the various modal time index combinations contained in the minimum cost path, a unified reference time axis is established, the original data sequences of each heterogeneous modality are synchronously mapped onto the unified reference time axis, and a unified time-aligned sequence is output.

[0280] A unified reference time axis is constructed, and the time period is divided into n synchronous time points based on the length n of the minimum cost path; the time indexes of the original sequences in the acoustic modality, the language modality, and the visual modality are respectively mapped according to the triple indexes in the minimum cost path, so that the three modalities have unique corresponding frames at each reference time point; the aligned three-modality data sequences are output on the unified time axis, and the specific output structure is:

[0281] ;

[0282] Each group of triple content is at the same time point tn, and is used to drive multi-modal tasks such as subtitle display, lip animation, speech synthesis, etc. in actual application.

[0283] AlignedSequence represents the unified output sequence obtained after multi-modal time alignment processing; the sequence maps the data of different modalities to the same time coordinate system based on the unified reference time axis.

[0284] tn: represents the aligned three-modality data unit at the nth unified time point tn, wherein:

[0285] tn: represents the aligned acoustic feature or speech segment of the acoustic modality at time point tn;

[0286] tn: represents the aligned language feature of the language modality at time point tn, such as word boundary probability or text label;

[0287] tn: represents the aligned visual feature of the visual modality at time point tn, such as lip key points or lip animation parameters.

[0288] S109, output a unified time-aligned sequence according to the minimum cost path.

[0289] According to the pairing relationship between each time frame in the minimum cost path, the acoustic, language and visual three modalities are mapped to a unified time axis, and a fused multi-modal alignment sequence is output. The sequence can be directly used for multi-modal tasks such as digital human driving, synchronous subtitle rendering, voice broadcasting, etc., and has advantages such as frame-level synchronization, fast response, and fault tolerance for abnormal modalities.

[0290] In an optional embodiment, in order to further improve the alignment robustness of the frame-level fusion feature in the scene with severe timing fluctuations, the system performs timing curvature-aware enhancement processing on the frame-level fusion feature vector. This method can also include the following steps:

[0291] First-order and second-order difference calculations are performed on the frame-level fusion feature vectors of the continuous multiple time frames in the frame-level fusion feature vector, to obtain the local change rate and local change curvature of the frame-level fusion feature vector.

[0292] The local change curvature is compared with a preset curvature threshold, and a target time segment with severe local change is identified according to the comparison result;

[0293] Based on the target time segment, the matching penalty weight of the corresponding target time segment in the multi-objective cost matrix is adjusted;

[0294] The adjusted multi-objective cost matrix is smoothed.

[0295] Specifically, the optional embodiment first obtains the fused frame-level feature vector sequence, and performs first-order difference and second-order difference operations on the sequence to obtain the local change rate (i.e., speed) and local change curvature (i.e., acceleration) of the sequence at each time frame. Preferably, the following form is used for calculation:

[0296] ;

[0297] Where t represents the time index, which identifies the position of the frame-level fusion feature vector in the time sequence; F t represents the frame-level fusion feature vector corresponding to the time index t, which is obtained by fusing the acoustic, language and visual modalities according to the weights; F t+1 represents the frame-level fusion feature vector at the next time index t+1.

[0298] Where, D t (1) represents the first-order difference value calculated at the time index t;

[0299] D t+1 (1) represents the first-order difference value calculated at the time index t+1;

[0300] D t (2) denotes the second-order difference result calculated at time index t, which is used to describe the degree of change in the fusion feature change rate, i.e., the local change curvature.

[0301] Then, the curvature amplitude at each time point is calculated and compared with a preset curvature threshold, so as to identify a set of time segments with local change being violent. The set can correspond to a situation of speech rate mutation, mouth shape jump, or text rearrangement, etc.

[0302] For the target time segments identified above, an enhanced penalty is imposed on the matching cost items in the corresponding multi-target cost matrix, i.e., the cost of non-optimal matching path is increased at these time instants, so as to enhance the alignment sensitivity of the system to these violent change areas.

[0303] To avoid introducing too large gradient fluctuation by the penalty factor, a sliding window can be used to smooth the adjusted cost matrix, so that the cost surface is more continuous and stable, and the stability of subsequent path search is improved.

[0304] In another optional embodiment, to improve the dynamic adaptability of the contribution of each modality in the fusion feature, a modality confidence adjustment mechanism is further introduced, and the specific steps are as follows:

[0305] For each single-modality sequence in the frame-level fusion feature vector, the modality sequence confidence between the current frame and the adjacent frames is calculated.

[0306] When the modality sequence confidence of the target single-modality sequence is lower than a preset confidence threshold, the contribution of the target single-modality sequence to the frame-level fusion feature vector is reduced through a scaling coefficient.

[0307] In this embodiment, for the frame-level fusion feature vector of each frame, three modality sub-vectors of the frame are respectively obtained: an acoustic modality sub-vector A t , a language modality sub-vector L t , and a visual modality sub-vector V t .

[0308] For each modality, the confidence value between the current frame and the adjacent frames is calculated. For example:

[0309] For the acoustic modality, whether the current frame is in a speech segment can be judged based on a voice activity detection (VAD) result;

[0310] For the language modality, the language boundary confidence of the current frame can be measured according to the mean or variance of the word boundary prediction probability value;

[0311] For the visual modality: the visual confidence can be calculated according to the completeness score of the face key points in the current frame, or whether the lip region is missing, etc.

[0312] When it is detected that the confidence of any one modality in the current frame is lower than a preset threshold (such as 0.3), the system will apply a scaling operation to the modality feature, reducing its influence on the fusion feature. For example, the original weights can be set as α=0.5, β=0.3, and γ=0.2. If the visual modality confidence is insufficient, γ is multiplied by a decay coefficient (such as 0.2), and the overall weight is re-normalized.

[0313] The embodiments of the device and system involved in the present application are described below:

[0314] Referring to Figure 6 The present application also provides an embodiment of an alignment device for a heterogeneous modality sequence, which comprises:

[0315] The first modality sequence generation unit 601 is configured to extract acoustic feature vectors and fundamental frequency features containing each frame based on a window function and an overlap rate from the input acoustic feature stream AFS, to form a first modality sequence.

[0316] The second modality sequence generation unit 602 is configured to perform word-level probability encoding on the input language modality sequence LBPV to generate a word boundary probability vector sequence.

[0317] The third modality sequence generation unit 603 is configured to extract lip motion trajectories in the input visual space key point matrix VSKM, and to construct a second modality sequence based on a target key point set.

[0318] The first calculation unit 604 is configured to calculate a mutual information function of the first modality sequence and the word boundary probability vector sequence based on a joint probability density.

[0319] The second calculation unit 605 is configured to calculate a distance function of the first modality sequence and the second modality sequence based on a weighted distance.

[0320] The third calculation unit 606 is configured to dynamically calculate a modality weight coefficient based on the mutual information function and the distance function, and to construct a frame-level fusion feature vector according to the modality weight coefficient.

[0321] The matrix construction unit 607 is configured to construct a multi-objective cost matrix based on the frame-level fusion feature vector, each element of the multi-objective cost matrix containing an inter-frame difference between the frame-level fusion feature vectors, a mutual information loss term, and a key point matching error term.

[0322] The path searching unit 608 is configured to perform path searching on the multi-target cost matrix based on a preset maximum slope constraint parameter, and obtain at least one minimum cost path satisfying continuity and monotonicity.

[0323] The output unit 609 is configured to output a unified time-aligned sequence according to the minimum cost path.

[0324] Optionally, the first calculation unit 604 is specifically configured to:

[0325] perform time window sliding sampling on the first modality sequence and the word boundary probability vector sequence respectively at a fixed step length, and extract a local feature vector pair in a corresponding time segment;

[0326] adopt a KDE method to construct a joint probability density function of each group of the local feature vector pair, wherein a kernel function is a Gaussian kernel, and a bandwidth parameter is adaptively adjusted according to a local entropy value of sampling;

[0327] respectively estimate an edge probability density function and the joint probability density function;

[0328] based on a preset information entropy, calculate a mutual information function.

[0329] Optionally, the second calculation unit 605 is specifically configured to:

[0330] select a preset key point subset from each frame of the normalized lip key point set, and construct a lip movement feature vector;

[0331] linearly map each frame of the acoustic feature vector in the first modality sequence to a feature space with the same dimension as the lip movement feature vector, and obtain a comparison pair;

[0332] calculate a distance function between two frames in the comparison pair through a preset distance function.

[0333] Optionally, the matrix construction unit 607 is specifically configured to:

[0334] for any two frame-level fusion feature vectors, calculate a frame difference value, the frame difference value being a square of a Euclidean distance between the two frame-level fusion feature vectors, and being used to measure a degree of change of features between a current time and an adjacent time;

[0335] by taking a negative value of a logarithm of the calculated frame-level mutual information value after adding a preset minimum constant, and constructing as a mutual information loss term;

[0336] based on a weighted square difference between the visual key point coordinates estimated based on the acoustic feature flow and the real visual key point coordinates, calculate a key point matching error term, wherein a weighting coefficient is set according to an importance of each key point;

[0337] linearly combine the inter-frame difference, mutual information loss term and key point matching error term according to preset weighting coefficients to obtain values of corresponding elements in the multi-objective cost matrix, the weighting coefficients are used to control proportions of the loss terms in the overall cost, and sums of the three weighting coefficients are one;

[0338] reference to maximum values in each term, map numerical values of all elements in the multi-objective cost matrix to a preset interval.

[0339] Optionally, the path searching unit 608 is specifically configured to:

[0340] predetermine a maximum slope constraint parameter in path searching, to limit maximum movement steps of the path in the time dimension and the modal dimension;

[0341] based on the maximum slope constraint parameter, perform dynamic path searching on the multi-objective cost matrix under monotonicity and continuity constraints to obtain a path sequence that minimizes the cumulative cost, the path sequence is composed of multiple nodes, and each node corresponds to a multi-modal frame combination;

[0342] Optionally, the output unit 609 is specifically configured to:

[0343] trace back from a starting boundary to a terminal boundary of the multi-objective cost matrix, and extract a modal time index combination corresponding to the minimum cost path;

[0344] According to each modal time index combination contained in the minimum cost path, a unified reference time axis is established, the original data sequences of each heterogeneous modal are synchronously mapped onto the unified reference time axis, and a unified time-aligned sequence is output.

[0345] Optionally, the third modal sequence generation unit 603 is specifically configured to:

[0346] receive a face key point sequence matrix containing multiple frames of video images, the matrix being two-dimensional spatial coordinate information of each key point in a face region identified from continuous frame images based on a visual acquisition module;

[0347] based on a predefined set of lip key point indexes, extract a lip region motion trajectory corresponding to a target frame from the face key point sequence matrix, and construct a lip region motion trajectory sequence, the set of lip key point indexes including coordinate indexes of key anatomical feature points such as upper lip, lower lip and corner of mouth;

[0348] convert the lip region motion trajectory sequence into a second modal sequence.

[0349] Optionally, further comprising: an adjusting unit 610, specifically configured to:

[0350] performing first-order and second-order difference calculation on the frame-level fusion feature vectors of the continuous multiple time frames in the frame-level fusion feature vectors to obtain local change rate and local change curvature of the frame-level fusion feature vectors;

[0351] comparing the local change curvature with a preset curvature threshold, and identifying a target time segment with a local change that is sharp according to a comparison result;

[0352] based on the target time segment, adjusting a matching penalty weight of a corresponding target time segment in the multi-target cost matrix;

[0353] performing smoothing processing on the adjusted multi-target cost matrix.

[0354] Optionally, the system further includes a coefficient scaling unit 611, specifically configured to:

[0355] for each single-modal sequence in each frame-level fusion feature vector, calculating a modal sequence confidence between a current frame and adjacent frames;

[0356] when a modal sequence confidence of a target single-modal sequence is lower than a preset confidence threshold, reducing a contribution degree of the target single-modal sequence to the frame-level fusion feature vector through a scaling coefficient.

[0357] Referring to Figure 7 The application further provides an alignment system for heterogeneous modal sequences, including:

[0358] a processor 701, a memory 702, an input and output unit 703, and a bus 704;

[0359] the processor 701 is connected with the memory 702, the input and output unit 703, and the bus 704;

[0360] the memory 702 stores a program, and the processor 701 invokes the program to execute any of the above methods.

[0361] The application further relates to a computer readable storage medium, which stores a program, and when the program runs on a computer, the computer executes any of the above methods.

[0362] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the above-described system, device and unit can refer to the corresponding process in the foregoing method embodiments, and will not be described here.

[0363] In several embodiments provided in the present application, it should be understood that the disclosed system, device and method can be implemented in other manners. For example, the described device embodiments are merely schematic. The division of the units is merely a logical function division. There can be another division manner for the actual implementation, for example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the displayed or discussed mutual couplings or direct couplings or communication connections can be indirect couplings or communication connections through some interfaces, devices or units, and can be in electrical, mechanical or other forms.

[0364] The units described as separated components can or can not be physically separated, and the components displayed as units can or can not be physical units, i.e., can be located in one place, or can be distributed on a plurality of network units. Some or all of the units can be selected according to actual needs to achieve the purposes of the embodiments.

[0365] In addition, each functional unit in the embodiments of the present application can be integrated in a processing unit, or each unit can exist physically as a separate unit, or two or more units can be integrated in one unit. The integrated unit can be implemented in the form of hardware, or in the form of a software functional unit.

[0366] When the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer readable storage medium. Based on such an understanding, the technical solutions of the present application essentially, or the part that contributes to the prior art, or all or a part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium, and includes several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in the embodiments of the present application. The foregoing storage medium includes: U disk, mobile hard disk, read-only memory (ROM, read-only memory), random access memory (RAM, random access memory), magnetic disk or optical disk, and various other media that can store program codes.

Claims

1. An alignment method for heterogeneous modal sequences, characterized in that, The method includes: For the input acoustic feature stream AFS, the acoustic feature vectors and fundamental frequency features of each frame are extracted based on the window function and overlap rate to form the first mode sequence; Word-level probabilistic encoding is performed on the input language modality sequence LBPV to generate a word boundary probability vector sequence; The lip motion trajectory is extracted from the input visual spatial keypoint matrix (VSKM), and a second modality sequence is constructed based on the target keypoint set. The mutual information function between the first modality sequence and the word boundary probability vector sequence is calculated based on the joint probability density. The distance function between the first modal sequence and the second modal sequence is calculated based on the weighted distance. The modality weight coefficients are dynamically calculated based on the mutual information function and the distance function, and a frame-level fusion feature vector is constructed based on the modality weight coefficients. A multi-objective cost matrix is ​​constructed based on the frame-level fusion feature vectors. Each element of the multi-objective cost matrix includes the inter-frame difference between the frame-level fusion feature vectors, the mutual information loss term, and the key point matching error term. Based on the preset maximum slope constraint parameter, a path search is performed on the multi-objective cost matrix to obtain at least one minimum cost path that satisfies continuity and monotonicity. Output a unified time-aligned sequence based on the minimum cost path.

2. The alignment method for heterogeneous modal sequences according to claim 1, characterized in that, The mutual information function calculated based on the joint probability density of the first modality sequence and the word boundary probability vector sequence includes: The first modality sequence and the word boundary probability vector sequence are subjected to time window sliding sampling with a fixed step size, and local feature vector pairs within the corresponding time segments are extracted. The KDE method is used to construct the joint probability density function of each pair of local feature vectors, where the kernel function is a Gaussian kernel and the bandwidth parameter is adaptively adjusted according to the sampled local entropy value. Estimate the marginal probability density function and the joint probability density function separately; The mutual information function is calculated based on a pre-defined information entropy formula.

3. The alignment method for heterogeneous modal sequences according to claim 1, characterized in that, The method for calculating the distance between the first modal sequence and the second modal sequence based on weighted distance includes: Select a subset of key points from the normalized set of lip key points in each frame to construct a lip movement feature vector; The acoustic feature vectors of each frame in the first modal sequence are transformed into a feature space with the same dimension as the lip movement feature vectors through linear mapping, and a comparison pair is obtained. The distance function between the two frames in the comparison pair is calculated using a preset distance function.

4. The alignment method for heterogeneous modal sequences according to claim 1, characterized in that, The construction of a multi-objective cost matrix based on the frame-level fused feature vectors, wherein each element of the multi-objective cost matrix includes the inter-frame difference between the frame-level fused feature vectors, a mutual information loss term, and a keypoint matching error term, including: For any two frame-level fused feature vectors, calculate the inter-frame difference, which is the square of the Euclidean distance between the two frame-level fused feature vectors, and is used to measure the degree of feature change between the current time and the adjacent time. The negative logarithm of the calculated frame-level mutual information value is then taken after adding a preset minimum constant and constructing the mutual information loss term. The keypoint matching error term is calculated based on the weighted squared difference between the visual keypoint coordinates estimated by the acoustic feature flow and the actual visual keypoint coordinates, where the weighting coefficients are set according to the importance of each keypoint. The inter-frame difference, mutual information loss term, and key point matching error term are linearly combined according to preset weighting coefficients to obtain the values ​​of the corresponding elements in the multi-objective cost matrix. The weighting coefficients are used to control the proportion of each loss term in the overall cost, and the sum of the three weighting coefficients is one. Using the maximum value in each item as a reference, the values ​​of all elements in the multi-objective cost matrix are mapped to a preset range.

5. The alignment method for heterogeneous modal sequences according to claim 1, characterized in that, The method involves performing a path search on the multi-objective cost matrix based on a preset maximum slope constraint parameter, and obtaining at least one minimum cost path that satisfies both continuity and monotonicity, including: Predetermine the maximum slope constraint parameter in the path search to limit the maximum step size of the path in the time and modal dimensions; Based on the maximum slope constraint parameter, a dynamic path search is performed on the multi-objective cost matrix under monotonicity and continuity constraints to obtain the path sequence that minimizes the cumulative cost. The path sequence consists of multiple nodes, and each node corresponds to a multimodal frame combination. The step of outputting a unified time-aligned sequence based on the minimum cost path includes: Backtrack from the starting boundary to the ending boundary of the multi-objective cost matrix and extract the modal time index combination corresponding to the minimum cost path; Based on the combination of time indices of each modality contained in the minimum cost path, a unified reference time axis is established, the original data sequences of each heterogeneous modality are synchronously mapped onto the unified reference time axis, and a unified time-aligned sequence is output.

6. The alignment method for heterogeneous modal sequences according to claim 1, characterized in that, The step of extracting the lip motion trajectory from the input visual spatial keypoint matrix (VSKM) and constructing a second modality sequence based on the target keypoint set includes: Receive a facial key point sequence matrix containing multiple frames of video images, wherein the facial key point sequence matrix is ​​the two-dimensional spatial coordinate information of each key point in the facial region identified from consecutive frame images by a visual acquisition module; Based on a predefined set of lip keypoint indexes, the motion trajectory of the lip region corresponding to the target frame is extracted from the face keypoint sequence matrix and constructed as a sequence of lip region motion trajectories. The set of lip keypoint indexes includes the coordinate indexes of key anatomical feature points, including at least the upper lip, lower lip, and corners of the mouth. The motion trajectory sequence of the lip region is converted into a second modal sequence.

7. The alignment method for heterogeneous modal sequences according to claim 1, characterized in that, The method further includes: The frame-level fusion feature vectors of multiple consecutive time frames in the frame-level fusion feature vector are subjected to first-order and second-order difference calculations to obtain the local change rate and local change curvature of the frame-level fusion feature vector. The local curvature is compared with a preset curvature threshold, and the target time segment with drastic local changes is identified based on the comparison result. Based on the target time segment, adjust the matching penalty weights of the corresponding target time segments in the multi-objective cost matrix; The adjusted multi-objective cost matrix is ​​then smoothed.

8. The alignment method for heterogeneous modal sequences according to claim 1, characterized in that, The method further includes: For each frame-level fused feature vector containing a single modal sequence, calculate the modal sequence confidence between the current frame and the frames before and after it. When the modality sequence confidence of the target single modality sequence is lower than the preset confidence threshold, the contribution of the target single modality sequence to the frame-level fusion feature vector is reduced by the scaling factor.

9. An alignment device for heterogeneous modal sequences, characterized in that, The device includes: The first modal sequence generation unit is used to extract acoustic feature vectors and fundamental frequency features of each frame from the input acoustic feature stream (AFS) based on the window function and overlap rate, and to form the first modal sequence. The second modality sequence generation unit is used to perform word-level probabilistic encoding on the input language modality sequence LBPV to generate a word boundary probability vector sequence. The third modality sequence generation unit is used to extract the lip motion trajectory from the input visual spatial keypoint matrix (VSKM) and construct the second modality sequence based on the target keypoint set. The first computing unit is used to calculate the mutual information function between the first modality sequence and the word boundary probability vector sequence based on the joint probability density; The second calculation unit is used to calculate the distance function between the first modality sequence and the second modality sequence based on the weighted distance; The third calculation unit is used to dynamically calculate the modality weight coefficients based on the mutual information function and the distance function, and to construct a frame-level fusion feature vector based on the modality weight coefficients; A matrix construction unit is used to construct a multi-objective cost matrix based on the frame-level fusion feature vectors. Each element of the multi-objective cost matrix includes the inter-frame difference between the frame-level fusion feature vectors, a mutual information loss term, and a key point matching error term. The path search unit is used to perform path search on the multi-objective cost matrix based on a preset maximum slope constraint parameter, and obtain at least one minimum cost path that satisfies continuity and monotonicity. The output unit is used to output a uniform time-aligned sequence based on the minimum cost path.

10. An alignment system for heterogeneous modal sequences, characterized in that, The system includes: Processor, memory, input / output units, and bus; The processor is connected to the memory, the input / output unit, and the bus; The memory stores a program, which the processor invokes to perform the method as described in any one of claims 1 to 8.

11. A computer-readable storage medium, characterized in that, The computer-readable storage medium contains a program that, when executed on a computer, performs the method as described in any one of claims 1 to 8.

Images