Multi-modal key frame extraction method and device based on Riemannian manifold and related equipment

Abstract

The invention relates to the technical field of artificial intelligence, can be applied to the field of science and technology finance / digital medical treatment, and discloses a multi-modal key frame extraction method and device based on a Riemannian manifold and related equipment. The method comprises the steps that visual, audio and text initial features of multi-modal data are extracted, visual spatial features are obtained through a visual conversion network, and audio time-frequency features are obtained through a voice recognition model; performing dimension normalization on the initial features, and aligning each mode in a feature space to obtain a unified dimension feature vector; mapping the feature vector to a product manifold space, constructing each modal sub-manifold and calculating a metric structure to obtain a multi-modal manifold feature; calculating geodesic distances of continuous frames, and identifying key frames according to geodesic difference sampling and local extremum detection to obtain a key frame index and a multi-modal manifold coordinate; and based on the similarity, calculating a fusion weight and carrying out weighted fusion to obtain a multi-modal key frame fusion feature. According to the method, the multi-modal key frame extraction accuracy and stability are improved.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology and can be applied to the fields of fintech / digital healthcare, particularly to a method, apparatus and related equipment for extracting multimodal keyframes based on Riemannian manifolds. Background Technology

[0002] Traditional video analysis and processing techniques primarily rely on image-based feature extraction, typically extracting static features from video frames for motion analysis or behavior recognition. However, single-modal analysis cannot effectively capture the complex spatiotemporal relationships and semantic information in videos, especially during keyframe extraction, where traditional methods struggle to accurately identify frames representing important moments in the video. With the development of multimodal technologies, audio and other sensor signals have been introduced to enhance keyframe extraction. For example, in security monitoring, audio signals are often used to help identify key moments in videos and help identify keyframes for timely detection of abnormal events. However, traditional multimodal fusion often relies on Euclidean space for feature alignment, neglecting the nonlinear geometric relationships between modalities, resulting in inaccurate keyframe extraction and affecting the effectiveness of video analysis. In the field of digital healthcare, traditional image diagnostic techniques typically analyze keyframes extracted from video images but fail to effectively combine other modal signals, such as electrocardiogram (ECG) or physiological signals. This means that the selection of keyframes during diagnosis cannot fully reflect the patient's health status, thus affecting diagnostic accuracy. Similarly, in the fintech sector, traditional video analytics techniques are used to extract keyframes from surveillance footage and identify suspicious behavior. However, when faced with real-time financial data and market fluctuations, they cannot effectively fuse this time-series data with keyframes in the video, resulting in the failure to capture and warn of potential risks in a timely manner. Overall, traditional technologies, when processing multimodal data, cannot accurately capture the deep nonlinear relationships between keyframes, resulting in poor fusion effects and impacting the accuracy and analytical effectiveness of keyframe extraction in various application scenarios. Summary of the Invention

[0003] The main technical problem addressed by the embodiments of this application is the poor multimodal data fusion effect in traditional keyframe extraction techniques, which cannot accurately capture the nonlinear relationship between keyframes.

[0004] To address the aforementioned technical problems, the first technical solution adopted in this application is: providing a multimodal keyframe extraction method based on Riemannian manifolds, comprising: extracting initial features of visual, audio, and text modalities from multimodal data; wherein the spatial features of the visual modality are extracted using a visual transformation network, and the time-frequency features of the audio modality are extracted using a speech recognition model; performing dimensionality normalization processing on the extracted initial features to map the features of each modality data in a unified feature space, obtaining an initial feature vector of unified dimension; and mapping the initial feature vector to a product manifold space to obtain the visual modality. The system identifies submanifolds corresponding to audio and text modalities and calculates the metric structure of each submanifold to obtain the fusion features of the multimodal manifolds. Keyframes are dynamically identified based on a geodesic difference sampling strategy by calculating the geodesic distance between consecutive frames in the product manifold space. Sparse keyframe indices and the multimodal manifold coordinates of the keyframes are obtained through local extremum detection. Feature fusion processing is performed on the multimodal manifold coordinates corresponding to the keyframes based on the keyframe indices. Fusion weights are calculated based on the similarity between the multimodal manifold coordinates, and weighted fusion is performed based on these weights to obtain the fusion features of the multimodal keyframes.

[0005] Optionally, the step of mapping the initial feature vector to the product manifold space to obtain submanifolds corresponding to the visual modality, audio modality, and text modality, and calculating the metric structure of each submanifold to obtain the fusion features of the multimodal manifold includes: calculating the feature covariance matrix of the visual modality and inverting the covariance matrix to obtain the visual submanifold corresponding to the visual modality; calculating the audio submanifold corresponding to the audio modality based on the Fisher information matrix of the audio features; calculating the probability density of the feature distribution of the text modality to obtain the text submanifold corresponding to the text modality; mapping the visual submanifold, the audio submanifold, and the text submanifold to the product manifold space to obtain the initial fusion features of the multimodal manifold; calculating the fusion weight of each submanifold in the initial fusion features of the multimodal manifold, and performing weighted fusion of each modality feature based on similarity calculation to obtain the multimodal fusion features.

[0006] Optionally, the step of calculating the geodesic distance between consecutive frames in the product manifold space, dynamically identifying keyframes according to the geodesic difference sampling strategy, and obtaining sparse keyframe indexes and multimodal manifold coordinates of the keyframes through local extremum detection includes: calculating the geometric relationships of the features of the visual, audio, and text modalities between the consecutive frames in the product manifold space to obtain the geodesic distance of each modality; calculating the manifold space distance at each moment according to the geodesic distance of each modality and the geometric relationships between each modality; performing local extremum detection on the distances in the product manifold space using a sliding window strategy to determine the moment when the geometric distance exceeds the maximum value within the window as the keyframe moment; performing local extremum detection based on the keyframe moment to obtain sparse keyframe indexes, and calculating the multimodal manifold coordinates of the visual, audio, and text modalities corresponding to the keyframe moment.

[0007] Optionally, the step of performing feature fusion processing on the multimodal manifold coordinates corresponding to the keyframe according to the keyframe index, calculating fusion weights based on the similarity between the multimodal manifold coordinates, and performing weighted fusion according to the fusion weights to obtain the fusion features of the multimodal keyframe includes: calculating the manifold coordinates of the visual modality, audio modality, and text modality at each keyframe time, and mapping the obtained manifold coordinates to the product manifold space; performing query and key-value projection operations to generate query vectors, key vectors, and value vectors, calculating the similarity between the query vector and the key vector, and mapping the query vector and the key vector to the product manifold space; calculating fusion weights based on the similarity between the manifold coordinates of the visual modality, audio modality, and text modality, and calculating a weighted value based on the logarithmic mapping value of each modality; and performing weighted fusion on the manifold coordinates of the visual modality, audio modality, and text modality according to the fusion weights and the weighted values ​​to obtain the fusion features of the multimodal keyframe.

[0008] Optionally, the step of performing dimensionality normalization on the extracted initial features to map the features of each modality data in a unified feature space to obtain an initial feature vector of unified dimension includes: normalizing the initial features of the visual modality by adjusting the value range of the visual modality features using a batch normalization mechanism; normalizing the initial features of the audio modality by adjusting the feature values ​​of the audio modality using a time-frequency feature normalization mechanism; normalizing the initial features of the text modality by adjusting the feature values ​​of the text modality using word vector normalization to make the dimension of the text modality features consistent with that of the visual modality and the audio modality; and merging the normalized feature vectors of the visual modality, the audio modality, and the text modality and mapping them to a unified feature space to obtain the initial feature vector of unified dimension.

[0009] Optionally, the step of calculating the geodesic distance between consecutive frames in the product manifold space includes: obtaining the visual modal manifold coordinates, audio modal manifold coordinates, and text modal manifold coordinates corresponding to the consecutive frames; constructing multimodal manifold coordinates of the consecutive frames in the product manifold space using the visual modal manifold coordinates, the audio modal manifold coordinates, and the text modal manifold coordinates; performing logarithmic mapping on the multimodal manifold coordinates on the visual submanifold, the audio submanifold, and the text submanifold to obtain the logarithmic mapping vector for each modality; calculating the norm of the difference between the logarithmic mapping vectors of the consecutive frames on each submanifold under the metric tensor constraints of each modality to obtain the submanifold distance of each modality; and weighting and summing the submanifold distance terms corresponding to each modality according to the modal weights to obtain the geodesic distance between consecutive frames in the product manifold space, wherein the geodesic distance satisfies: ,in, Represents the geodesic distance between consecutive frames in the product manifold space. This represents the manifold coordinates of the multimodal features of the previous frame on the submanifold within a series of frames. This represents the manifold coordinates of the multimodal features of the next frame in a series of frames on the submanifold. N Indicates the number of modes. i Indicates modal index, Indicates the first i Modal weighting coefficients, Indicates the first i Modal manifold mapping, This represents the logarithmic mapping operation. Indicates the first i Modal metric tensor In terms of metric tensors Norm operations under constraints.

[0010] Optionally, the step of performing local extremum detection on the distances in the product manifold space using a sliding window strategy to determine the moment when the geometric distance exceeds the maximum value within the window is the keyframe moment includes: obtaining a geodesic distance sequence between consecutive frames in the product manifold space; setting the sliding window width of the geodesic distance sequence and determining the movement step size of the sliding window on the geodesic distance sequence; performing a local extremum detection operation based on the geodesic distance sequence segment covered by the sliding window, and determining whether the geodesic distance corresponding to the current moment is the maximum value within the coverage area of ​​the sliding window based on the operation result; if the geodesic distance corresponding to the current moment satisfies the local extremum determination condition, then the current moment is determined to be the keyframe moment, wherein the keyframe moment satisfies: ,in, This indicates the keyframe determination result. Indicates an indicator function, This represents the geodesic distance at the current moment.w Indicates the radius of the sliding window. This represents the maximum distance between geodesics within the coverage area of ​​the sliding window.

[0011] To address the aforementioned technical problems, the second technical solution adopted in this application is: providing a multimodal keyframe extraction device based on Riemannian manifolds, comprising: an initial feature extraction module, used to extract initial features of visual, audio, and textual modalities from multimodal data, wherein the spatial features of the visual modality are extracted through a visual transformation network, and the time-frequency features of the audio modality are extracted through a speech recognition model; an initial feature vector module, used to perform dimensionality normalization processing on the extracted initial features so that the features of each modality data are mapped in a unified feature space to obtain an initial feature vector of unified dimension; and a manifold fusion feature module, used to map the initial feature vector to a product manifold space to obtain... The system identifies submanifolds corresponding to the visual, audio, and text modalities, calculates the metric structure of each submanifold, and obtains the fusion features of the multimodal manifolds. A geodesic distance calculation module is used to dynamically identify keyframes based on a geodesic difference sampling strategy by calculating the geodesic distance between consecutive frames in the product manifold space, and obtains sparse keyframe indices and the multimodal manifold coordinates of the keyframes through local extremum detection. A keyframe fusion feature module is used to perform feature fusion processing on the multimodal manifold coordinates corresponding to the keyframes based on the keyframe indices, calculates fusion weights based on the similarity between the multimodal manifold coordinates, and performs weighted fusion based on the fusion weights to obtain the fusion features of the multimodal keyframes.

[0012] To solve the above-mentioned technical problems, the third technical solution adopted in the embodiments of this application is: to provide an electronic device, including: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the multimodal keyframe extraction method based on Riemannian manifold as described above.

[0013] To solve the above-mentioned technical problems, the fourth technical solution adopted in the embodiments of this application is: to provide a non-volatile computer-readable storage medium, wherein the non-volatile computer-readable storage medium stores computer-executable instructions, and when the computer-executable instructions are executed by an electronic device, the electronic device executes the multimodal keyframe extraction method based on Riemannian manifold as described above.

[0014] Unlike related technologies, this application first extracts features from the visual, audio, and text modalities during multimodal keyframe extraction, forming initial feature vectors of a unified dimension. This ensures that cross-modal information has a comparable numerical basis within the same feature space, reducing the interference of scale and dimensional differences between different modalities on subsequent distance measurement and fusion results. Furthermore, the initial feature vectors are mapped to a product manifold space, and corresponding submanifolds and metric structures are constructed for each modality. This maintains the consistency of the intramodal structure of multimodal features in a nonlinear geometric space and obtains interpretable geometric metric constraints, thus transforming the characterization of feature differences from linear distance to geometric distance constrained by a metric tensor. In the keyframe recognition stage, based on continuous frames in the product manifold space... Geodesic distance characterizes the intensity of the true geometric changes of multimodalities over time. Combined with a geodesic difference sampling strategy and local extremum detection, a sparse keyframe index and the corresponding multimodal manifold coordinates of the keyframes are formed, thereby reducing the computational burden introduced by redundant frames and improving the localization accuracy of key change segments. In the keyframe fusion stage, similarity-driven fusion weights are calculated around the keyframe index for the multimodal manifold coordinates of the keyframes, and weighted fusion is completed to form multimodal keyframe fusion features. This enables the fusion process to adaptively highlight effective information with stronger cross-modal consistency and suppress the influence of noisy modes or weakly correlated modes, thereby improving the robustness and discriminativeness of keyframe representation and enhancing the feature expression quality for downstream applications such as retrieval, classification, and summarization. Attached Figure Description

[0015] One or more embodiments are illustrated by way of example with reference to the accompanying drawings. These illustrations do not constitute a limitation on the embodiments. Elements having the same reference numerals in the drawings are denoted as similar elements. Unless otherwise stated, the figures in the drawings are not to be limited by scale.

[0016] Figure 1 This is a schematic diagram of the operating environment of the multimodal keyframe extraction method based on Riemannian manifold provided in the embodiments of this application.

[0017] Figure 2 This is a schematic diagram of the execution flow of the multimodal keyframe extraction method based on Riemannian manifold provided in the embodiments of this application.

[0018] Figure 3 This is a schematic diagram of the execution flow of obtaining the fusion features of multimodal manifolds in the multimodal keyframe extraction method based on Riemannian manifolds provided in the embodiments of this application.

[0019] Figure 4 This is a schematic diagram of the execution flow of obtaining the fusion features of multimodal keyframes in the multimodal keyframe extraction method based on Riemannian manifold provided in the embodiments of this application.

[0020] Figure 5This is a schematic diagram of the system structure of the multimodal keyframe extraction device based on Riemannian manifold provided in the embodiments of this application.

[0021] Figure 6 This is a schematic diagram of the hardware structure of an electronic device that performs a multimodal keyframe extraction method based on Riemannian manifolds, as provided in an embodiment of this application. Detailed Implementation

[0022] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application. Software tools, components, or servers not belonging to this company that appear in the embodiments of this application are merely illustrative examples and do not represent actual use.

[0023] It should be noted that, unless otherwise specified, the various features in the embodiments of this application can be combined with each other, all of which are within the protection scope of this application. Furthermore, although functional modules are divided in the device schematic diagram and a logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device schematic diagram or the order in the flowchart.

[0024] Unless otherwise defined, all technical and scientific terms used in this specification have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the scope of this application. The term "and / or" as used in this specification includes any and all combinations of one or more of the associated listed items.

[0025] To facilitate understanding of this embodiment, a multimodal keyframe extraction method based on Riemannian manifolds disclosed in this application will first be described in detail. Please refer to [link to relevant documentation]. Figure 1 , Figure 1 This is a schematic diagram of the operating environment for the multimodal keyframe extraction method based on Riemannian manifolds provided in the embodiments of this application, such as... Figure 1 As shown, the execution entity of the Riemannian-manifold-based multimodal keyframe extraction method provided in this application is generally an electronic device with a certain computing power, such as a computer. In some possible implementations, this Riemannian-manifold-based multimodal keyframe extraction method can be implemented by a processor calling computer-readable instructions stored in memory. Figure 1The computer equipment mentioned can be a server. A server can be a standalone server or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms. This can be understood as... Figure 1 The number of computer devices shown is merely illustrative and can be expanded in any number according to actual needs.

[0026] Please continue reading. Figure 2 , Figure 2 This is a schematic diagram of the execution flow of the multimodal keyframe extraction method based on Riemannian manifolds provided in the embodiments of this application, as shown below. Figure 2 As shown, it includes the following steps: S1. Extract initial features from the visual, audio, and text modal data. Spatial features of the visual modality are extracted using a visual transformation network, and temporal-frequency features of the audio modality are extracted using a speech recognition model.

[0027] Initial feature extraction transforms multimodal inputs into computationally computable structured representations. The visual transformation network focuses on extracting spatial structural information of the visual modality, enabling stable spatial representations of shapes, textures, and regional relationships within the image. The speech recognition model focuses on extracting temporal and frequency structural information of the audio modality, enabling discriminative temporal and frequency representations of the frequency band energy changes and temporal patterns of the speech signal. Initial features for the text modality are used to express semantic content and contextual relationships. The formation of these multimodal initial features provides the foundational input for subsequent cross-modal distance measurement and fusion within a unified framework, enabling keyframe discrimination to achieve stronger scene coverage without relying on a single modality.

[0028] As an example, in fintech scenarios, video modalities can originate from corporate credit due diligence images, post-loan verification images, or risk inspection images. Visual conversion networks extract spatial structural features such as the layout of business premises, signage, personnel activities, and the display of invoices or vouchers. Speech recognition models extract the time-frequency features of due diligence interview speech. Text modalities originate from the semantic features of key points in credit granting materials, due diligence minutes, or risk warning clauses. Multimodal initial features are used to form a structured expression of risk verification information and improve information coverage. In digital healthcare scenarios, video modalities can originate from remote consultation images, rehabilitation training images, or endoscopic segments. Visual conversion networks extract spatial features such as tissue structure, limb movement trajectories, instrument positions, and changes in screen areas. Speech recognition models extract time-frequency cues from doctor-patient dialogue speech. Text modalities originate from the semantic features of key points in electronic medical records, consultation records, or examination conclusions. Multimodal initial features are used to form a joint characterization of clinical information and improve the robustness of key segment localization.

[0029] S2. Perform dimension normalization on the extracted initial features so that the features of each modality are mapped in a unified feature space to obtain an initial feature vector of unified dimension.

[0030] Dimensionality normalization eliminates differences in dimensionality, numerical range, and distribution among different modal features, enabling visual, audio, and textual features to be aligned and compared within a unified feature space. Dimensionality normalization also limits the dominance of feature magnitudes on distance calculations, preventing bias in subsequent geometric measurements caused by high-dimensional or large-magnitude modalities. The initial feature vectors with unified dimensions enable additivity and weightability of cross-modal distance terms, providing stable input for product manifold space modeling, geodesic distance calculation, and similarity-driven fusion weight solving, thereby improving the numerical stability of keyframe discrimination and fusion features.

[0031] As an example, in the fintech scenario, due diligence image features, interview voice features, and credit granting text features differ in dimensionality and numerical range. Dimensional normalization makes different modal features comparable within a unified feature space, reducing the dominant effect of single modal amplitude on distance measurement, thereby improving the numerical stability of risk change detection and key segment selection. In the digital healthcare scenario, image texture and morphological features, voice time-frequency features, and medical text semantic features have significant differences in distribution. Dimensional normalization aligns multimodal features to a unified scale, reducing the interference of noise and abnormal amplitude on geometric measurements, making key frame discrimination more reflective of changes in condition and description.

[0032] As an optional implementation, the process of obtaining the initial feature vector of uniform dimension in step S2 above may also include the following steps S21 to S24.

[0033] S21. Normalize the initial features of the visual modality and adjust the value range of the visual modality features using a batch normalization mechanism.

[0034] S22. Normalize the initial features of the audio modality and adjust the feature values ​​of the audio modality using a time-frequency feature standardization mechanism.

[0035] S23. Normalize the initial features of the text modality and use word vector normalization to adjust the feature values ​​of the text modality so that the dimension of the text modality features is consistent with that of the visual and audio modalities.

[0036] S24. Merge the normalized feature vectors of the visual modality, audio modality, and text modality, and map them to a unified feature space to obtain an initial feature vector of unified dimension.

[0037] Through steps S21 to S24, the initial features of the visual, audio, and text modalities are unified and constrained in terms of numerical distribution, scale range, and dimensional structure. The batch normalization mechanism is used to stabilize the statistical distribution of visual features within a batch and suppress amplitude drift. The time-frequency feature standardization mechanism is used to compress the dynamic range of audio time-frequency features and reduce scale shift caused by differences in acquisition conditions. Word vector normalization is used to standardize the amplitude of text semantic vectors and form a dimensional structure consistent with that of visual and audio features. The multimodal feature vectors are aligned and additively constructed in a unified feature space, so that cross-modal distance measurement, similarity calculation, and weight solution have a consistent numerical basis, reducing the impact of single-modal scale bias on geometric measurement results and improving the stability and consistency of subsequent product manifold mapping, geodesic distance calculation, and keyframe recognition processes.

[0038] In the product manifold space construction stage, a Riemannian manifold is introduced to provide a metric-based nonlinear geometric space for the feature representations of visual, audio, and text modalities. The Riemannian manifold defines the local inner product relationship at any point on the manifold through a metric structure, thereby extending feature differences from linear distances in Euclidean space to distance metrics constrained by geometric structure. This allows the length, angle, and similarity judgments between features to reflect the statistical correlation and distribution structure of modal features. In this stage, visual, audio, and text modalities respectively construct visual, audio, and text submanifolds, and determine the metric structures matching each submanifold. The metric structure of the visual submanifold is obtained by inverting the visual feature covariance matrix, used to characterize the scale differences of visual features in relevant directions. The metric structure of the audio submanifold is determined by the Fisher information matrix of the audio features, used to characterize the sensitivity and information content of audio features to statistical model parameters. The metric structure of the text submanifold is formed by describing the probability density of the text feature distribution, used to characterize the centralization and dispersion of semantic features in the distribution space. Each submanifold combines in the product manifold space to form a multimodal manifold, enabling multimodal features to maintain their respective geometric structures while having a unified metric basis, providing a consistent geometric representation for subsequent inter-frame variation characterization and keyframe localization based on geodesic distance.

[0039] S3. Map the initial feature vectors to the product manifold space to obtain the submanifolds corresponding to the visual modality, audio modality, and text modality, and calculate the metric structure of each submanifold to obtain the fusion features of the multimodal manifold.

[0040] Product manifold mapping is used to transfer multimodal features from the linear assumptions of Euclidean space to a geometric space capable of expressing nonlinear structures. This allows visual, audio, and text features to maintain intramodal structural consistency across their respective submanifolds, while simultaneously forming a jointly measurable global space within the product manifold framework. The construction of submanifolds and the computation of metric structures determine the local geometric scale and distance measurement methods for each modality on the manifold, ensuring that subsequent distance calculations reflect true geometric changes rather than simple vector differences. The formation of multimodal manifold fusion features enables cross-modal associations to be represented geometrically, enhancing the ability to express complex semantic consistency, cross-modal complementarity, and higher-order nonlinear relationships, providing a more reliable geometric basis for keyframe recognition.

[0041] As an example, in the fintech scenario, there is a non-linear relationship between changes in the business environment presented in images, risk information expressed in voice, and clause constraints in text materials. Product manifold modeling maps each modality to its respective submanifold and configures the metric structure, so that cross-modal relationships can be expressed in the form of geometric relationships, thereby enhancing the ability to represent risk consistency, information conflict, and abnormal changes. In the digital healthcare scenario, changes in lesion morphology in images, changes in key prompts in voice, and changes in symptom records in text exhibit non-linear coupling. Product manifold space and metric structure make multimodal changes measurable under a unified geometric framework, which is conducive to unifying lesion evolution, symptom prompts, and record changes into comparable geometric signals.

[0042] As an alternative implementation method, please continue reading. Figure 3 , Figure 3 This is a schematic diagram illustrating the execution flow of the multimodal keyframe extraction method based on Riemannian manifolds provided in the embodiments of this application, which obtains the fused features of the multimodal manifold. Figure 3 As shown, it can specifically include the following steps S31 to S35.

[0043] S31. By calculating the feature covariance matrix of the visual modality and inverting the covariance matrix, the visual submanifold corresponding to the visual modality is obtained.

[0044] The inversion operation of the covariance matrix of visual modal features is used to characterize the structural information of visual features at the level of statistical correlation. The correlation constraints reflected by the covariance are transformed into a structured description that can be used for geometric measurement. This enables the distance calculation of visual modal on submanifold to reflect the correlation coupling and scale direction differences between features, thus providing a stable measurement basis for the geometric modeling of visual modal.

[0045] S32. Calculate the audio submanifold corresponding to the audio mode based on the Fisher information matrix of audio features.

[0046] Fisher's information matrix based on audio features is used to characterize the information content and sensitivity of audio features in the parameter space of the probabilistic model. It maps the strength of changes in audio time-frequency features into a measurable geometric structure, so that the measurement of audio modes on submanifolds not only reflects numerical differences, but also reflects the degree of influence of audio features on statistical model parameters, thereby enhancing the discriminativeness of the geometric representation of audio modes.

[0047] S33. By calculating the probability density of the feature distribution of the text modality, the text submanifold corresponding to the text modality is obtained.

[0048] By using the probability density of text modality feature distribution to describe the structural patterns of text semantic features at the statistical distribution level, the geometric representation of text modality can reflect the centrality and discreteness of semantic vectors in the distribution space, thereby providing a statistical structural basis for the construction of text modality submanifolds and improving the measurability of semantic changes.

[0049] S34. Map the visual submanifold, audio submanifold, and text submanifold to the product manifold space to obtain the initial fusion features of the multimodal manifold.

[0050] Visual submanifolds, audio submanifolds, and text submanifolds are mapped to product manifold space to form a unified multimodal geometric framework. This allows each modality to maintain its own geometric structure while entering a jointly measurable overall space, thereby transforming cross-modal associations from linear splicing to geometric combination relationships and providing a consistent spatial carrier for subsequent dynamic changes based on geodesic distance.

[0051] S35. Calculate the fusion weight of each submanifold in the initial fusion features of the multimodal manifold, and perform weighted fusion of each modal feature based on similarity calculation to obtain the multimodal fusion features.

[0052] Based on the initial fusion features of the multimodal manifold, the fusion weights of each submanifold are calculated, and similarity-driven weighted fusion is introduced to adaptively adjust the contribution of different modes in the fusion process. This allows modes with higher consistency and stronger geometric similarity to receive higher weights, thereby suppressing the interference of noisy or inconsistent modes on the fusion results and obtaining more robust and consistent multimodal fusion features.

[0053] Through steps S31 to S35, the visual modality constructs a visual submanifold using the statistical correlation structure of the feature covariance matrix and forms a metric-compatible structural constraint using the inversion operation. The audio modality uses the Fisher information matrix to characterize the information sensitivity of audio features in the probability model parameter space and constructs an audio submanifold. The text modality uses the probability density of the feature distribution to describe the statistical morphology of semantic features and constructs a text submanifold. The visual, audio, and text submanifolds are unified and combined in the product manifold space, allowing multimodal information to enter a jointly metric overall geometric framework while maintaining their respective geometric structures. Based on the initial fusion features of the multimodal manifolds, similarity-driven fusion weights are introduced and weighted fusion is performed, so that the modal contribution is adaptively adjusted with cross-modal consistency and geometric similarity, suppressing the influence of noisy or weakly correlated modalities on the fusion results, and obtaining multimodal fusion features with both geometric consistency and discriminativeness, providing a stable geometric representation basis for subsequent geodesic distance characterization and keyframe localization.

[0054] As a concrete example, in the process of mapping the initial feature vectors to the product manifold space, suppose the multimodal input contains N Types of modes, with mode index as i The initial feature vector of uniform dimension for each modality is denoted as . Submanifolds are constructed and metric structures are determined for different modalities. For the visual modality, a visual submanifold is constructed. The metric structure is determined by the inverse of the sample covariance matrix of the image patch features, satisfying: Among them, the sample covariance matrix satisfy: , K Represented as the number of image patches, Represented as a mean vector, T For transpose operation, x For visual feature vectors, construct audio submanifolds for audio modalities. The metric structure of the audio modality is determined using the Fisher information matrix corresponding to the audio features, satisfying: ,in, Represented as the Fisher information matrix of audio features, p Represented as the probability density of the characteristic distribution, x Represented as an audio feature vector, c This is expressed as a constraint constant. Furthermore, the visual submanifold, audio submanifold, and the submanifolds corresponding to the other modalities are combined under a product structure, and the product manifold satisfies: The metric tensor of the product manifold is composed of the direct sum of the metrics of each submanifold, satisfying: ,in, Represented as learnable modal weight parameters, N For the number of modes, For the first i The metric tensor of each submanifold; through the above submanifold construction and metric structure determination, the initial eigenvectors of the multimodal modes form a unified geometric representation in the product manifold space, providing a metric basis for subsequent geodesic distance calculation and fusion processing.

[0055] In the keyframe recognition stage, Riemannian manifold operations are introduced to characterize the true geometrical variation between consecutive frames within the product manifold space. Geodesic distance, defined on the Riemannian manifold, measures the shortest path length along the manifold surface between the multimodal manifold coordinates of two frames under metric structure constraints, thus avoiding the neglect of curvature and local scale differences by Euclidean distance in nonlinear spaces. To achieve the computability of geodesic distance, logarithmic mapping operations are performed on the multimodal manifold coordinates in the product manifold space on the visual, audio, and text submanifolds, mapping points on the manifolds to their corresponding tangent spaces. This allows the differences between consecutive frames to be expressed through vector differences in the tangent spaces. The metric tensor further constrains the norm calculation of the tangent space vector differences, ensuring that the distance calculation reflects the anisotropic scale and statistical structure of each submanifold, yielding the submanifold distance for each modality. The distances of each modal submanifold are weighted and summed to form a geodesic distance sequence in the product manifold space. The geodesic distance sequence reflects the geometric intensity of multimodal information changes over time. Based on local extremum detection using a sliding window, the geodesic distance sequence is peak-discriminated. The position where the geometric change reaches its relative maximum in the local neighborhood is determined as the key frame moment. Thus, the dynamic identification of key frames is achieved by driving the geometric change peak, and the key frame index and the corresponding multimodal manifold coordinates are output.

[0056] S4. By calculating the geodesic distance between consecutive frames in the product manifold space, key frames are dynamically identified according to the geodesic difference sampling strategy, and sparse key frame indexes and multimodal manifold coordinates of key frames are obtained through local extremum detection.

[0057] Geodesic distance measures the true geometric change magnitude of multimodal representations between adjacent frames in the product manifold space. Geodesic metrics, under manifold constraints, reflect the direction and magnitude of modal feature changes, making them more suitable than Euclidean distance for representing continuous evolution in nonlinear spaces. A geodesic difference sampling strategy adaptively allocates sampling density based on the strength of change, giving higher sampling attention to segments with drastic changes and reducing redundant sampling for segments with gradual changes, thus forming a sparser but more informative set of candidate keyframes. Local extremum detection explicitly marks significant inflection points in a continuous change sequence, allowing keyframe indices to correspond to the peak positions of geometric changes. The resulting keyframe indices and multimodal manifold coordinates provide accurate localization and computable geometric input for subsequent fusion.

[0058] As an example, in fintech scenarios, geodesic distance in the product manifold space characterizes the geometric variation of risk information in consecutive frames. Geodesic difference sampling and local extremum detection locate moments such as sudden changes in the business environment, key document display, and changes in risk caliber during credit verification. It outputs sparse keyframe indexes and multimodal manifold coordinates to reduce redundancy and retain high-information-density segments. In digital healthcare scenarios, geodesic distance is sensitive to morphological changes in lesion areas, physician-recommended key examination items, and newly added symptoms in medical history descriptions. Geodesic difference sampling and local extremum detection locate changes with stronger clinical significance, reducing the risk of information omission in long video retrieval and summaries.

[0059] As an optional implementation, the process of local extremum detection in step S4 above may also include the following steps S41 to S44.

[0060] S41. Calculate the geometric relationships of the features of the visual modality, audio modality, and text modality between consecutive frames in the product manifold space to obtain the geodesic distance of each modality.

[0061] The geometric relationships of consecutive frames in visual, audio and text modalities are calculated and geodesic distance is introduced to characterize the true variation range of modal features in a manifold geometry manner. This transforms the feature changes from linear differences in Euclidean space to geometric differences under the constraints of submanifold structures, thereby improving the sensitivity to nonlinear semantic evolution, cross-modal consistent changes and abrupt events.

[0062] S42. Calculate the manifold space distance at each moment based on the geodesic distance of each mode and the geometric relationship between each mode.

[0063] Based on the geodesic distances of each modality, the manifold spatial distance at each moment is formed to merge multimodal changes into a comparable unified change signal, so that visual changes, audio changes and text changes are accumulated and aligned on the same metric scale. This provides a monotonically consistent distance sequence that can be used for threshold and peak detection for subsequent keyframe discrimination, and reduces the impact of occasional fluctuations in single modality on the overall discrimination.

[0064] S43. By using a sliding window strategy to perform local extremum detection on the distance in the product manifold space, the moment when the geometric distance exceeds the maximum value within the window is determined as the key frame moment.

[0065] The sliding window strategy and local extremum detection are used to separate significant peaks in the distance sequence of the manifold space from the continuously changing background. The maximum value determination within the window is used to take the moment of sudden increase in distance as the inflection point of change for structured localization, so that key frame discrimination does not depend on a fixed threshold but becomes relative change intensity discrimination, thereby maintaining the consistency of key frame localization under different scenes, different video lengths and different modal noise conditions.

[0066] S44. Perform local extremum detection based on key frame moments to obtain sparse key frame indices, and calculate the multimodal manifold coordinates of the visual modality, audio modality, and text modality corresponding to the key frame moments.

[0067] Local extremum detection under keyframe moment constraints is used to solidify peak moments into sparse keyframe indices, so that the keyframe set achieves a balance between quantity and information density. Each keyframe index is associated with the visual modality, audio modality and text modality manifold coordinates of the corresponding moment, thereby forming a computable geometric input for subsequent fusion and representation construction, improving the cross-modal consistency and reusability of keyframe representation.

[0068] Through steps S41 to S44, the multimodal geometric changes of continuous frames in the product manifold space are decomposed into geodesic measurements of the visual submanifold, audio submanifold, and text submanifold, quantifying the change amplitude of each modality in the form of distance under manifold geometric constraints. The multimodal geodesic distances are further converged into a unified distance signal at each moment, aligning visual structural changes, audio temporal-frequency changes, and text semantic changes on the same scale and forming comparable temporal change trajectories. Sliding window and local extremum detection establish a local comparison benchmark on the distance trajectory, determining the moment when the change intensity reaches a relative peak in the local neighborhood as the keyframe moment, thereby separating significant inflection points in the continuous change process from background fluctuations and reducing the dependence of fixed threshold selection on scene differences and noise levels. The formation of keyframe moments and sparse keyframe indexes solidifies key change points into an indexable discrete set, and associates each keyframe index with the corresponding visual modality, audio modality, and text modality manifold coordinates, so that subsequent fusion weight solution and keyframe fusion feature construction have a consistent geometric input and temporal localization basis.

[0069] As a preferred embodiment, the process of calculating the geodesic distance between consecutive frames in the product manifold space in step S4 above may further include the following steps S451 to S454.

[0070] S451. Obtain the visual modal manifold coordinates, audio modal manifold coordinates, and text modal manifold coordinates corresponding to consecutive frames, and construct the multimodal manifold coordinates of consecutive frames in the product manifold space using the visual modal manifold coordinates, audio modal manifold coordinates, and text modal manifold coordinates.

[0071] Visual modal manifold coordinates, audio modal manifold coordinates, and text modal manifold coordinates are combined under time alignment to form multimodal manifold coordinates in a product manifold space. This is used to solidify cross-modal information at the same moment in a unified geometric carrier, while preserving the structural constraints of the visual submanifold, audio submanifold, and text submanifold. This allows cross-modal changes to be compared in the same coordinate system, reduces distance distortion caused by differences in different modal scales, and improves the consistency of characterizing changes in consecutive frames.

[0072] S452. Perform logarithmic mapping on the multimodal manifold coordinates over the visual, audio, and text submanifolds to obtain the logarithmic mapping vectors for each modality.

[0073] Logarithmic mapping is performed on visual, audio, and text submanifolds to transform nonlinear coordinate relationships on the manifolds into approximately linear descriptions in the tangent space. This enables multimodal manifold coordinates to have a differential and summable operational form, thereby transferring the nonlinear effects caused by curvature to the logarithmic mapping domain for processing, enhancing the computability and numerical stability of continuous frame change calculations.

[0074] S453. Under the metric tensor constraints of each modality, calculate the norm of the logarithmic mapping vector difference of consecutive frames on each submanifold to obtain the submanifold distance of each modality.

[0075] Norm calculation under metric tensor constraints incorporates the direction and magnitude of the logarithmic mapping vector difference into the geometric metric system, enabling the distance metric to reflect the anisotropic scale and related structure of the submanifold. This avoids the bias caused by the simple Euclidean difference ignoring the statistical structure, and the resulting submanifold distance can more accurately reflect the true magnitude of visual structural changes, audio time-frequency changes, and text semantic changes in their respective geometric spaces.

[0076] S454. Weighted summation of the submanifold distance terms corresponding to each mode according to the mode weights yields the geodesic distance between consecutive frames in the product manifold space. The geodesic distance satisfies: ,in, Represents the geodesic distance between consecutive frames in the product manifold space. This represents the manifold coordinates of the multimodal features of the previous frame on the submanifold within a series of frames. This represents the manifold coordinates of the multimodal features of the next frame in a series of frames on the submanifold. N Indicates the number of modes. i Indicates modal index, Indicates the first i Modal weighting coefficients, Indicates the first i Modal manifold mapping, This represents the logarithmic mapping operation. Indicates the first i Modal metric tensor In terms of metric tensors Norm operations under constraints.

[0077] The modal weighted summation converges the distances of each submanifold into the geodesic distance of the product manifold space, so that the overall distance simultaneously reflects the multimodal contribution and allows the weight parameters to adjust the reliability and influence of different modes. This reduces the interference of noisy modes or weakly correlated modes on the overall distance, forming a geodesic distance sequence that is more sensitive to and more stable to multimodal geometric changes in continuous frames, and provides a more discriminative distance signal for determining the change peak at key frame moments.

[0078] As another preferred embodiment, the process of determining the key frame time in step S4 above also includes the following steps S461 to S464.

[0079] S461. Obtain the geodesic distance sequence between consecutive frames in the product manifold space.

[0080] The acquisition of geodesic distance sequences is used to transform the multimodal geometric changes of continuous frames in the product manifold space into one-dimensional time-series signals, enabling the subsequent discrimination process to handle cross-modal change intensity in a unified distance domain. The geodesic distance sequences also retain the constraints of manifold metrics on nonlinear structures, thus providing a geometrically meaningful continuous quantitative basis for locating key change points.

[0081] S462. Set the sliding window width of the geodesic distance sequence and determine the step size of the sliding window on the geodesic distance sequence.

[0082] The settings of the sliding window width and the movement step size are used to determine the time range and detection resolution of local comparison. The window width controls the scale of local extremum determination, so that key change points have controllable sensitivity at different time scales. The movement step size controls the scanning density of the distance sequence, so that local extremum detection achieves a balance between computational overhead and positioning accuracy, thereby improving the adaptability of key frame determination under different video lengths and different change frequencies.

[0083] S463. Based on the geodesic distance sequence segment covered by the sliding window, perform local extremum detection operation, and determine whether the geodesic distance at the current time is the maximum value within the coverage area of ​​the sliding window based on the operation result.

[0084] The local extremum detection operation on the sliding window overlay segment is used to establish a relative change benchmark in the local time neighborhood. By comparing the magnitude of the geodesic distance at the current time with the geodesic distance at other times in the window, it is determined whether a local peak is formed at the current time, thereby separating significant abrupt changes in the distance sequence from background fluctuations and reducing the dependence of global threshold selection on scene changes and noise levels.

[0085] S464. If the geodesic distance at the current moment satisfies the local extremum determination condition, then the current moment is determined to be a keyframe moment. The keyframe moment satisfies: ,in, This indicates the keyframe determination result. Indicates an indicator function, This represents the geodesic distance at the current moment. w Indicates the radius of the sliding window. This represents the maximum distance between geodesics within the coverage area of ​​the sliding window.

[0086] The satisfaction of the local extremum determination condition is used to solidify the local peak moment into the key frame moment. The indicator function expresses the regularization process of converting the continuous distance signal into the discrete key frame determination result. The window radius limits the comparison range and constrains the locality of the key frame determination, so that the key frame moment corresponds to the position of the relative maximum change of the geodesic distance in the local neighborhood, thereby improving the stability of key frame positioning and reducing false detections caused by non-critical fluctuations.

[0087] S5. Based on the keyframe index, perform feature fusion processing on the multimodal manifold coordinates corresponding to the keyframes, calculate the fusion weights based on the similarity between the multimodal manifold coordinates, and perform weighted fusion based on the fusion weights to obtain the fused features of the multimodal keyframes.

[0088] Feature fusion processing further integrates the multimodal manifold coordinates located by keyframe indexing into a single keyframe fusion feature, enabling the keyframe representation to simultaneously incorporate visual structure, audio temporal cues, and textual semantic cues. Similarity-driven fusion weight calculation quantifies the consistency and contribution of each modality at the current keyframe, allowing weight allocation to adaptively change based on cross-modal consistency and geometric similarity, avoiding modal bias caused by fixed weights. Weighted fusion completes cross-modal integration at the manifold coordinate level, resulting in multimodal keyframe fusion features that are superior in information density, robustness, and semantic consistency, improving the recognition ability and stability of keyframe representations in complex scenes.

[0089] As an example, in the fintech scenario, the multimodal manifold coordinates corresponding to the keyframe index are used to calculate the intermodal similarity and generate fusion weights. This allows image verification clues, interview voice clues, and credit granting text clues to receive higher weights at key moments with higher consistency. The multimodal keyframe fusion features obtained by weighted fusion are more representative of risk points and improve the consistency of judgment. In the digital healthcare scenario, the image manifold coordinates, voice manifold coordinates, and text manifold coordinates corresponding to the keyframe are weighted and fused under the constraint of similarity-driven fusion weights. This allows image anomalies, key medical orders, and medical records to form more consistent fusion features at the same key moment, enhancing the reliability of key segment representation and improving the consistency of key event localization.

[0090] As an alternative implementation method, please continue reading. Figure 4 , Figure 4 This is a schematic diagram illustrating the execution flow of obtaining the fused features of multimodal keyframes in the multimodal keyframe extraction method based on Riemannian manifolds provided in the embodiments of this application, as shown below. Figure 4 As shown, the process can specifically include the following steps S51 to S54.

[0091] S51. Calculate the manifold coordinates of the visual modality, audio modality, and text modality at each keyframe moment, and map the obtained manifold coordinates to the product manifold space.

[0092] The coordinates of the visual, audio, and text modalities at key frame moments are uniformly extracted and placed in the product manifold space to solidify the multimodal information of the key frame into a computable object on the same geometric carrier. This allows subsequent similarity calculations and weight solutions to be carried out under consistent geometric constraints, avoiding measurement distortion caused by direct fusion of different modalities at different spatial scales, and improving the cross-modal alignment of key frame representations.

[0093] S52. Perform query and key-value projection operations to generate query vector, key vector and value vector, calculate the similarity between query vector and key vector, and map query vector and key vector to product manifold space.

[0094] The projection operation of query vector, key vector and value vector is used to organize the multimodal manifold information at the keyframe into a structured representation that can be used for relevance modeling. The similarity between query and key is used to characterize the matching strength and correlation between multimodal features. The relevance calculation is improved from simple vector similarity to learnable projection relevance, which enables the fusion process to selectively aggregate complementary information of different modalities within the keyframe, thereby enhancing the fusion process's ability to suppress noise and weakly related information.

[0095] S53. Calculate the fusion weights based on the similarity between the manifold coordinates of the visual modality, audio modality, and text modality, and calculate the weighted value based on the logarithmic mapping value of each modality.

[0096] The calculation of fusion weights is used to quantify the contribution distribution of vision, audio, and text at keyframes. The similarity-driven weight formation mechanism gives higher weights to modalities with stronger consistency and closer correlation. The logarithmic mapping value is used to transform the nonlinear coordinate relationship on the manifold into an additivity description in the tangent space. The calculation of the weight value constrains the magnitude and direction of the modal contribution on the tangent space scale, so that the weight allocation not only reflects the degree of correlation, but also reflects the consistency of the direction of geometric change, thereby improving the geometric consistency and discriminability of the fusion result.

[0097] S54. Based on the fusion weights and weighted values, the manifold coordinates of the visual modality, audio modality, and text modality are weighted and fused to obtain the fusion features of the multimodal keyframes.

[0098] Weighted fusion, constrained by both fusion weights and weighted values, is used to achieve unified aggregation of multimodal coordinates within the product manifold space. This enables the fusion process to simultaneously possess "correlation-driven contribution allocation" and "numerical combination under geometric constraints," thereby forming more stable and representative multimodal keyframe fusion features. This reduces the impact of single-modal abnormal fluctuations on keyframe representation and improves the discriminative ability of keyframe features.

[0099] Through steps S51 to S54, the visual, audio, and textual manifold coordinates at keyframe moments are uniformly organized into a product manifold space, enabling the multimodal information of keyframes to have a computable and consistent representation on the same geometric carrier. The projection operation of query vectors, key vectors, and value vectors transforms multimodal manifold information into structured variables that can be used for correlation modeling. The similarity between query vectors and key vectors is used to characterize the matching strength between multimodal features and support attention allocation. The fusion weights obtained based on manifold coordinate similarity are used to quantify the contribution ratio of vision, audio, and text at keyframes. The logarithmic mapping value provides an additivity description in the tangent space and is used to form a weighted value to constrain the combination of different modalities in terms of geometric direction and amplitude. The weighted fusion under the joint constraint of fusion weights and weighted values ​​completes cross-modal aggregation under the condition of maintaining the geometric structure of the product manifold, forming a multimodal keyframe fusion feature with higher information density and stronger cross-modal consistency, thereby improving the stability and discriminability of keyframe representation in complex scenes.

[0100] As a concrete example, in the process of performing feature fusion processing on the multimodal manifold coordinates corresponding to keyframes based on keyframe indices, let the product manifold be... Let x be the feature vector of a keyframe on the product manifold. Perform projection operations on the query vector, key vector, and value vector of the keyframe set. The projection operation satisfies: in, W Q , W k , W v Represented as a learnable projection matrix, T This indicates the transpose operation. Represented as a product manifold Logarithmic mapping operations on; Q, K, VThese are the query vector set, key vector set, and value vector set corresponding to the keyframe set, respectively. Further, a perceptual attention weight is calculated based on the correlation between the query vector and the key vector, and the attention weight satisfies: in, Indicates the first i The query vector and the first j Attention weights corresponding to each key vector Represents a product manifold Exponential mapping operations on, Represents the metric tensor g Induced inner product operation, d The dimension is vector. Based on attention weights, aggregation operations are performed under manifold structure constraints to obtain fused features. The aggregation operation satisfies: in, Represented as a product manifold Exponential mapping operations on, Indicates the first j A vector of values Z This represents the fusion feature after the keyframe set is aggregated, and is used to characterize the fusion result of multimodal keyframes.

[0101] As a more concrete example, taking a news video summarization task, the input is a 10-minute news video at 25fps, totaling approximately 15,000 frames, along with synchronized audio. In the feature extraction stage, Vision Transformer (ViT) is used to extract visual features for each video frame, outputting a 768-dimensional CLS token representation per frame. For the audio signal, Wav2Vec 2.0 is used to extract audio features, aligning them frame-by-frame to obtain a 1024-dimensional audio time-frequency feature vector per frame. In the manifold embedding stage, visual and audio features are mapped to a product manifold space: the metric structure of the visual modality is estimated by the covariance of features from 300 random image patches; the metric structure of the audio modality is determined by the Fisher information matrix corresponding to the MFCC features, which is calculated using the Hessian matrix of the MFCC. In the keyframe detection stage, geodesic distances between consecutive frames are calculated in the product manifold space to form a geodesic distance sequence. Local extrema of the geodesic distances are detected within a 30-frame sliding window, resulting in 120 keyframe indices selected from 15,000 frames. Simultaneously, the multimodal manifold coordinates corresponding to these 120 keyframes are obtained. In the feature fusion stage, the multimodal manifold coordinates of the keyframes are input into a Metric AwareTransformer. Fusion weights are calculated based on a metric-aware attention mechanism, and weighted fusion is performed to generate a 128-dimensional unified representation as the multimodal keyframe fusion feature for news video summarization, used for subsequent downstream tasks such as summary generation, retrieval, or classification.

[0102] The multimodal keyframe extraction method based on Riemannian manifolds provided in this application performs unified dimensional normalization and feature space alignment on visual, audio, and text modal features. It maps the initial multimodal features to a product manifold space and constructs corresponding submanifolds and metric structures for each modality. The geodesic distance under metric structure constraints characterizes the true variation amplitude of continuous frames in nonlinear geometric space. A geodesic difference sampling strategy and sliding window local extremum detection are combined to achieve dynamic recognition and sparse index output of keyframes. Furthermore, a metric-aware attention fusion mechanism is introduced at the keyframe level. Fusion weights are calculated based on manifold coordinate similarity, and weighted aggregation is performed in the logarithmic mapping domain to obtain multimodal keyframe fusion features. This maintains geometric consistency and improves the stability and discriminativeness of cross-modal fusion during multimodal data processing, reduces the impact of scale differences and noise fluctuations between different modalities on keyframe localization, and enhances the ability of keyframe extraction results to characterize complex scenes and nonlinear relationships. Ultimately, this improves the feature representation quality and analysis effect for downstream tasks such as retrieval, classification, and summarization.

[0103] Please continue reading. Figure 5 , Figure 5 This is a schematic diagram of the system structure of the multimodal keyframe extraction device based on Riemannian manifold provided in the embodiments of this application, as shown below. Figure 5 As shown, the Riemannian manifold-based multimodal keyframe extraction device 50 includes: an initial feature extraction module 51, an initial feature vector module 52, a manifold fusion feature module 53, a geodesic distance calculation module 54, and a keyframe fusion feature module 55. The initial feature extraction module 51 is specifically used to extract initial features from the visual, audio, and text modal data. The spatial features of the visual modality are extracted through a visual transformation network, and the time-frequency features of the audio modality are extracted through a speech recognition model. The initial feature vector module 52 is specifically used to perform dimensionality normalization processing on the extracted initial features so that the features of each modality are mapped in a unified feature space to obtain an initial feature vector of unified dimension. The manifold fusion feature module 53 is specifically used to map the initial feature vector to a product manifold space to obtain the submanifolds corresponding to the visual, audio, and text modalities, and calculate... The metric structure of each submanifold is used to obtain the fusion features of the multimodal manifold; the geodesic distance calculation module 54 is specifically used to dynamically identify key frames by calculating the geodesic distance between consecutive frames in the product manifold space, according to the geodesic difference sampling strategy, and obtain sparse key frame indexes and the multimodal manifold coordinates of the key frames through local extremum detection; the key frame fusion feature module 55 is specifically used to perform feature fusion processing on the multimodal manifold coordinates corresponding to the key frames according to the key frame index, calculate the fusion weight based on the similarity between the multimodal manifold coordinates, and perform weighted fusion according to the fusion weight to obtain the fusion features of the multimodal key frames.

[0104] As an optional implementation, the manifold fusion feature module 53 is further specifically used to calculate the feature covariance matrix of the visual modality and invert the covariance matrix to obtain the visual submanifold corresponding to the visual modality; calculate the audio submanifold corresponding to the audio modality based on the Fisher information matrix of the audio features; obtain the text submanifold corresponding to the text modality by calculating the probability density of the feature distribution of the text modality; map the visual submanifold, the audio submanifold, and the text submanifold to the product manifold space to obtain the initial fusion features of the multimodal manifold; calculate the fusion weight of each submanifold in the initial fusion features of the multimodal manifold, and perform weighted fusion of each modality feature based on similarity calculation to obtain the multimodal fusion features.

[0105] As an optional implementation, the geodesic distance calculation module 54 is further specifically used to calculate the geometric relationship between the features of the visual modality, audio modality, and text modality between consecutive frames in the product manifold space to obtain the geodesic distance of each modality; calculate the manifold space distance at each moment based on the geodesic distance of each modality and the geometric relationship between each modality; perform local extremum detection on the distance in the product manifold space using a sliding window strategy to determine the moment when the geometric distance exceeds the maximum value within the window as the key frame moment; perform local extremum detection based on the key frame moment to obtain a sparse key frame index, and calculate the multimodal manifold coordinates of the visual modality, audio modality, and text modality corresponding to the key frame moment.

[0106] As an optional implementation, the keyframe fusion feature module 55 is further specifically used to calculate the manifold coordinates of the visual modality, audio modality, and text modality at each keyframe moment, and map the obtained manifold coordinates to the product manifold space; perform query and key-value projection operations to generate query vectors, key vectors, and value vectors, calculate the similarity between the query vector and the key vector, and map the query vector and the key vector to the product manifold space; calculate fusion weights based on the similarity between the manifold coordinates of the visual modality, audio modality, and text modality, and calculate a weighted value based on the logarithmic mapping value of each modality; and perform weighted fusion of the manifold coordinates of the visual modality, audio modality, and text modality based on the fusion weights and the weighted values ​​to obtain the fusion features of the multimodal keyframe.

[0107] As an optional implementation, the initial feature vector module 52 is further specifically used to normalize the initial features of the visual modality by using a batch normalization mechanism to adjust the value range of the visual modality features; to normalize the initial features of the audio modality by using a time-frequency feature normalization mechanism to adjust the feature values ​​of the audio modality; to normalize the initial features of the text modality by using word vector normalization to adjust the feature values ​​of the text modality, so that the dimension of the text modality features is consistent with that of the visual modality and the audio modality; and to merge the normalized feature vectors of the visual modality, the audio modality, and the text modality, mapping them to a unified feature space to obtain the initial feature vector of a unified dimension.

[0108] As an optional implementation, the geodesic distance calculation module 54 is further specifically used to obtain the visual modal manifold coordinates, audio modal manifold coordinates, and text modal manifold coordinates corresponding to consecutive frames; construct multimodal manifold coordinates of consecutive frames in the product manifold space using the visual modal manifold coordinates, the audio modal manifold coordinates, and the text modal manifold coordinates; perform logarithmic mapping on the multimodal manifold coordinates on the visual submanifold, the audio submanifold, and the text submanifold to obtain the logarithmic mapping vector of each modality; calculate the norm of the difference of the logarithmic mapping vectors of consecutive frames on each submanifold under the metric tensor constraints of each modality to obtain the submanifold distance of each modality; and sum the submanifold distance terms corresponding to each modality according to the modal weights to obtain the geodesic distance between consecutive frames in the product manifold space, wherein the geodesic distance satisfies: ,in, Represents the geodesic distance between consecutive frames in the product manifold space. This represents the manifold coordinates of the multimodal features of the previous frame on the submanifold within a series of frames. This represents the manifold coordinates of the multimodal features of the next frame in a series of frames on the submanifold. N Indicates the number of modes. i Indicates modal index, Indicates the first i Modal weighting coefficients, Indicates the first i Modal manifold mapping, This represents the logarithmic mapping operation. Indicates the first i Modal metric tensor In terms of metric tensors Norm operations under constraints.

[0109] As an optional implementation, the geodesic distance calculation module 54 is further specifically used to obtain the geodesic distance sequence between consecutive frames in the product manifold space; set the sliding window width of the geodesic distance sequence and determine the moving step size of the sliding window on the geodesic distance sequence; perform local extremum detection operation according to the geodesic distance sequence segment covered by the sliding window, and determine whether the geodesic distance corresponding to the current time is the maximum value within the coverage area of ​​the sliding window according to the operation result; if the geodesic distance corresponding to the current time satisfies the local extremum determination condition, then the current time is determined to be a key frame time, wherein the key frame time satisfies: ,in, This indicates the keyframe determination result. Indicates an indicator function, This represents the geodesic distance at the current moment. w Indicates the radius of the sliding window. This represents the maximum distance between geodesics within the coverage area of ​​the sliding window.

[0110] It should be noted that the aforementioned multimodal keyframe extraction device based on Riemannian manifolds can execute the multimodal keyframe extraction method based on Riemannian manifolds provided in the embodiments of this application, and has the corresponding functional modules and beneficial effects of the method. Technical details not described in detail in the embodiments of the multimodal keyframe extraction device based on Riemannian manifolds can be found in the multimodal keyframe extraction method based on Riemannian manifolds provided in the embodiments of this application.

[0111] Figure 6 This is a schematic diagram of the hardware structure of an electronic device that performs a multimodal keyframe extraction method based on Riemannian manifolds, as provided in the embodiments of this application. Figure 6 As shown, the electronic device 600 includes: One or more processors 610 and memory 620, Figure 6 Take the 610 processor as an example.

[0112] The processor 610 and the memory 620 can be connected via a bus or other means. Figure 6 Taking the example of a connection between China and Israel via a bus.

[0113] The memory 620, as a non-volatile computer-readable storage medium, can be used to store non-volatile software programs, non-volatile computer-executable programs, and modules, such as the program instructions / modules corresponding to the Riemannian-based multimodal keyframe extraction method in the embodiments of this application. The processor 610 executes various server functions and data processing by running the non-volatile software programs, instructions, and modules stored in the memory 620, thereby implementing the Riemannian-based multimodal keyframe extraction method described in the above embodiments.

[0114] The memory 620 may include a program storage area and a data storage area. The program storage area may store the operating system and applications required for at least one function; the data storage area may store data created based on the use of the Riemannian-based multimodal keyframe extraction device. Furthermore, the memory 620 may include high-speed random access memory and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other non-volatile solid-state storage device. In some embodiments, the memory 620 may optionally include memory remotely located relative to the processor 610, and this remote memory may be connected to the Riemannian-based multimodal keyframe extraction device via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0115] The one or more modules are stored in the memory 620. When executed by the one or more processors 610, they perform the multimodal keyframe extraction method based on Riemannian manifolds in any of the above method embodiments. For example, they perform the methods described above. Figure 2 Method steps S1 to S5, Figure 3 Method steps S31 to S35, Figure 4 The method steps S51 to S54 are implemented. Figure 5 The functions of modules 51-55 in the document.

[0116] The above-described product can perform the methods provided in the embodiments of this application, and has the corresponding functional modules and beneficial effects for performing the methods. Technical details not described in detail in this embodiment can be found in the methods provided in the embodiments of this application.

[0117] This application provides a non-volatile computer-readable storage medium storing computer-executable instructions that are executed by one or more processors, for example... Figure 6 One of the processors 610 can enable the above-described one or more processors to execute the Riemannian-based multimodal keyframe extraction method in any of the above method embodiments, for example, to execute the above-described... Figure 2 Method steps S1 to S5, Figure 3 Method steps S31 to S35, Figure 4 The method steps S51 to S54 are implemented. Figure 5 The functions of modules 51-55 in the document.

[0118] This application provides a computer program product, which includes a computer program stored on a non-volatile computer-readable storage medium. The computer program includes program instructions that, when executed by an electronic device, enable the electronic device to perform the Riemannian-based multimodal keyframe extraction method described in any of the above method embodiments. For example, it can execute the method described above. Figure 2 Method steps S1 to S5, Figure 3 Method steps S31 to S35, Figure 4 The method steps S51 to S54 are implemented. Figure 5 The functions of modules 51-55 in the document.

[0119] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0120] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented using software and a general-purpose hardware platform, or of course, using hardware. Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. The storage medium can be a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM), etc.

[0121] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and not to limit them; under the concept of this application, the technical features of the above embodiments or different embodiments can also be combined, the steps can be implemented in any order, and there are many other variations of different aspects of this application as described above, which are not provided in detail for the sake of brevity; although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the foregoing embodiments, or make equivalent substitutions for some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application.

Claims

1. A multimodal keyframe extraction method based on Riemannian manifolds, characterized in that, include: Initial features of visual, audio, and text modalities are extracted from multimodal data. The spatial features of the visual modality are extracted using a visual transformation network, and the time-frequency features of the audio modality are extracted using a speech recognition model. The extracted initial features are subjected to dimension normalization so that the features of each modality are mapped in a unified feature space to obtain an initial feature vector of unified dimension. The initial feature vectors are mapped to the product manifold space to obtain the submanifolds corresponding to the visual modality, audio modality, and text modality. The metric structure of each submanifold is calculated to obtain the fusion features of the multimodal manifold. By calculating the geodesic distance between consecutive frames in the product manifold space, key frames are dynamically identified according to the geodesic difference sampling strategy, and sparse key frame indexes and multimodal manifold coordinates of the key frames are obtained through local extremum detection. Based on the keyframe index, feature fusion processing is performed on the multimodal manifold coordinates corresponding to the keyframe. The fusion weight is calculated based on the similarity between the multimodal manifold coordinates, and weighted fusion is performed based on the fusion weight to obtain the fused features of the multimodal keyframe.

2. The multimodal keyframe extraction method based on Riemannian manifolds according to claim 1, characterized in that, The steps of mapping the initial feature vector to the product manifold space to obtain submanifolds corresponding to the visual modality, audio modality, and text modality, and calculating the metric structure of each submanifold to obtain the fusion features of the multimodal manifold include: By calculating the feature covariance matrix of the visual modality and inverting the covariance matrix, the visual submanifold corresponding to the visual modality is obtained. The audio submanifold corresponding to the audio mode is calculated based on the Fisher information matrix of the audio features; By calculating the probability density of the feature distribution of the text modality, the text submanifold corresponding to the text modality is obtained; The visual submanifold, the audio submanifold, and the text submanifold are mapped to the product manifold space to obtain the initial fusion features of the multimodal manifold; The fusion weights of each submanifold in the initial fusion features of the multimodal manifold are calculated, and the modal features are weighted and fused based on similarity calculation to obtain the multimodal fusion features.

3. The multimodal keyframe extraction method based on Riemannian manifolds according to claim 1, characterized in that, The steps of calculating the geodesic distance between consecutive frames in the product manifold space, dynamically identifying key frames according to the geodesic difference sampling strategy, and obtaining sparse key frame indices and the multimodal manifold coordinates of the key frames through local extremum detection include: The geometric relationships of the features of the visual modality, audio modality, and text modality between the consecutive frames in the product manifold space are calculated to obtain the geodesic distance of each modality; The manifold space distance at each moment is calculated based on the geodesic distance of each mode and the geometric relationship between each mode; Local extrema detection of distances in the product manifold space is performed using a sliding window strategy, and the moment when the geometric distance exceeds the maximum value within the window is determined as the keyframe moment. Local extremum detection is performed based on the keyframe moments to obtain sparse keyframe indices, and the multimodal manifold coordinates of the visual modality, audio modality, and text modality corresponding to the keyframe moments are calculated.

4. The multimodal keyframe extraction method based on Riemannian manifolds according to claim 1, characterized in that, The step involves performing feature fusion processing on the multimodal manifold coordinates corresponding to the keyframes based on the keyframe index, calculating fusion weights based on the similarity between the multimodal manifold coordinates, and performing weighted fusion based on the fusion weights to obtain the fused features of the multimodal keyframes, including: Calculate the manifold coordinates of the visual modality, audio modality, and text modality at each keyframe moment, and map the obtained manifold coordinates to the product manifold space; Perform query and key-value projection operations to generate query vector, key vector and value vector, calculate the similarity between the query vector and the key vector, and map the query vector and the key vector to the product manifold space; Based on the similarity between the manifold coordinates of the visual modality, audio modality, and text modality, the fusion weight is calculated, and the weighted value is calculated based on the logarithmic mapping value of each modality; Based on the fusion weights and the weighted values, the manifold coordinates of the visual modality, audio modality, and text modality are weighted and fused to obtain the fusion features of the multimodal keyframes.

5. The multimodal keyframe extraction method based on Riemannian manifolds according to claim 1, characterized in that, The step of performing dimensionality normalization on the extracted initial features to map the features of each modality in a unified feature space and obtain an initial feature vector of unified dimension includes: The initial features of the visual modality are normalized, and the value range of the visual modality features is adjusted by a batch normalization mechanism. The initial features of the audio modality are normalized, and the feature values ​​of the audio modality are adjusted using a time-frequency feature normalization mechanism; The initial features of the text modality are normalized, and word vector normalization is used to adjust the feature values ​​of the text modality so that the dimension of the text modality features is consistent with that of the visual modality and the audio modality. The normalized feature vectors of the visual modality, the audio modality, and the text modality are merged and mapped to a unified feature space to obtain the initial feature vector of a unified dimension.

6. The multimodal keyframe extraction method based on Riemannian manifolds according to claim 1, characterized in that, The step of calculating the geodesic distance between consecutive frames in the product manifold space includes: Obtain the visual modal manifold coordinates, audio modal manifold coordinates, and text modal manifold coordinates corresponding to consecutive frames, and construct the multimodal manifold coordinates of consecutive frames in the product manifold space using the visual modal manifold coordinates, the audio modal manifold coordinates, and the text modal manifold coordinates; On the visual submanifold, the audio submanifold, and the text submanifold, a logarithmic mapping is performed on the coordinates of the multimodal manifold to obtain the logarithmic mapping vector for each modality; Under the metric tensor constraints of each modality, the norm of the logarithmic mapping vector difference of consecutive frames on each submanifold is calculated to obtain the submanifold distance of each modality; The submanifold distance terms corresponding to each mode are weighted and summed according to the mode weights to obtain the geodesic distance between consecutive frames in the product manifold space, wherein the geodesic distance satisfies: ,in, Represents the geodesic distance between consecutive frames in the product manifold space. This represents the manifold coordinates of the multimodal features of the previous frame on the submanifold within a series of frames. This represents the manifold coordinates of the multimodal features of the next frame in a series of frames on the submanifold. N Indicates the number of modes. i Indicates modal index, Indicates the first i Modal weighting coefficients, Indicates the first i Modal manifold mapping, This represents the logarithmic mapping operation. Indicates the first i The modal metric tensor In terms of metric tensors Norm operations under constraints.

7. The multimodal keyframe extraction method based on Riemannian manifolds according to claim 3, characterized in that, The step of performing local extremum detection on the distance in the product manifold space using a sliding window strategy, and determining the moment when the geometric distance exceeds the maximum value within the window as the keyframe moment, includes: Obtain the geodesic distance sequence between consecutive frames in the product manifold space; Set the sliding window width of the geodesic distance sequence and determine the movement step size of the sliding window on the geodesic distance sequence; Based on the geodesic distance sequence segment covered by the sliding window, a local extremum detection operation is performed, and the result is used to determine whether the geodesic distance at the current moment is the maximum value within the coverage area of ​​the sliding window. If the geodesic distance at the current moment satisfies the local extremum determination condition, then the current moment is determined to be a key frame moment, wherein the key frame moment satisfies: ,in, This indicates the keyframe determination result. Indicates an indicator function, This represents the geodesic distance at the current moment. w Indicates the radius of the sliding window. This represents the maximum distance between geodesics within the coverage area of ​​the sliding window.

8. A multimodal keyframe extraction device based on Riemannian manifolds, characterized in that, include: The initial feature extraction module is used to extract initial features from the visual modality, audio modality, and text modality in the multimodal data. The spatial features of the visual modality are extracted through a visual transformation network, and the time-frequency features of the audio modality are extracted through a speech recognition model. The initial feature vector module is used to perform dimension normalization processing on the extracted initial features so that the features of each modality data are mapped in a unified feature space to obtain an initial feature vector of unified dimension. The manifold fusion feature module is used to map the initial feature vector to the product manifold space to obtain the submanifolds corresponding to the visual modality, audio modality and text modality, and to calculate the metric structure of each submanifold to obtain the fusion features of the multimodal manifold; The geodesic distance calculation module is used to dynamically identify key frames by calculating the geodesic distance between consecutive frames in the product manifold space, based on the geodesic difference sampling strategy, and obtain sparse key frame indexes and multimodal manifold coordinates of the key frames through local extremum detection. The keyframe fusion feature module is used to perform feature fusion processing on the multimodal manifold coordinates corresponding to the keyframe according to the keyframe index, calculate the fusion weight based on the similarity between the multimodal manifold coordinates, and perform weighted fusion according to the fusion weight to obtain the fusion features of the multimodal keyframe.

9. An electronic device, characterized in that, include: At least one processor; as well as, A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, which, when executed by the at least one processor, enables the at least one processor to perform the multimodal keyframe extraction method based on Riemannian manifolds as described in any one of claims 1-7.

10. A non-volatile computer-readable storage medium, characterized in that, The non-volatile computer-readable storage medium stores computer-executable instructions, which, when executed by an electronic device, cause the electronic device to perform the multimodal keyframe extraction method based on Riemannian manifolds as described in any one of claims 1-7.

Images