Multi-modal data analysis method and system based on artificial intelligence
By processing multimodal data through multi-scale time warping and a four-layer progressive interactive network, the problems of rigid temporal alignment and shallow modal interaction in multimodal sentiment analysis are solved, achieving more accurate and detailed sentiment analysis.
Patent Information
- Application Number
- CN202610106569.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-01-27
- Publication Date
- 2026-05-15
AI Technical Summary
Existing multimodal sentiment analysis suffers from rigid temporal alignment and shallow modal interaction, making it difficult to achieve comprehensive and nuanced multidimensional sentiment perception.
Employing multi-scale time warping techniques and a four-layer progressive interaction network, this method processes multimodal data through dynamic alignment and hierarchical interaction networks. By combining deformable attention mechanisms and graph convolution operations, it achieves adaptive alignment and deep interaction between modalities.
It improves the accuracy and comprehensiveness of multimodal sentiment analysis, enabling a better understanding of complex emotional states, identification of consistency, complementarity, contradiction, and dependency, and providing more detailed sentiment analysis results.
Smart Images
Figure CN122045672A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of multimodal sentiment analysis, specifically a multimodal data analysis method and system based on artificial intelligence. Background Technology
[0002] The rise of multimodal sentiment analysis stems from the needs of real-world scenarios and the impetus of technological advancements. It aims to perceive and understand human emotions more comprehensively and nuancedly by integrating multi-dimensional information such as text, voice, and vision, thereby driving artificial intelligence towards a more humane and trustworthy direction. However, existing multimodal sentiment data analysis often suffers from problems such as rigidity in temporal alignment and superficiality in modal interaction. Summary of the Invention
[0003] To address the aforementioned issues and overcome the shortcomings of existing technologies, this invention designs an AI-based multimodal data analysis method. Addressing the rigidity of temporal alignment in multimodal sentiment analysis, this invention introduces a dynamic, multi-layered solution. First, a coarse-grained overall alignment is established using multi-scale time warping technology, analyzing the correspondence between modalities at different time scales. At a coarser time scale (e.g., sentences or complete expressions), the system identifies approximate alignment relationships; then, these alignments are refined at progressively finer scales (phrases, words, sub-words). This hierarchical approach mimics the natural human process of processing multimodal information—first grasping the overall structure, then focusing on detailed correspondences. Simultaneously, a deformable attention mechanism provides local, adaptive alignment capabilities. For each word in the text, the system predicts which time point of other modal information should be focused on based on semantic relevance. This prediction is not random but rather based on content learning results to guide which types of words should be associated with which acoustic features or visual changes. Addressing the shallowness of modal interaction in multimodal sentiment analysis, this invention constructs a four-layer progressive interaction network, capturing specific interactions through a bidirectional and content-aware word-level interaction layer. The system employs a comprehensive, localized cross-modal correspondence model—linking emotional words with specific intonation changes or visual features. A phrase-level interaction layer organizes related units into semantic groups, with connection strength determined by the underlying word-level interaction strength. Graph convolution operations facilitate information flow between semantically similar nodes, enhancing the system's ability to recognize different patterns. A sentence-level interaction layer introduces a collaborative attention mechanism to achieve bidirectional understanding between different modalities. A relational modeling layer identifies the types of relationships in multimodal data: consistent relationships (all modalities express the same emotion), complementary relationships (different modalities provide different aspects of emotion), contradictory relationships (intermodal expressions conflict), and dependent relationships (the emotional expression of one modality depends on the context of another modality). This relational awareness further facilitates the understanding and analysis of complex emotional states.
[0004] The multimodal data analysis method based on artificial intelligence provided by this invention specifically includes the following steps:
[0005] Step S1: Data Acquisition. Collect multimodal data related to human emotions, including audio, images, and text.
[0006] Step S2: Multimodal feature extraction. Different extraction methods are used to extract and align features for different modalities to obtain multimodal features. These multimodal features include word-level, phrase-level, and sentence-level alignment features, text features, audio features, and visual features. The multimodal feature extraction specifically includes the following steps:
[0007] Step S21: Multi-granularity encoding of text modalities. The text is encoded in multiple granularities to obtain text features, including word-level features, phrase-level features, and sentence-level features. Specifically, this includes the following steps:
[0008] Step S211: Word-level feature extraction, using a hierarchical BERT encoder to extract word-level features from the text;
[0009] Step S212: Phrase-level feature extraction, using learnable sliding window pooling for phrase-level feature extraction: ;
[0010] In the formula, Represents pooling operations, Represents word-level features, Represents window size. Represents phrase-level features;
[0011] Step S213: Sentence-level feature extraction. Use bidirectional LSTM to capture inter-sentence dependencies on phrase-level features to obtain sentence-level features;
[0012] Step S22: Joint spectral-semantic encoding of audio modalities. Fourier transform the audio features to generate a spectrogram. Convolutional feature extraction is performed on the spectrogram to obtain the spectral encoding. High-level semantics in the audio features are extracted through pre-trained Wav2Vec 2.0 to obtain semantic features. The spectral encoding and semantic features are gated and fused to obtain the audio features.
[0013] Step S23: Spatiotemporal hierarchical encoding of visual modalities. 3D-ResNet is used to extract spatiotemporal features of the image. A temporal difference network and attention aggregation are introduced to optimize the spatiotemporal features, resulting in visual features: ; ; ;
[0014] In the formula, Represents spatiotemporal characteristics, Represents a gated neural network. For intermediate parameters, Represents attention weight, Represents attention score, Represents the sequence length of the images. Represents visual characteristics;
[0015] Step S24: Dynamic temporal alignment. Dynamic temporal alignment is performed on text features to obtain word-level, phrase-level, and sentence-level alignment features. This includes the following steps:
[0016] Step S241: Design the cost matrix. By calculating the L2 norm between word-level features, phrase-level features, and sentence-level features, as well as the temporal penalty weights, a multi-scale dynamic time warping cost matrix is obtained. ;
[0017] In the formula, and It represents any two modalities among word-level features, phrase-level features, and sentence-level features. Representing the L2 paradigm, Represents the time-series penalty weight. This represents the time-series penalty, specifically the time-distance penalty. and These represent the temporal lengths of the two modal characteristics, respectively. Represents the cost matrix;
[0018] Step S242: Multi-scale path search. Perform multi-scale path search on the cost matrix, that is, find the optimal alignment path at three different granularities: word, phrase, and sentence. ;
[0019] In the formula, Representing the scale level, namely words, phrases, and sentences. Represents the optimal alignment path across multiple scales;
[0020] Step S243: Attention-guided alignment learning. Word-level, phrase-level, and sentence-level features are mutually aligned using a deformable attention mechanism to obtain aligned features at the word, phrase, and sentence levels. The calculation of mutual alignment is as follows: ; ; ;
[0021] In the formula, Represents attention score, This represents the query vector and key vector of the m-modality and the feature vector of the n-modality, computed through the attention mechanism. This represents the learnable alignment offset. Represents multilayer perceptron processing. This represents the feature obtained by aligning the m-mode to the n-mode;
[0022] Step S25: Multimodal feature construction, the set of word-level, phrase-level and sentence-level alignment features, audio features and visual features constitutes multimodal features;
[0023] Step S3: Hierarchical interactive network processing, which integrates word-level, phrase-level, and sentence-level alignment features with text features, audio features, and visual features through a four-layer hierarchical interaction and fusion process to obtain fused multimodal features. This includes the following steps:
[0024] Step S31: Word-level interaction. Through the cross-attention mechanism, the word-level features are unidirectionally interacted with the audio and visual features respectively to obtain word-level interaction features.
[0025] Step S32: Phrase-level interaction. Take the phrase-level alignment feature corresponding to each phrase in the text as a node, map the word-level interaction features proportionally, obtain the edge weights between nodes, construct a cross-modal graph through nodes and edge weights, and perform graph convolution processing on the cross-modal graph to obtain phrase-level interaction features.
[0026] Step S33: Sentence-level interaction. A collaborative attention mechanism is used to perform bidirectional interaction between sentence-level alignment features and audio features to obtain sentence-level interaction features.
[0027] Step S34: Cross-modal interaction. Using RelationNet, pairwise relationships are modeled between sentence-level alignment features, audio features, and visual features to obtain relationship tensors between the three modalities.
[0028] Step S35: Interaction path fusion. An adaptive path selection mechanism is used to fuse the interaction paths of the four layers to obtain cross-modal fused interaction features. ; ;
[0029] In the formula, Representing different characteristics Path weights between Represents the activation function. Represents the hierarchy of interaction. This represents the result obtained from the interaction at layer p. Represents cross-modal fusion interaction features;
[0030] Step S36: Feature fusion, which fuses text features, audio features, visual features, and cross-modal fusion interaction features to obtain fused multimodal features;
[0031] Step S4: Multimodal sentiment analysis, predicting outputs from the fused multimodal features to obtain the multimodal sentiment analysis results.
[0032] The present invention provides an artificial intelligence-based multimodal data analysis system, including a multimodal data acquisition module, a multimodal feature extraction module, a multimodal feature fusion module, and a multimodal sentiment analysis module;
[0033] The multimodal data acquisition module collects multimodal data related to human emotions, including audio, images, and text.
[0034] The multimodal feature extraction module uses different extraction methods to extract and align features for different modalities to obtain multimodal features, including word-level, phrase-level, and sentence-level alignment features, text features, audio features, and visual features.
[0035] The multimodal feature fusion module performs hierarchical interaction and fusion of word-level, phrase-level, and sentence-level alignment features with text features, audio features, and visual features in four layers to obtain fused multimodal features.
[0036] The multimodal sentiment analysis module processes the fused multimodal features using an activation function to obtain the predicted output value, which is the multimodal sentiment analysis result. A threshold is set for the predicted output value, and the predicted output value with different thresholds corresponds to different sentiment recognition results.
[0037] The beneficial results achieved by the present invention using the above solution are as follows:
[0038] (1) To address the rigidity of temporal alignment in multimodal sentiment analysis, this invention introduces a dynamic, multi-layered solution. It first establishes overall alignment at a coarse-grained level using multi-scale temporal warping technology, then analyzes the correspondence between modalities at different time scales. At a coarser time scale (e.g., sentences or complete expressions), the system identifies approximate alignment relationships; then, these alignments are refined at progressively finer scales (phrases, words, sub-words). This hierarchical approach mimics the natural process of human processing of multimodal information—first grasping the overall structure, then focusing on detailed correspondences. Simultaneously, a deformable attention mechanism provides local, adaptive alignment capabilities. For each word in the text, the system predicts which time point of other modal information should be focused on based on semantic relevance. This prediction is not random but rather based on the learning results of the content to guide which types of words should be associated with which acoustic features or visual changes.
[0039] (2) To address the shallowness of modal interaction in multimodal sentiment analysis, this invention constructs a four-layer progressive interaction network. Through a bidirectional and content-aware word-level interaction layer, specific and local cross-modal correspondences are captured—how a certain emotional word is associated with specific intonation changes or visual features. Through a phrase-level interaction layer, related units are organized into semantic groups, with the connection strength determined by the word-level interaction strength at the bottom layer. Graph convolution operations are used to allow information to flow between semantically similar nodes, improving the system's ability to recognize different patterns. Through a sentence-level interaction layer, a collaborative attention mechanism is introduced to achieve bidirectional understanding between different modalities. Through a relation modeling layer, the types of relationships in multimodal data are identified: consistency relationships (all modalities express the same emotion), complementary relationships (different modalities provide different aspects of emotion), contradictory relationships (conflicts in expression between modalities), and dependency relationships (the emotional expression of one modality depends on the context of another modality). Relational awareness is used to further understand and analyze complex emotional states. Attached Figure Description
[0040] Figure 1 A block diagram of the multimodal data analysis system based on artificial intelligence provided by this invention;
[0041] Figure 2 This is a flowchart illustrating a multimodal data analysis method based on artificial intelligence.
[0042] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used together with the embodiments of the invention to explain the invention and do not constitute a limitation thereof. Detailed Implementation
[0043] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.
[0044] Example 1, see Figure 2 The multimodal data analysis method based on artificial intelligence provided by this invention specifically includes the following steps:
[0045] Step S1: Data Acquisition. Collect multimodal data related to human emotions, including audio, images, and text.
[0046] Step S2: Multimodal feature extraction. For features of different modalities, different extraction methods are used to extract and align features to obtain multimodal features. The multimodal features include word-level, phrase-level and sentence-level alignment features, text features, audio features and visual features.
[0047] Step S3: Hierarchical interactive network processing, which performs hierarchical interaction and fusion of word-level, phrase-level and sentence-level alignment features, text features, audio features and visual features in 4 layers to obtain fused multimodal features;
[0048] Step S4: Multimodal sentiment analysis, predicting outputs from the fused multimodal features to obtain the multimodal sentiment analysis results.
[0049] Example 2, this example is based on the above example, step S2 specifically includes the following steps:
[0050] Step S21: Multi-granularity encoding of text modalities. Multi-granularity encoding is performed on the text to obtain text features, which include word-level features, phrase-level features, and sentence-level features.
[0051] Step S22: Joint spectral-semantic encoding of audio modalities. Fourier transform the audio features to generate a spectrogram. Convolutional feature extraction is performed on the spectrogram to obtain the spectral encoding. High-level semantics in the audio features are extracted through pre-trained Wav2Vec 2.0 to obtain semantic features. The spectral encoding and semantic features are gated and fused to obtain the audio features.
[0052] Step S23: Spatiotemporal hierarchical coding of visual modalities. 3D-ResNet is used to extract spatiotemporal features of the image. Temporal difference network and attention aggregation are introduced to optimize the spatiotemporal features to obtain visual features.
[0053] Step S24: Dynamic temporal alignment. Perform dynamic temporal alignment on the text features to obtain alignment features at the word, phrase, and sentence levels.
[0054] Example 3, this example is based on the above example, step S21 specifically includes the following steps:
[0055] Step S211: Word-level feature extraction, using a hierarchical BERT encoder to extract word-level features from the text;
[0056] Step S212: Phrase-level feature extraction, using learnable sliding window pooling for phrase-level feature extraction: ;
[0057] In the formula, Represents pooling operations, Represents word-level features, Represents window size. Represents phrase-level features;
[0058] Step S213: Sentence-level feature extraction. Use bidirectional LSTM to capture inter-sentence dependencies on phrase-level features to obtain sentence-level features.
[0059] Example 4, this example is based on the above example, step S24 specifically includes the following steps:
[0060] Step S241: Design the cost matrix. By calculating the L2 norm between word-level features, phrase-level features, and sentence-level features, as well as the temporal penalty weights, a multi-scale dynamic time warping cost matrix is obtained. ;
[0061] In the formula, and It represents any two modalities among word-level features, phrase-level features, and sentence-level features. Representing the L2 paradigm, Represents the time-series penalty weight. This represents the time-series penalty, specifically the time-distance penalty. and These represent the temporal lengths of the two modal characteristics, respectively. Represents the cost matrix;
[0062] Step S242: Multi-scale path search. Perform multi-scale path search on the cost matrix, that is, find the optimal alignment path at three different granularities: word, phrase, and sentence. ;
[0063] In the formula, Representing the scale level, namely words, phrases, and sentences. Represents the optimal alignment path across multiple scales;
[0064] Step S243: Attention-guided alignment learning. Word-level, phrase-level, and sentence-level features are mutually aligned using a deformable attention mechanism to obtain aligned features at the word, phrase, and sentence levels. The calculation of mutual alignment is as follows: ; ; ;
[0065] In the formula, Represents attention score, This represents the query vector and key vector of the m-modality and the feature vector of the n-modality, computed through the attention mechanism. This represents the learnable alignment offset. Represents multilayer perceptron processing. This represents the feature obtained by aligning the m-mode to the n-mode.
[0066] Example 5, this example is based on the above examples, step S3 specifically includes the following steps:
[0067] Step S31: Word-level interaction. Through the cross-attention mechanism, the word-level features are unidirectionally interacted with the audio and visual features respectively to obtain word-level interaction features.
[0068] Step S32: Phrase-level interaction. Take the phrase-level alignment feature corresponding to each phrase in the text as a node, map the word-level interaction features proportionally, obtain the edge weights between nodes, construct a cross-modal graph through nodes and edge weights, and perform graph convolution processing on the cross-modal graph to obtain phrase-level interaction features.
[0069] Step S33: Sentence-level interaction. A collaborative attention mechanism is used to perform bidirectional interaction between sentence-level alignment features and audio features to obtain sentence-level interaction features.
[0070] Step S34: Cross-modal interaction. Using RelationNet, pairwise relationships are modeled between sentence-level alignment features, audio features, and visual features to obtain relationship tensors between the three modalities.
[0071] Step S35: Interaction path fusion. An adaptive path selection mechanism is used to fuse the interaction paths of the four layers to obtain cross-modal fused interaction features. ; ;
[0072] In the formula, Representing different characteristics Path weights between Represents the activation function. Represents the hierarchy of interaction. This represents the result obtained from the interaction at layer p. Represents cross-modal fusion interaction features;
[0073] Step S36: Feature fusion, which fuses text features, audio features, visual features, and cross-modal fusion interaction features to obtain fused multimodal features.
[0074] Example 6, see Figure 1 This embodiment is based on the above embodiment and is an artificial intelligence-based multimodal data analysis system, including a multimodal data acquisition module, a multimodal feature extraction module, a multimodal feature fusion module, and a multimodal sentiment analysis module;
[0075] The multimodal data acquisition module collects multimodal data related to human emotions, including audio, images, and text.
[0076] The multimodal feature extraction module uses different extraction methods to extract and align features for different modalities to obtain multimodal features, including word-level, phrase-level, and sentence-level alignment features, text features, audio features, and visual features.
[0077] The multimodal feature fusion module performs hierarchical interaction and fusion of word-level, phrase-level, and sentence-level alignment features with text features, audio features, and visual features in four layers to obtain fused multimodal features.
[0078] The multimodal sentiment analysis module processes the fused multimodal features using an activation function to obtain the predicted output value, which is the multimodal sentiment analysis result. A threshold is set for the predicted output value, and the predicted output value with different thresholds corresponds to different sentiment recognition results.
[0079] Example 7: The above solution was applied to a student emotion monitoring system on an online education platform, and the results are as follows:
[0080] Application scenarios:
[0081] In online classrooms, teachers have difficulty perceiving students' emotional state in real time, which affects the effectiveness of interaction and adjustments to teaching.
[0082] Implementation method:
[0083] The system collects students' audio (voice answers / questions), video (facial expressions / postures), and text (chat messages) data in real time;
[0084] System operation process:
[0085] Multimodal feature extraction:
[0086] Text: Using BERT to extract sentiment keywords from student questions;
[0087] Audio: Intonation and speaking speed were analyzed using Wav2Vec 2.0;
[0088] Visual: Capture changes in students' facial expressions (such as confusion, focus, fatigue) using 3D-ResNet;
[0089] Dynamic timing alignment:
[0090] Align emotional words in speech with facial expression changes on multiple scales, such as the temporal correlation between the word "cannot understand" and the frowning expression.
[0091] Hierarchical interaction analysis:
[0092] Identifying sentiment patterns through a four-layer interactive network:
[0093] Consistency: Low voice + frustrated expression → confirms the student is in low spirits;
[0094] Contradiction: The student's voice was positive but their expression was tired → The student was forcing themselves to stay alert;
[0095] Output and feedback:
[0096] The system generates real-time emotion analysis reports, which can prompt teachers to adjust their teaching pace or provide individualized support.
[0097] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus.
[0098] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.
[0099] The present invention and its embodiments have been described above. This description is not restrictive, and the accompanying drawings are only one embodiment of the present invention; the actual structure is not limited thereto. In conclusion, if those skilled in the art are inspired by this description and design similar structures and embodiments without departing from the spirit of the invention, such designs should fall within the protection scope of the present invention.
Claims
1. A multimodal data analysis method based on artificial intelligence, characterized in that: Specifically, the following steps are included: Step S1: Data Acquisition. Collect multimodal data related to human emotions, including audio, images, and text. Step S2: Multimodal feature extraction. For features of different modalities, different extraction methods are used to extract and align features to obtain multimodal features. The multimodal features include word-level, phrase-level and sentence-level alignment features, text features, audio features and visual features. Step S3: Hierarchical interactive network processing, which performs hierarchical interaction and fusion of word-level, phrase-level and sentence-level alignment features, text features, audio features and visual features in 4 layers to obtain fused multimodal features; Step S4: Multimodal sentiment analysis, predicting outputs from the fused multimodal features to obtain the multimodal sentiment analysis results.
2. The multimodal data analysis method based on artificial intelligence according to claim 1, characterized in that: Step S2 specifically includes the following steps: Step S21: Multi-granularity encoding of text modalities. Multi-granularity encoding is performed on the text to obtain text features, which include word-level features, phrase-level features, and sentence-level features. Step S22: Joint spectral-semantic encoding of audio modalities. Fourier transform the audio features to generate a spectrogram. Convolutional feature extraction is performed on the spectrogram to obtain the spectral encoding. High-level semantics in the audio features are extracted through pre-trained Wav2Vec 2.0 to obtain semantic features. The spectral encoding and semantic features are gated and fused to obtain the audio features. Step S23: Spatiotemporal hierarchical coding of visual modalities. 3D-ResNet is used to extract spatiotemporal features of the image. Temporal difference network and attention aggregation are introduced to optimize the spatiotemporal features to obtain visual features. Step S24: Dynamic temporal alignment. Perform dynamic temporal alignment on the text features to obtain alignment features at the word, phrase, and sentence levels.
3. The multimodal data analysis method based on artificial intelligence according to claim 2, characterized in that: Step S24 specifically includes the following steps: Step S241: Design the cost matrix. By calculating the L2 norm between word-level features, phrase-level features, and sentence-level features, as well as the temporal penalty weights, a multi-scale dynamic time warping cost matrix is obtained. ； In the formula, and It represents any two modalities among word-level features, phrase-level features, and sentence-level features. Representing the L2 paradigm, Represents the time-series penalty weight. This represents the time-series penalty, specifically the time-distance penalty. and These represent the temporal lengths of the two modal characteristics, respectively. Represents the cost matrix; Step S242: Multi-scale path search, performing multi-scale path search on the cost matrix, that is, finding the optimal alignment path at three different granularities: word, phrase, and sentence. Step S243: Attention-guided alignment learning. Word-level, phrase-level, and sentence-level features are mutually aligned using a deformable attention mechanism to obtain aligned features at the word, phrase, and sentence levels. The calculation of mutual alignment is as follows: ； ； ； In the formula, Represents attention score, This represents the query vector and key vector of the m-modality and the feature vector of the n-modality, computed through the attention mechanism. This represents the learnable alignment offset. Represents multilayer perceptron processing. This represents the feature obtained by aligning the m-mode to the n-mode.
4. An artificial intelligence-based multimodal data analysis system, used to implement the artificial intelligence-based multimodal data analysis method according to any one of claims 1 to 3, characterized in that: It includes a multimodal data acquisition module, a multimodal feature extraction module, a multimodal feature fusion module, and a multimodal sentiment analysis module; The multimodal data acquisition module collects multimodal data related to human emotions, including audio, images, and text. The multimodal feature extraction module uses different extraction methods to extract and align features for different modalities to obtain multimodal features, including word-level, phrase-level, and sentence-level alignment features, text features, audio features, and visual features. The multimodal feature fusion module performs hierarchical interaction and fusion of word-level, phrase-level, and sentence-level alignment features with text features, audio features, and visual features in four layers to obtain fused multimodal features. The multimodal sentiment analysis module processes the fused multimodal features using an activation function to obtain the predicted output value, which is the multimodal sentiment analysis result. A threshold is set for the predicted output value, and the predicted output value with different thresholds corresponds to different sentiment recognition results.