Multi-modal session data analysis method and system based on time axis fusion analysis

Abstract

The invention relates to the technical field of session data analysis, and discloses a time axis fusion analysis-based multi-modal session data analysis method and system, and the method comprises the steps: obtaining multi-modal session data, and enabling each modal session data in the multi-modal session data to have a corresponding modal session time axis; according to a preset anti-interference fitting algorithm, all modal session time axes are calibrated, and a unified wall clock time axis of the multi-modal session data is generated; identifying key information nodes in the multi-modal session data based on a unified wall clock time axis, and calculating an emotion vector corresponding to each moment in the multi-modal session data to obtain an emotion intensity curve of the multi-modal session data; and according to all the key information nodes and the emotional intensity curve, generating an interactive emotional map of the multi-modal session data. Therefore, by implementing the multi-modal session data analysis method, the traceability of the analysis result can be improved while the analysis accuracy and objectivity of the multi-modal session data are improved.

Description

Technical Field

[0001] This invention relates to the field of conversation data analysis technology, and in particular to a multimodal conversation data analysis method and system based on time-axis fusion analysis. Background Technology

[0002] In the current field of conversation data analysis, especially in scenarios such as user interviews, market research, and customer service quality inspection, it is usually necessary to rely on analysts to manually process conversation records. Traditional methods mainly focus on the qualitative interpretation of text content. By using speech-to-text technology to obtain text, analysts need to read the transcribed text line by line, qualitatively interpret key information from the text based on personal experience, and subjectively describe the emotional fluctuations during the conversation.

[0003] However, practice has revealed several significant drawbacks to this method: First, manual annotation is inefficient and struggles to handle massive amounts of data; second, differing subjective judgment standards among analysts lead to ambiguous definitions of key information, poor consistency and repeatability of analysis results, and an inability to effectively compare different projects; third, sentiment analysis is mostly fragmented and qualitative descriptions, lacking quantitative expression over a continuous timeline, making it difficult to accurately track the trajectory and magnitude of sentiment changes and their dynamic correlation with key events. Therefore, it is particularly important to propose a technical solution that improves the accuracy and objectivity of multimodal conversation data analysis while also enhancing the traceability of the analysis results. Summary of the Invention

[0004] This invention provides a method and system for analyzing multimodal conversation data based on time-axis fusion analysis, which can improve the accuracy and objectivity of multimodal conversation data analysis while enhancing the traceability of analysis results.

[0005] To address the aforementioned technical problems, the first aspect of this invention discloses a multimodal conversation data analysis method based on time-axis fusion analysis, the method comprising: Acquire multimodal session data, wherein each modal session data has a corresponding modal session timeline; According to the preset anti-interference fitting algorithm, all the modal session time axes are calibrated to generate a unified wall clock time axis for the multimodal session data; Based on the unified wall clock timeline, key information nodes in the multimodal session data are identified, and each key information node has a corresponding feature identifier. Based on the unified wall clock time axis, the emotion vector corresponding to each moment in the multimodal conversation data is calculated to obtain the emotion intensity curve of the multimodal conversation data; Based on all the key information nodes and the sentiment intensity curve, an interactive sentiment map of the multimodal conversation data is generated. The interactive sentiment map includes the sentiment change trend and key information distribution of the multimodal conversation data.

[0006] As an optional implementation, in the first aspect of the present invention, the step of calibrating all the modal session timelines according to a preset anti-interference fitting algorithm to generate a unified wall clock timeline for the multimodal session data includes: For each modal session data, extract the set of event anchor points for that modal session data; Match the event anchor point sets of all the modal session data to obtain a set of associated event anchor point pairs between different modal session data; Based on the set of associated event anchor points and the preset anti-interference fitting algorithm, a global mapping relationship between the time axis of all modal sessions in the multimodal session data and the time axis of the standard reference wall clock is generated; Based on the global mapping relationship, a unified wall clock timeline for the multimodal session data is generated.

[0007] As an optional implementation, in the first aspect of the invention, before generating the unified wall clock timeline of the multimodal session data according to the global mapping relationship, the method further includes: Based on the global mapping relationship, calculate the fitting residual for each associated event anchor pair in the associated event anchor pair set; Analyze the distribution characteristics of the fitting residuals based on all the aforementioned fitting residuals; Determine whether there are any feature mutation target points in the fitted residual distribution features; If it exists, then based on the feature mutation target point, determine the calibration benchmark modal session time axis in all modal session time axes; based on the feature mutation point, divide the calibration benchmark modal session time axis into a corresponding number of time periods; for each time period, based on all the associated event anchor points contained in the time period and the preset anti-interference fitting algorithm, generate the local mapping relationship corresponding to the time period; Based on all the aforementioned local mapping relationships, a unified wall clock timeline for the multimodal session data is generated; If it does not exist, the operation of generating a unified wall clock time axis for the multimodal session data based on the global mapping relationship is triggered.

[0008] As an optional implementation, in the first aspect of the present invention, identifying key information nodes in the multimodal session data based on the unified wall clock timeline includes: Based on the unified wall clock timeline, semantic sentence sequences are extracted from the multimodal conversation data; For each semantic sentence in the semantic sentence sequence, extract the text content features and context association features of that semantic sentence; Based on the text content features and the context association features, the key information nodes of the semantic sentence are matched.

[0009] As an optional implementation, in the first aspect of the present invention, the step of calculating the sentiment vector corresponding to each moment in the multimodal conversation data based on the unified wall clock time axis to obtain the sentiment intensity curve of the multimodal conversation data includes: For each semantic sentence, calculate the sentiment vector of the semantic sentence, and the sentiment vector is used to represent the intensity distribution of the corresponding semantic sentence on different sentiment dimensions; The sentiment vector of each semantic clause is mapped to the time corresponding to that semantic clause on the unified wall clock time axis to generate a discrete sentiment vector sequence; The discrete sentiment vector sequence is subjected to time series smoothing to generate the sentiment intensity curve of the multimodal conversation data.

[0010] As an optional implementation, in a first aspect of the invention, generating an interactive sentiment map of the multimodal conversation data based on all the key information nodes and the sentiment intensity curve includes: For each key information node, in the unified wall clock timeline, the key information node is matched with the emotion vector of the emotion intensity curve at the same time. Based on the contextual association features of the key information node, the contextual consistency measure corresponding to the key information node is calculated. The contextual consistency measure is used to represent the degree of consistency between the semantic clause corresponding to the key information node and the neighboring semantic clauses in terms of topic. Based on the feature identifier of the key information node, the corresponding matched sentiment vector, and the corresponding context consistency measure, the fusion priority value of the key information node is calculated. The fusion priority value is used to represent the importance of the corresponding key information node in the multimodal conversation data. Based on the fusion priority value of all the key information nodes, a corresponding interactive marker is generated on the time axis of the emotion intensity curve. Each interactive marker is associated with the feature identifier of the corresponding key information node, and the interactive marker is used as an interactive interface to trigger access to the corresponding key information node. The emotional intensity curve with the interactive markers is integrated with the feature identifiers of all the key information nodes to generate an interactive emotional map of the multimodal conversation data.

[0011] A second aspect of this invention discloses a multimodal conversation data analysis system based on time-axis fusion analysis, the system comprising: The acquisition module is used to acquire multimodal session data, wherein each modal session data in the multimodal session data has a corresponding modal session timeline; The calibration module is used to calibrate all the modal session timelines according to a preset anti-interference fitting algorithm, and generate a unified wall clock timeline for the multimodal session data. The identification module is used to identify key information nodes in the multimodal session data based on the unified wall clock timeline, and each key information node has a corresponding feature identifier; The calculation module is used to calculate the sentiment vector corresponding to each moment in the multimodal conversation data based on the unified wall clock time axis, so as to obtain the sentiment intensity curve of the multimodal conversation data. The generation module is used to generate an interactive sentiment map of the multimodal conversation data based on all the key information nodes and the sentiment intensity curve. The interactive sentiment map includes the sentiment change trend and key information distribution of the multimodal conversation data.

[0012] As an optional implementation, in the second aspect of the present invention, the calibration module calibrates all the modal session timelines according to a preset anti-interference fitting algorithm, and the specific method for generating a unified wall clock timeline for the multimodal session data includes: For each modal session data, extract the set of event anchor points for that modal session data; Match the event anchor point sets of all the modal session data to obtain a set of associated event anchor point pairs between different modal session data; Based on the set of associated event anchor points and the preset anti-interference fitting algorithm, a global mapping relationship between the time axis of all modal sessions in the multimodal session data and the time axis of the standard reference wall clock is generated; Based on the global mapping relationship, a unified wall clock timeline for the multimodal session data is generated.

[0013] As an optional implementation, in a second aspect of the invention, the calculation module is further configured to calculate the fitting residual of each associated event anchor pair in the associated event anchor pair set according to the global mapping relationship before the generation module generates the unified wall clock time axis of the multimodal session data according to the global mapping relationship. The system also includes: The analysis module is used to analyze the distribution characteristics of the fitting residuals based on all the said fitting residuals; The judgment module is used to determine whether there are feature mutation target points in the fitting residual distribution features; A determination module is used to determine, if present, a calibration reference modal session time axis based on the characteristic mutation target point across all modal session time axes; The segmentation module is used to divide the calibration reference modal session time axis into a corresponding number of time periods based on the characteristic mutation points; The generation module is further configured to generate a local mapping relationship for each time period based on all the associated event anchors contained in the time period and the preset anti-interference fitting algorithm. The generation module is also used to generate a unified wall clock timeline for the multimodal session data based on all the local mapping relationships. If it does not exist, the generation module is triggered to perform the operation of generating a unified wall clock time axis for the multimodal session data based on the global mapping relationship.

[0014] As an optional implementation, in a second aspect of the present invention, the specific method by which the identification module identifies key information nodes in the multimodal session data based on the unified wall clock timeline includes: Based on the unified wall clock timeline, semantic sentence sequences are extracted from the multimodal conversation data; For each semantic sentence in the semantic sentence sequence, extract the text content features and context association features of that semantic sentence; Based on the text content features and the context association features, the key information nodes of the semantic sentence are matched.

[0015] As an optional implementation, in a second aspect of the present invention, the specific method by which the calculation module calculates the emotion vector corresponding to each moment in the multimodal conversation data based on the unified wall clock time axis to obtain the emotion intensity curve of the multimodal conversation data includes: For each semantic sentence, calculate the sentiment vector of the semantic sentence, and the sentiment vector is used to represent the intensity distribution of the corresponding semantic sentence on different sentiment dimensions; The sentiment vector of each semantic clause is mapped to the time corresponding to that semantic clause on the unified wall clock time axis to generate a discrete sentiment vector sequence; The discrete sentiment vector sequence is subjected to time series smoothing to generate the sentiment intensity curve of the multimodal conversation data.

[0016] As an optional implementation, in a second aspect of the invention, the specific method by which the generation module generates the interactive sentiment map of the multimodal conversation data based on all the key information nodes and the sentiment intensity curve includes: For each key information node, in the unified wall clock timeline, the key information node is matched with the emotion vector of the emotion intensity curve at the same time. Based on the contextual association features of the key information node, the contextual consistency measure corresponding to the key information node is calculated. The contextual consistency measure is used to represent the degree of consistency between the semantic clause corresponding to the key information node and the neighboring semantic clauses in terms of topic. Based on the feature identifier of the key information node, the corresponding matched sentiment vector, and the corresponding context consistency measure, the fusion priority value of the key information node is calculated. The fusion priority value is used to represent the importance of the corresponding key information node in the multimodal conversation data. Based on the fusion priority value of all the key information nodes, a corresponding interactive marker is generated on the time axis of the emotion intensity curve. Each interactive marker is associated with the feature identifier of the corresponding key information node, and the interactive marker is used as an interactive interface to trigger access to the corresponding key information node. The emotional intensity curve with the interactive markers is integrated with the feature identifiers of all the key information nodes to generate an interactive emotional map of the multimodal conversation data.

[0017] A third aspect of this invention discloses another multimodal conversation data analysis system based on time-axis fusion analysis, the system comprising: Memory containing executable program code; A processor coupled to the memory; The processor calls the executable program code stored in the memory to execute the multimodal session data analysis method based on time axis fusion analysis disclosed in the first aspect of the present invention.

[0018] The fourth aspect of the present invention discloses a computer storage medium storing computer instructions, which, when invoked, are used to execute the multimodal session data analysis method based on time axis fusion analysis disclosed in the first aspect of the present invention.

[0019] Compared with the prior art, the embodiments of the present invention have the following beneficial effects: In this embodiment of the invention, multimodal conversation data is acquired, with each modal conversation having a corresponding modal conversation timeline. All modal conversation timelines are calibrated using a preset anti-interference fitting algorithm to generate a unified wall clock timeline for the multimodal conversation data. Based on the unified wall clock timeline, key information nodes in the multimodal conversation data are identified, with each key information node having a corresponding feature identifier. Based on the unified wall clock timeline, the sentiment vector corresponding to each moment in the multimodal conversation data is calculated to obtain the sentiment intensity curve of the multimodal conversation data. Based on all key information nodes and the sentiment intensity curve, an interactive sentiment map of the multimodal conversation data is generated, including the sentiment change trend and key information distribution of the multimodal conversation data. It is evident that implementing this invention, by "calibrating all modal conversation timelines using a preset anti-interference fitting algorithm to generate a unified wall clock timeline," can improve the consistency of multi-source heterogeneous time data, thereby facilitating the establishment of a reliable time benchmark for subsequent analysis. This, in turn, helps ensure the accuracy of key event identification and sentiment analysis results in the time dimension, achieving precise temporal modeling of the conversation process. By "identifying key information nodes and calculating emotional intensity curves based on a unified wall clock timeline," the structure and quantification of conversation content analysis can be improved. This facilitates the systematic capture of core events and emotional fluctuations within the conversation, overcoming the subjectivity and bias of manual analysis and achieving an objective and comprehensive understanding of the conversation content. By "generating an interactive emotional map based on all key information nodes and emotional intensity curves," the visualization and relevance of the analysis results can be enhanced. This allows for the intuitive presentation of the spatiotemporal coupling relationship between key information and emotional changes, supporting users in conducting exploratory data analysis and in-depth evidence tracing. It enables an integrated analysis experience from macro trends to micro evidence. In summary, this invention improves the accuracy and objectivity of multimodal conversation data analysis while enhancing the traceability of the analysis results. Attached Figure Description

[0020] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0021] Figure 1 This is a flowchart illustrating a multimodal conversation data analysis method based on time-axis fusion analysis disclosed in an embodiment of the present invention; Figure 2 This is a flowchart illustrating another multimodal conversation data analysis method based on time-axis fusion analysis disclosed in an embodiment of the present invention; Figure 3This is a schematic diagram of the structure of a multimodal conversation data analysis system based on time axis fusion analysis disclosed in an embodiment of the present invention; Figure 4 This is a schematic diagram of another multimodal conversation data analysis system based on time axis fusion analysis disclosed in an embodiment of the present invention; Figure 5 This is a schematic diagram of the structure of another multimodal conversation data analysis system based on time axis fusion analysis disclosed in an embodiment of the present invention. Detailed Implementation

[0022] To enable those skilled in the art to better understand the present invention, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0023] The terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this invention are used to distinguish different objects, not to describe a specific order. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or end that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or ends.

[0024] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of the invention. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0025] This invention discloses a multimodal conversation data analysis method and system based on timeline fusion analysis. By "calibrating the timelines of all modal conversations according to a preset anti-interference fitting algorithm to generate a unified wall clock timeline," the consistency of multi-source heterogeneous time data can be improved. This facilitates the establishment of a reliable time benchmark for subsequent analysis, ensuring the accuracy of key event identification and sentiment analysis results in the time dimension, and achieving precise temporal modeling of the conversation process. By "identifying key information nodes and calculating sentiment intensity curves based on the unified wall clock timeline," the structure and quantification of conversation content analysis can be improved. This allows for the systematic capture of core events and emotional fluctuations in the conversation, overcoming the subjectivity and bias of manual analysis and achieving an objective and comprehensive insight into the conversation content. By generating an interactive sentiment map based on all key information nodes and sentiment intensity curves, the visualization and relevance of the analysis results can be improved. This facilitates the intuitive presentation of the spatiotemporal coupling relationship between key information and sentiment changes, thereby supporting users in conducting exploratory data analysis and in-depth evidence tracing. It achieves an integrated analysis experience from macro trends to micro evidence. In summary, this invention can improve the accuracy and objectivity of multimodal conversation data analysis while enhancing the traceability of the analysis results. These will be described in detail below.

[0026] Example 1 Please see Figure 1 , Figure 1 This is a flowchart illustrating a multimodal conversation data analysis method based on time-axis fusion analysis disclosed in an embodiment of the present invention. Figure 1 The described multimodal conversation data analysis method based on time-axis fusion analysis can be applied to interview / conversation devices, and also to smart devices associated with these devices. These smart devices include, but are not limited to, one or more of the following: switching devices, mobile terminal devices, cloud devices, edge computing devices, relay devices, base station devices, urban management devices, and intelligent connected devices. This invention does not limit the scope of these applications. Figure 1 As shown, this multimodal session data analysis method based on time-axis fusion analysis can include the following operations: 101. Obtain multimodal session data. Each modal session data has a corresponding modal session timeline. In this embodiment of the invention, optionally, the multimodal conversation data may include at least an audio stream obtained from interview recordings, and transcribed text or manually edited transcript text obtained through automatic speech recognition (ASR). The audio stream may have an audio timeline based on the recording device's clock, and the text data may have a text timeline based on the transcription or recording process. Furthermore, it may include device log events or front-end operation tracking data synchronized with the conversation process, all of which have their own timestamps / timelines.

[0027] 102. Based on the preset anti-interference fitting algorithm, calibrate the timelines of all modal conversations to generate a unified wall clock timeline for multimodal conversation data; In this embodiment of the invention, optionally, a preset anti-interference fitting algorithm can be used to process the aforementioned multimodal data. The core is to solve the problem of global offset and nonlinear drift between different modal time axes. Through this algorithm, one or more mapping functions are established to uniformly calibrate the original timestamps of modalities such as audio and text to a unified wall clock time axis with the project start time or a certain absolute time as the zero point, and the unit of this time axis is milliseconds.

[0028] In this embodiment of the invention, as an optional implementation, the above-mentioned method of calibrating all modal session timelines according to a preset anti-interference fitting algorithm to generate a unified wall clock timeline for multimodal session data includes: For each modal session data, extract the set of event anchor points for that modal session data; Match the event anchor sets of all modal session data to obtain a set of associated event anchor pairs between different modal session data; Based on the set of associated event anchor points and a preset anti-interference fitting algorithm, a global mapping relationship is generated between the time axis of all modal sessions in the multimodal session data and the time axis of the standard reference wall clock. A unified wall clock timeline for multimodal session data is generated based on the global mapping relationship.

[0029] In this embodiment of the invention, optionally, for extracting the set of event anchors: acoustic event anchors can be extracted from the audio stream. Specific methods include: detecting significant acoustic events in the audio such as non-speech segment pauses (duration exceeding a threshold, such as 500 milliseconds), laughter, sighs, etc., as well as word-level or clause-level boundary time points provided by the ASR system. Each anchor includes its type and a timestamp on the modal time axis. Further, optionally, semantic event anchors can be extracted from the text. Specific methods include: using rules or models to match fixed interjections (such as "um," "ah"), the beginning of host questions (such as "please," "could you," etc.), and salient event descriptions identified from the text (such as "[laughter]," "[applause]"). Each anchor also includes its type and position information on the text's timeline. Alternatively, for the set of associated event anchor pairs obtained through matching, the anchors on both the audio and text sides can be matched. The matching principle is temporal proximity, that is, for each acoustic event anchor, a semantic event anchor with the closest timestamp and corresponding semantic type is found on the text side to form an associated event anchor pair. For example, the laughter at 10 seconds in the audio is paired with an anchor in the text that describes "[laughter]" and is located nearby. Further, optionally, for generating a global mapping relationship: the set of successfully matched anchor pairs is treated as a set of sample points (audio timeline t_audio, text timeline t_text). Optionally, a comprehensive confidence weight is calculated for each associated event anchor pair, which can be based on a fusion of the confidence of acoustic event detection, the confidence of automatic speech recognition (ASR) word-level alignment, and the confidence of text-side rule matching. In the subsequent robust fitting algorithm, this weight will be used to weight anchor pairs of different qualities to reduce the negative impact of low-confidence anchors on the fitting results. Optionally, a robust linear regression algorithm (such as Huber regression) is used for fitting. The core of this algorithm is: based on standard linear regression (the goal is to min Σ(t_text - (a * t_audio + b))^2), the weight of sample points with excessive residuals (i.e., mismatched anchor pairs) is reduced, rather than completely removed, to fit a more robust global linear mapping relationship t_wall = a * t_audio + b. Here, t_wall refers to the standard reference wall clock time axis, which is an ideal, drift-free reference benchmark, and a and b are fitting parameters. Optionally, the above-described anti-interference fitting process may further include an outlier anchor point suppression step. After the initial fitting, the median absolute deviation (MAD) of the residuals of all anchor point pairs is calculated, and anchor point pairs with abnormally large residuals (e.g., exceeding the median ± 3 times the MAD) are identified as outliers and removed. The remaining anchor point pairs are then used for refitting to further improve calibration robustness.

[0030] Alternatively, for generating a unified wall clock timeline: the mapping relationship t_wall = a * t_audio + b obtained above can be applied to map each point on the audio timeline to the wall clock timeline. For timestamps of text and other modalities, they are indirectly unified to the wall clock timeline through their synchronization relationship with the audio timeline, ultimately obtaining a unified wall clock timeline for the entire session data.

[0031] As can be seen, implementing this optional embodiment, by "extracting and matching event anchor point sets from different modalities to obtain a set of associated event anchor point pairs," can improve the feasibility and accuracy of cross-modal time alignment. This is beneficial for providing high-quality, interpretable correspondence data for calibration, and thus lays the foundation for building robust mapping relationships, enabling targeted, data-driven timestamp calibration. By "generating a global mapping relationship based on the set of associated event anchor point pairs and a preset anti-interference fitting algorithm," the negative impact of mismatched anchor points on calibration results can be reduced. This is beneficial for improving the robustness and reliability of time axis calibration, and thus enabling the acquisition of an accurate unified time axis even in the presence of noisy data, achieving highly robust multimodal time synchronization.

[0032] In this optional embodiment, as an optional implementation, before generating the unified wall clock timeline of multimodal session data based on the global mapping relationship, the method further includes: Based on the global mapping relationship, calculate the fitting residual for each associated event anchor pair in the associated event anchor pair set; Analyze the distribution characteristics of the fitting residuals based on all the fitting residuals; Determine whether there are characteristic abrupt target points in the feature distribution characteristics of the fitted residuals; If it exists, then based on the feature mutation target point, determine the calibration benchmark modal session time axis on all modal session time axes; based on the feature mutation point, divide the calibration benchmark modal session time axis into a corresponding number of time periods; for each time period, based on all the associated event anchor points contained in that time period and the preset anti-interference fitting algorithm, generate the local mapping relationship corresponding to that time period; Based on all local mapping relationships, a unified wall clock timeline for multimodal session data is generated; If it does not exist, the operation of generating a unified wall clock time axis based on the global mapping relationship is triggered.

[0033] In this embodiment of the invention, optionally, for calculating the fitting residuals and analyzing the distribution: after obtaining the global linear mapping, the fitting residual for each anchor pair (i.e., t_text - (a * t_audio + b)) can be calculated. The distribution of all residuals is then analyzed. If the residuals exhibit a clear, irregular diffusion trend over time, rather than a random distribution, it suggests the presence of nonlinear drift. Further, optionally, to determine whether there are characteristic mutation target points, a change point detection algorithm (such as the PELT algorithm) can be used to analyze the residual sequence. This algorithm automatically finds points in the residual sequence where the mean and variance undergo significant changes; these points are the characteristic mutation target points, indicating that the time drift pattern may have changed. Further optionally, for segmented calibration: if a characteristic mutation target point is detected, segmentation processing is performed. Specifically, for determining the calibration baseline: the audio timeline is typically chosen as the calibration baseline modal session timeline because it is continuous and the most original time source. For dividing time segments: based on the mutation point, the audio timeline is divided into multiple consecutive time segments. For generating local mapping relationships: within each time segment, only anchor pairs falling within that time segment are used, and the aforementioned anti-interference fitting algorithm is reused to calculate the independent local linear mapping relationship t_wall = a_k * t_audio + b_k for that time segment, where a_k and b_k correspond to a and b as mentioned above. Alternatively, for generating a unified timeline: Finally, using the obtained segmented mapping sequence, the audio times for different time periods are mapped to the wall clock times, thereby generating a high-precision unified wall clock timeline that overcomes nonlinear drift. If no significant abrupt changes are detected, the aforementioned global mapping relationship is directly applied.

[0034] As a further optional dynamic calibration strategy, the system can also support online calibration. That is, after generating a unified wall clock timeline, during long sessions, at fixed intervals (e.g., every 5 minutes), the time mapping parameters are locally re-estimated using newly generated associated event anchor pairs, and smoothly transitioned with the parameters of the previous segment, in order to continuously suppress slow time drift caused by factors such as device temperature drift.

[0035] As can be seen, implementing this optional embodiment, by "calculating the fitting residuals, analyzing their distribution characteristics, and determining whether there are characteristic mutation target points," can improve the detection capability of complex time drift patterns. This facilitates the adaptive identification of nonlinear distortions that simple linear models cannot handle, and further helps to dynamically determine the necessity of segmented calibration, enabling intelligent selection of calibration strategies. By "determining the calibration reference time axis and performing segmented fitting when mutation points exist," the fitting accuracy of nonlinear time drift can be improved. This helps to effectively eliminate accumulated errors caused by equipment temperature drift, sampling rate differences, etc., and thus helps to maintain high-precision time alignment throughout long-term sessions, achieving strong adaptability to complex real-world scenarios.

[0036] 103. Based on a unified wall clock timeline, identify key information nodes in multimodal conversation data. Each key information node has a corresponding feature identifier. In this embodiment of the invention, optionally, in-depth analysis of the text content can be performed on a unified time reference. The system automatically identifies key information nodes in the conversation, including but not limited to the following categories: pain points (such as operational lag), highlights (such as convenient functions), and decision points (such as determining a solution). Each node is assigned a feature identifier, which at least includes the node's category, the start and end time boundaries on a unified timeline, and the text evidence fragment used as the basis for judgment.

[0037] 104. Based on the unified wall clock time axis, calculate the sentiment vector corresponding to each moment in the multimodal conversation data to obtain the sentiment intensity curve of the multimodal conversation data; In this embodiment of the invention, optionally, the system performs sentiment analysis on the conversation in parallel or alternately. It calculates sentiment on a clause-by-clause basis; that is, it calculates the sentiment vector for each semantic clause and maps it to its corresponding time point to obtain the sentiment vector for each time point. The sentiment values ​​are then mapped onto a unified time axis, and time series smoothing techniques (such as exponential moving average) are used to connect the discrete sentiment points into a continuous, time-varying sentiment intensity curve. This curve reflects the overall sentiment fluctuation trend of the conversation.

[0038] 105. Based on all key information nodes and sentiment intensity curves, generate an interactive sentiment map of the multimodal conversation data. The interactive sentiment map includes the sentiment change trend and key information distribution of the multimodal conversation data.

[0039] Optionally, the system then integrates key information nodes with the sentiment intensity curve. The system places nodes on the sentiment curve based on their timestamps and calculates a comprehensive integration priority value to characterize the node's importance. This ultimately generates an interactive sentiment map with a timeline on the horizontal axis and sentiment intensity on the vertical axis, embedding key nodes as clickable markers. Users can visually see the correspondence between sentiment trends and key events and can click on markers to revisit the original context.

[0040] As can be seen, implementing this embodiment of the invention, by "calibrating the timelines of all modal conversations according to a preset anti-interference fitting algorithm and generating a unified wall clock timeline," can improve the consistency of multi-source heterogeneous time data. This is beneficial for establishing a reliable time benchmark for subsequent analysis, and further helps ensure the accuracy of key event identification and sentiment analysis results in the time dimension, achieving precise temporal modeling of the conversation process. By "identifying key information nodes and calculating sentiment intensity curves based on the unified wall clock timeline," the structure and quantification of conversation content analysis can be improved, which is beneficial for systematically capturing core events and emotional fluctuations in the conversation. This helps overcome the subjectivity and bias of manual analysis, achieving an objective and comprehensive insight into the conversation content. By "generating an interactive sentiment map based on all key information nodes and sentiment intensity curves," the visualization and relevance of the analysis results can be improved, which is beneficial for intuitively presenting the spatiotemporal coupling relationship between key information and sentiment changes. This is beneficial for supporting users to conduct exploratory data analysis and in-depth evidence tracing, achieving an integrated analysis experience from macro trends to micro evidence. In summary, this embodiment of the invention can improve the accuracy and objectivity of multimodal conversation data analysis while improving the traceability of analysis results.

[0041] Example 2 Please see Figure 2 , Figure 2 This is a flowchart illustrating another multimodal conversation data analysis method based on time-axis fusion analysis disclosed in an embodiment of the present invention. Figure 2 The described multimodal conversation data analysis method based on time-axis fusion analysis can be applied to interview / conversation devices, and also to smart devices associated with these devices. These smart devices include, but are not limited to, one or more of the following: switching devices, mobile terminal devices, cloud devices, edge computing devices, relay devices, base station devices, urban management devices, and intelligent connected devices. This invention does not limit the scope of these applications. Figure 2 As shown, this multimodal session data analysis method based on time-axis fusion analysis can include the following operations: 201. Obtain multimodal session data. Each modal session data in the multimodal session data has a corresponding modal session timeline. 202. Based on the preset anti-interference fitting algorithm, calibrate the timelines of all modal conversations to generate a unified wall clock timeline for multimodal conversation data; 203. Based on the unified wall clock timeline, extract semantic sentence sequences from multimodal conversation data; 204. For each semantic sentence in the semantic sentence sequence, extract the text content features and contextual features of that semantic sentence; 205. Based on text content features and contextual features, match the key information nodes of the semantic sentence. Each key information node has a corresponding feature identifier. In this embodiment of the invention, optionally, for extracting semantic sentence sequences, the text can be automatically segmented based on a unified wall clock timeline. Sentence segmentation rules can be combined with Chinese punctuation (periods, question marks, etc.), the duration of silent pauses output by ASR (e.g., if it exceeds 300 milliseconds, the sentence is broken), and speaker switching points, etc. Further optional, for feature extraction: Text content features can include entities such as product names, module names, and scene words extracted from sentences using Named Entity Recognition (NER) technology; as well as keywords matched through a predefined trigger word list (such as "lag", "efficiency", "decision" etc.). Contextual features: These can refer to taking the current clause as the center and selecting neighboring clauses within a certain window size (e.g., ±2 clauses) before and after it. The sentiment polarity (positive / negative) and speech act type (e.g., statement, inquiry, commitment) of these neighboring clauses are calculated as the contextual features of the current clause. Optionally, for matching key information nodes: the features described above for each semantic sentence are input into a multi-label classification model (e.g., a text encoder based on the Transformer architecture with multiple classification heads connected in parallel at the top). This model shares the feature representation at the bottom layer, while the top layer determines the probability that the sentence belongs to categories such as "pain point," "highlight," and "decision point." When the probability exceeds a threshold or a rule is triggered, the sentence is determined to be a key information node.

[0042] Specifically, this multi-label classification model can use a shared underlying encoder (such as the Transformer architecture) to extract general features of sentences, and then connect multiple specific classifiers (or classification heads) in parallel at the top level to determine the probability that the sentence belongs to categories such as "pain point", "highlight", and "decision point", thereby achieving multi-task joint recognition.

[0043] 206. Based on the unified wall clock time axis, calculate the sentiment vector corresponding to each moment in the multimodal conversation data to obtain the sentiment intensity curve of the multimodal conversation data; In this embodiment of the invention, as an optional implementation, the above-mentioned calculation of the sentiment vector corresponding to each moment in the multimodal conversation data based on a unified wall clock time axis to obtain the sentiment intensity curve of the multimodal conversation data includes: For each semantic clause, calculate the sentiment vector of that semantic clause. The sentiment vector is used to represent the intensity distribution of the corresponding semantic clause in different sentiment dimensions. The sentiment vector of each semantic clause is mapped to the time corresponding to that semantic clause on a unified wall clock time axis, generating a discrete sequence of sentiment vectors; Time series smoothing is performed on discrete emotion vector sequences to generate emotion intensity curves for multimodal conversation data.

[0044] In this embodiment of the invention, optionally, for calculating the sentiment vector of a sentence: a sentiment analysis model (such as a finely tuned pre-trained language model) can be used to calculate it for each semantic sentence. The model outputs the probability distribution of the sentence belonging to different sentiment dimensions (such as "positive-negative-neutral"), forming a sentiment vector. Preferably, the sentiment vector is a multi-dimensional vector used to represent the probability distribution on different sentiment dimensions, such as [positive probability p_positive, negative probability p_negative, neutral probability p_neutral]. Optionally, the sentiment vector can also be represented in a comprehensive scalar form, for example, by calculating the difference between the positive probability and the negative probability, a continuous scalar value representing the sentiment polarity and intensity is obtained.

[0045] Alternatively, to improve the accuracy of sentiment intensity, a strategy of model-lexicon fusion can be adopted. That is, based on the sentiment score output by the model, the lexicon intensity value calculated by the sentiment lexicon (after correction considering negation words and degree adverbs) is combined and weighted (for example, the model score has a weight of 0.7 and the lexicon intensity has a weight of 0.3) to obtain the final sentiment vector.

[0046] Further optionally, for generating a discrete sentiment vector sequence: the sentiment vector of each clause can be assigned to the center time ((start time start_ms + end time end_ms) / 2) on a unified wall clock time axis, thereby obtaining a series of discrete sentiment data points on the time axis.

[0047] Further, if the conversation involves multiple speakers, to avoid bias caused by differences in individual emotional expression, the emotional intensity values ​​within each speaker can be standardized (e.g., using Z-Score normalization or normalization based on median absolute deviation) before generating the final emotional curve, and then the whole curve can be smoothed. Alternatively, for time series smoothing, the Exponential Moving Average (EMA) algorithm can be applied to the discrete series described above. This algorithm ensures that the sentiment value at the current moment depends not only on the current clause but also on the sentiment values ​​at previous moments, with the weights decaying exponentially over time. The formula is logically S_t = α * V_t + (1-α) * S_(t-1), where V_t is the discrete vector at time t, S_t is the smoothed vector at time t, and α is the smoothing factor. Similarly, S_(t-1) is the smoothed vector at time t-1. This process transforms discrete points into a continuous, smooth sentiment intensity curve, effectively filtering out short-term noise and reflecting macro trends.

[0048] As can be seen, implementing the embodiments of the present invention, by "calculating the sentiment vector of each semantic clause in different sentiment dimensions," can improve the granularity and richness of sentiment description. This is beneficial for going beyond simple positive / negative binary classification, capturing more complex sentiment states, and thus enabling a more accurate portrayal of the speaker's emotional experience, achieving multi-dimensional and refined sentiment analysis. By "smoothing the discrete sentiment vector sequence over time to generate a sentiment intensity curve," the random fluctuations and noise interference in sentiment analysis can be reduced. This is beneficial for obtaining a smooth and continuous trajectory of sentiment changes, which in turn facilitates a clearer identification of sentiment trends and key turning points, achieving robust and easily analyzable sentiment time series modeling.

[0049] 207. Based on all key information nodes and sentiment intensity curves, generate an interactive sentiment map of the multimodal conversation data. The interactive sentiment map includes the sentiment change trend and key information distribution of the multimodal conversation data.

[0050] Optionally, when a conversation involves multiple speakers, the interactive emotion map supports displaying independent emotion intensity curves for each speaker on separate tracks, which can be overlaid to compare the emotion change trends of different speakers.

[0051] In this embodiment of the invention, for other supplementary explanations of steps 201, 202, 206, and 207, please refer to the supplementary explanations of steps 101, 102, 104, and 105 in Embodiment 1. This embodiment of the invention will not repeat these explanations.

[0052] As can be seen, implementing this embodiment of the invention, by "extracting semantic sentence sequences based on a unified wall clock timeline," can improve the correlation accuracy between text analysis and the timeline. This helps ensure accurate time labels for subsequent key information identification and sentiment analysis, and further facilitates precise synchronization of text content with other modal data such as audio streams, enabling cross-modal analysis based on time. By "extracting text content features and contextual features for each semantic sentence," the depth and breadth of semantic understanding can be improved. This facilitates comprehensive judgment by combining local keywords with global contextual information, thereby improving the accuracy and contextual relevance of key information node identification and achieving context-aware intelligent information extraction. By "matching key information nodes based on the aforementioned features," the automation and standardization of key information identification can be improved, reducing reliance on human experience, ensuring consistency of results, and facilitating the formation of reusable analysis standards across different projects and analysts, achieving efficient and repeatable key information mining.

[0053] In this embodiment of the invention, as another optional implementation, the above-mentioned interactive sentiment map of multimodal conversation data generated based on all key information nodes and sentiment intensity curves includes: For each key information node, in the unified wall clock timeline, the key information node is matched with the emotion vector of the emotion intensity curve at the same time. Based on the contextual association features of the key information node, the contextual consistency measure corresponding to the key information node is calculated. The contextual consistency measure is used to represent the degree of consistency between the semantic clause of the corresponding key information node and the adjacent semantic clauses in terms of topic. Based on the feature identifier of the key information node, the corresponding matched sentiment vector, and the corresponding context consistency measure, the fusion priority value of the key information node is calculated. The fusion priority value is used to represent the importance of the corresponding key information node in the multimodal conversation data. Based on the fusion priority value of all key information nodes, corresponding interactive markers are generated on the time axis of the emotion intensity curve. Each interactive marker is associated with the feature identifier of the corresponding key information node. The interactive markers are used as interactive interfaces to trigger access to the corresponding key information nodes. By integrating the sentiment intensity curves with interactive markers with the feature identifiers of all key information nodes, an interactive sentiment map of multimodal conversation data is generated.

[0054] In this embodiment of the invention, optionally, for node and emotion vector matching: the emotion vector value at the same moment on the emotion intensity curve can be found based on the timestamp of the key information node on the unified wall clock time axis (usually the center moment of the node's time period). Further, optionally, for calculating the context consistency metric, the topic similarity between the sentence containing the current key information node and its neighboring sentences can be calculated. Specifically, this can be achieved by calculating the cosine similarity of sentence vectors (e.g., generated using models such as SimCSE or E5) and taking the average as the context consistency metric. Higher consistency indicates that the node is not an isolated event and its importance may be higher. Alternatively, a weighted scoring formula can be designed to calculate the fusion priority value. For example: Score = w1 * g(Node_Type, Emotion_Vector) + w2 * Context_Consistency + w3 * Other_Factors. Here, g is a function related to the node type; for "pain point" nodes, the negative portion of the emotion vector contributes a positive score; for "bright spot" nodes, the positive portion contributes a positive score. w1, w2, and w3 are weights, and Other_Factors can refer to other possible factors, such as the node's own confidence level. This fusion priority value quantifies the importance of the node. Specifically, when calculating the fusion priority value, the contribution of the sentiment vector differs for different categories of key information nodes. For example, for "pain point" nodes, the negative part of the sentiment vector (or the overall negative value of the sentiment) contributes a positive score; for "bright spot" nodes, the positive part of the sentiment vector (or the overall positive value of the sentiment) contributes a positive score; and for "decision point" nodes, the absolute value of the sentiment vector (representing the intensity of sentiment fluctuation) contributes a positive score.

[0055] Further, optionally, for generating and integrating interactive markers: interactive markers (such as icons or highlighted points) can be generated at corresponding positions on the timeline of the sentiment curve based on priority scores. The visual prominence of the markers (such as size and color intensity) can be correlated with the priority scores. Each marker is bound to all information (feature identifiers) of its corresponding node and has a backend link; clicking on it will jump to the original audio or text segment. Finally, the marked sentiment curves, node lists, and other elements are integrated into a single visual interface to form an interactive sentiment map. As can be seen, implementing this optional embodiment, by "calculating the fusion priority value based on the feature identifiers of key information nodes, matched sentiment vectors, and contextual consistency metrics," can improve the comprehensiveness and objectivity of the importance assessment of key information nodes. This facilitates the weighted ranking of multi-dimensional information such as content, sentiment, and contextual consistency, thereby enabling the automatic selection of the most noteworthy core segments in the conversation and achieving data-driven automatic importance assessment. By "generating interactive markers based on the fusion priority value and associating them with feature identifiers," the interactivity and operability of the analysis results can be improved. This allows users to quickly locate high-priority content and drill down for further investigation, significantly improving the efficiency of data exploration and evidence verification, and achieving an efficient and intuitive human-computer collaborative analysis loop. By "integrating sentiment curves with interactive markers with key node feature identifiers to generate an interactive sentiment map," the integration and insight of the analysis results can be improved. This allows for the centralized display of sentiment trends, key events, and their relationships in a single view, providing users with a unified analysis tool that combines a global perspective with in-depth detail, achieving a one-stop, interactive conversation analysis and insight platform.

[0056] Example 3 Please see Figure 3 , Figure 3 This is a schematic diagram of a multimodal conversation data analysis system based on time-axis fusion analysis disclosed in an embodiment of the present invention. This multimodal conversation data analysis system based on time-axis fusion analysis can be applied to interview / conversation devices, and also to intelligent devices associated with the interview / conversation devices. These intelligent devices include, but are not limited to, one or more of the following: switching devices, mobile terminal devices, cloud devices, edge computing devices, relay devices, base station devices, urban management devices, and intelligent connected devices. The embodiments of the present invention do not impose limitations on this. Figure 3 As shown, this multimodal conversation data analysis system based on time-axis fusion analysis may include: The acquisition module 301 is used to acquire multimodal session data, and each modal session data in the multimodal session data has a corresponding modal session timeline; The calibration module 302 is used to calibrate the time axis of all modal sessions according to the preset anti-interference fitting algorithm and generate a unified wall clock time axis for multimodal session data. The identification module 303 is used to identify key information nodes in multimodal session data based on a unified wall clock timeline. Each key information node has a corresponding feature identifier. The calculation module 304 is used to calculate the sentiment vector corresponding to each moment in the multimodal conversation data based on the unified wall clock time axis, so as to obtain the sentiment intensity curve of the multimodal conversation data. The generation module 305 is used to generate an interactive sentiment map of multimodal conversation data based on all key information nodes and sentiment intensity curves. The interactive sentiment map includes the sentiment change trend and key information distribution of the multimodal conversation data.

[0057] As can be seen, implementing this embodiment of the invention, by "calibrating the timelines of all modal conversations according to a preset anti-interference fitting algorithm and generating a unified wall clock timeline," can improve the consistency of multi-source heterogeneous time data. This is beneficial for establishing a reliable time benchmark for subsequent analysis, and further helps ensure the accuracy of key event identification and sentiment analysis results in the time dimension, achieving precise temporal modeling of the conversation process. By "identifying key information nodes and calculating sentiment intensity curves based on the unified wall clock timeline," the structure and quantification of conversation content analysis can be improved, which is beneficial for systematically capturing core events and emotional fluctuations in the conversation. This helps overcome the subjectivity and bias of manual analysis, achieving an objective and comprehensive insight into the conversation content. By "generating an interactive sentiment map based on all key information nodes and sentiment intensity curves," the visualization and relevance of the analysis results can be improved, which is beneficial for intuitively presenting the spatiotemporal coupling relationship between key information and sentiment changes. This is beneficial for supporting users to conduct exploratory data analysis and in-depth evidence tracing, achieving an integrated analysis experience from macro trends to micro evidence. In summary, this embodiment of the invention can improve the accuracy and objectivity of multimodal conversation data analysis while improving the traceability of analysis results.

[0058] In this embodiment of the invention, as an optional implementation, the calibration module 302 calibrates all modal session timelines according to a preset anti-interference fitting algorithm, and generates a unified wall clock timeline for multimodal session data in the following specific ways: For each modal session data, extract the set of event anchor points for that modal session data; Match the event anchor sets of all modal session data to obtain a set of associated event anchor pairs between different modal session data; Based on the set of associated event anchor points and a preset anti-interference fitting algorithm, a global mapping relationship is generated between the time axis of all modal sessions in the multimodal session data and the time axis of the standard reference wall clock. A unified wall clock timeline for multimodal session data is generated based on the global mapping relationship.

[0059] As can be seen, implementing this optional embodiment, by "extracting and matching event anchor point sets from different modalities to obtain a set of associated event anchor point pairs," can improve the feasibility and accuracy of cross-modal time alignment. This is beneficial for providing high-quality, interpretable correspondence data for calibration, and thus lays the foundation for building robust mapping relationships, enabling targeted, data-driven timestamp calibration. By "generating a global mapping relationship based on the set of associated event anchor point pairs and a preset anti-interference fitting algorithm," the negative impact of mismatched anchor points on calibration results can be reduced. This is beneficial for improving the robustness and reliability of time axis calibration, and thus enabling the acquisition of an accurate unified time axis even in the presence of noisy data, achieving highly robust multimodal time synchronization.

[0060] In this optional embodiment, as an optional implementation, the above-mentioned calculation module 304 is further configured to calculate the fitting residual of each associated event anchor pair in the associated event anchor pair set according to the global mapping relationship before the generation module 305 generates the unified wall clock time axis of multimodal session data according to the global mapping relationship. Optional, such as Figure 4 As shown, the system also includes: Analysis module 306 is used to analyze the distribution characteristics of the fitting residuals based on all the fitting residuals; Decision module 307 is used to determine whether there are feature mutation target points in the fitting residual distribution features; The determination module 308 is used to determine the calibration reference modal session time axis based on the characteristic mutation target point across all modal session time axes, if it exists. The segmentation module 309 is used to divide the calibration reference modal session time axis into a corresponding number of time periods based on the characteristic mutation points; The generation module 305 is also used to generate a local mapping relationship for each time period based on all the associated event anchors contained in that time period and a preset anti-interference fitting algorithm. The generation module 305 is also used to generate a unified wall clock timeline for multimodal session data based on all local mapping relationships; If it does not exist, the generation module 305 is triggered to perform the operation of generating a unified wall clock time axis for multimodal session data based on the global mapping relationship.

[0061] As can be seen, implementing this optional embodiment, by "calculating the fitting residuals, analyzing their distribution characteristics, and determining whether there are characteristic mutation target points," can improve the detection capability of complex time drift patterns. This facilitates the adaptive identification of nonlinear distortions that simple linear models cannot handle, and further helps to dynamically determine the necessity of segmented calibration, enabling intelligent selection of calibration strategies. By "determining the calibration reference time axis and performing segmented fitting when mutation points exist," the fitting accuracy of nonlinear time drift can be improved. This helps to effectively eliminate accumulated errors caused by equipment temperature drift, sampling rate differences, etc., and thus helps to maintain high-precision time alignment throughout long-term sessions, achieving strong adaptability to complex real-world scenarios.

[0062] In this embodiment of the invention, as another optional implementation, the specific method by which the identification module 303 identifies key information nodes in multimodal session data based on a unified wall clock timeline includes: Based on a unified wall clock timeline, semantic sentence sequences are extracted from multimodal conversation data; For each semantic sentence in the semantic sentence sequence, extract the text content features and contextual features of that semantic sentence; Based on text content features and contextual features, the key information nodes of the semantic sentence are matched.

[0063] As can be seen, implementing this embodiment of the invention, by "extracting semantic sentence sequences based on a unified wall clock timeline," can improve the correlation accuracy between text analysis and the timeline. This helps ensure accurate time labels for subsequent key information identification and sentiment analysis, and further facilitates precise synchronization of text content with other modal data such as audio streams, enabling cross-modal analysis based on time. By "extracting text content features and contextual features for each semantic sentence," the depth and breadth of semantic understanding can be improved. This facilitates comprehensive judgment by combining local keywords with global contextual information, thereby improving the accuracy and contextual relevance of key information node identification and achieving context-aware intelligent information extraction. By "matching key information nodes based on the aforementioned features," the automation and standardization of key information identification can be improved, reducing reliance on human experience, ensuring consistency of results, and facilitating the formation of reusable analysis standards across different projects and analysts, achieving efficient and repeatable key information mining.

[0064] In this optional embodiment, as an optional implementation, the calculation module 304 calculates the emotion vector corresponding to each moment in the multimodal conversation data based on the unified wall clock time axis, and obtains the emotion intensity curve of the multimodal conversation data in the following specific ways: For each semantic clause, calculate the sentiment vector of that semantic clause. The sentiment vector is used to represent the intensity distribution of the corresponding semantic clause in different sentiment dimensions. The sentiment vector of each semantic clause is mapped to the time corresponding to that semantic clause on a unified wall clock time axis, generating a discrete sequence of sentiment vectors; Time series smoothing is performed on discrete emotion vector sequences to generate emotion intensity curves for multimodal conversation data.

[0065] As can be seen, implementing the embodiments of the present invention, by "calculating the sentiment vector of each semantic clause in different sentiment dimensions," can improve the granularity and richness of sentiment description. This is beneficial for going beyond simple positive / negative binary classification, capturing more complex sentiment states, and thus enabling a more accurate portrayal of the speaker's emotional experience, achieving multi-dimensional and refined sentiment analysis. By "smoothing the discrete sentiment vector sequence over time to generate a sentiment intensity curve," the random fluctuations and noise interference in sentiment analysis can be reduced. This is beneficial for obtaining a smooth and continuous trajectory of sentiment changes, which in turn facilitates a clearer identification of sentiment trends and key turning points, achieving robust and easily analyzable sentiment time series modeling.

[0066] In this optional embodiment, as another optional implementation, the specific method by which the generation module 305 generates an interactive sentiment map of multimodal conversation data based on all key information nodes and sentiment intensity curves includes: For each key information node, in the unified wall clock timeline, the key information node is matched with the emotion vector of the emotion intensity curve at the same time. Based on the contextual association features of the key information node, the contextual consistency measure corresponding to the key information node is calculated. The contextual consistency measure is used to represent the degree of consistency between the semantic clause of the corresponding key information node and the adjacent semantic clauses in terms of topic. Based on the feature identifier of the key information node, the corresponding matched sentiment vector, and the corresponding context consistency measure, the fusion priority value of the key information node is calculated. The fusion priority value is used to represent the importance of the corresponding key information node in the multimodal conversation data. Based on the fusion priority value of all key information nodes, corresponding interactive markers are generated on the time axis of the emotion intensity curve. Each interactive marker is associated with the feature identifier of the corresponding key information node. The interactive markers are used as interactive interfaces to trigger access to the corresponding key information nodes. By integrating the sentiment intensity curves with interactive markers with the feature identifiers of all key information nodes, an interactive sentiment map of multimodal conversation data is generated.

[0067] As can be seen, implementing this optional embodiment, by "calculating the fusion priority value based on the feature identifiers of key information nodes, matched sentiment vectors, and contextual consistency metrics," can improve the comprehensiveness and objectivity of the importance assessment of key information nodes. This facilitates the weighted ranking of multi-dimensional information such as content, sentiment, and contextual consistency, thereby enabling the automatic selection of the most noteworthy core segments in the conversation and achieving data-driven automatic importance assessment. By "generating interactive markers based on the fusion priority value and associating them with feature identifiers," the interactivity and operability of the analysis results can be improved. This allows users to quickly locate high-priority content and drill down for further investigation, significantly improving the efficiency of data exploration and evidence verification, and achieving an efficient and intuitive human-computer collaborative analysis loop. By "integrating sentiment curves with interactive markers with key node feature identifiers to generate an interactive sentiment map," the integration and insight of the analysis results can be improved. This allows for the centralized display of sentiment trends, key events, and their relationships in a single view, providing users with a unified analysis tool that combines a global perspective with in-depth detail, achieving a one-stop, interactive conversation analysis and insight platform.

[0068] Example 4 Please see Figure 5 , Figure 5 This is a schematic diagram of another multimodal conversation data analysis system based on time-axis fusion analysis disclosed in an embodiment of the present invention. This multimodal conversation data analysis system based on time-axis fusion analysis can be applied to interview / conversation devices, and also to intelligent devices associated with the interview / conversation devices. These intelligent devices include, but are not limited to, one or more of the following: switching devices, mobile terminal devices, cloud devices, edge computing devices, relay devices, base station devices, urban management devices, and intelligent connected devices. The embodiments of the present invention do not impose limitations on these devices. Figure 5 As shown, this multimodal conversation data analysis system based on time-axis fusion analysis may include: Memory 401 that stores executable program code.

[0069] Processor 402 coupled to memory 401.

[0070] The processor 402 calls the executable program code stored in the memory 401 to execute the steps in the multimodal session data analysis method based on time axis fusion analysis described in Embodiment 1 or Embodiment 2 of the present invention.

[0071] Example 5 This invention discloses a computer storage medium storing computer instructions. When these computer instructions are invoked, they are used to execute the steps in the multimodal session data analysis method based on time axis fusion analysis described in Embodiment 1 or Embodiment 2 of this invention.

[0072] Example 6 This invention discloses a computer program product, which includes a non-transitory computer storage medium storing a computer program, and the computer program is operable to cause a computer to perform the steps in the multimodal session data analysis method based on time axis fusion analysis described in Embodiment 1 or Embodiment 2.

[0073] The system embodiments described above are merely illustrative. The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical modules; that is, they may be located in one place or distributed across multiple network modules. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0074] Through the detailed description of the above embodiments, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, including read-only memory (ROM), random access memory (RAM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), one-time programmable read-only memory (OTPROM), electrically-Erasable Programmable Read-Only Memory (EEPROM), compact disc read-only memory (CD-ROM) or other optical disc storage, disk storage, magnetic tape storage, or any other computer-readable medium that can be used to carry or store data.

[0075] Finally, it should be noted that the multimodal conversation data analysis method and system based on time axis fusion analysis disclosed in the embodiments of the present invention are merely preferred embodiments of the present invention and are only used to illustrate the technical solutions of the present invention, not to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A multimodal conversation data analysis method based on time-axis fusion analysis, characterized in that, The method includes: Acquire multimodal session data, wherein each modal session data has a corresponding modal session timeline; According to the preset anti-interference fitting algorithm, all the modal session time axes are calibrated to generate a unified wall clock time axis for the multimodal session data; Based on the unified wall clock timeline, key information nodes in the multimodal session data are identified, and each key information node has a corresponding feature identifier. Based on the unified wall clock time axis, the emotion vector corresponding to each moment in the multimodal conversation data is calculated to obtain the emotion intensity curve of the multimodal conversation data; Based on all the key information nodes and the sentiment intensity curve, an interactive sentiment map of the multimodal conversation data is generated. The interactive sentiment map includes the sentiment change trend and key information distribution of the multimodal conversation data.

2. The multimodal conversation data analysis method based on time-axis fusion analysis according to claim 1, characterized in that, The step of calibrating all the modal session timelines according to a preset anti-interference fitting algorithm to generate a unified wall clock timeline for the multimodal session data includes: For each modal session data, extract the set of event anchor points for that modal session data; Match the event anchor point sets of all the modal session data to obtain a set of associated event anchor point pairs between different modal session data; Based on the set of associated event anchor points and the preset anti-interference fitting algorithm, a global mapping relationship between the time axis of all modal sessions in the multimodal session data and the time axis of the standard reference wall clock is generated; Based on the global mapping relationship, a unified wall clock timeline for the multimodal session data is generated.

3. The multimodal conversation data analysis method based on time-axis fusion analysis according to claim 2, characterized in that, Before generating the unified wall clock timeline of the multimodal session data based on the global mapping relationship, the method further includes: Based on the global mapping relationship, calculate the fitting residual for each associated event anchor pair in the associated event anchor pair set; Analyze the distribution characteristics of the fitting residuals based on all the aforementioned fitting residuals; Determine whether there are any feature mutation target points in the fitted residual distribution features; If it exists, then based on the feature mutation target point, determine the calibration benchmark modal session time axis in all modal session time axes; based on the feature mutation point, divide the calibration benchmark modal session time axis into a corresponding number of time periods; for each time period, based on all the associated event anchor points contained in the time period and the preset anti-interference fitting algorithm, generate the local mapping relationship corresponding to the time period; Based on all the aforementioned local mapping relationships, a unified wall clock timeline for the multimodal session data is generated; If it does not exist, the operation of generating a unified wall clock time axis for the multimodal session data based on the global mapping relationship is triggered.

4. The multimodal conversation data analysis method based on time-axis fusion analysis according to any one of claims 1-3, characterized in that, The process of identifying key information nodes in the multimodal session data based on the unified wall clock timeline includes: Based on the unified wall clock timeline, semantic sentence sequences are extracted from the multimodal conversation data; For each semantic sentence in the semantic sentence sequence, extract the text content features and context association features of that semantic sentence; Based on the text content features and the context association features, the key information nodes of the semantic sentence are matched.

5. The multimodal conversation data analysis method based on time-axis fusion analysis according to claim 4, characterized in that, The step of calculating the sentiment vector corresponding to each moment in the multimodal conversation data based on the unified wall clock time axis, and obtaining the sentiment intensity curve of the multimodal conversation data, includes: For each semantic sentence, calculate the sentiment vector of the semantic sentence, and the sentiment vector is used to represent the intensity distribution of the corresponding semantic sentence on different sentiment dimensions; The sentiment vector of each semantic clause is mapped to the time corresponding to that semantic clause on the unified wall clock time axis to generate a discrete sentiment vector sequence; The discrete sentiment vector sequence is subjected to time series smoothing to generate the sentiment intensity curve of the multimodal conversation data.

6. The multimodal conversation data analysis method based on time-axis fusion analysis according to claim 4, characterized in that, The step of generating an interactive sentiment map of the multimodal conversation data based on all the key information nodes and the sentiment intensity curve includes: For each key information node, in the unified wall clock timeline, the key information node is matched with the emotion vector of the emotion intensity curve at the same time. Based on the contextual association features of the key information node, the contextual consistency measure corresponding to the key information node is calculated. The contextual consistency measure is used to represent the degree of consistency between the semantic clause corresponding to the key information node and the neighboring semantic clauses in terms of topic. Based on the feature identifier of the key information node, the corresponding matched sentiment vector, and the corresponding context consistency measure, the fusion priority value of the key information node is calculated. The fusion priority value is used to represent the importance of the corresponding key information node in the multimodal conversation data. Based on the fusion priority value of all the key information nodes, a corresponding interactive marker is generated on the time axis of the emotion intensity curve. Each interactive marker is associated with the feature identifier of the corresponding key information node, and the interactive marker is used as an interactive interface to trigger access to the corresponding key information node. The emotional intensity curve with the interactive markers is integrated with the feature identifiers of all the key information nodes to generate an interactive emotional map of the multimodal conversation data.

7. A multimodal conversation data analysis system based on time-axis fusion analysis, characterized in that, The system includes: The acquisition module is used to acquire multimodal session data, wherein each modal session data in the multimodal session data has a corresponding modal session timeline; The calibration module is used to calibrate all the modal session timelines according to a preset anti-interference fitting algorithm, and generate a unified wall clock timeline for the multimodal session data. The identification module is used to identify key information nodes in the multimodal session data based on the unified wall clock timeline, and each key information node has a corresponding feature identifier; The calculation module is used to calculate the sentiment vector corresponding to each moment in the multimodal conversation data based on the unified wall clock time axis, so as to obtain the sentiment intensity curve of the multimodal conversation data. The generation module is used to generate an interactive sentiment map of the multimodal conversation data based on all the key information nodes and the sentiment intensity curve. The interactive sentiment map includes the sentiment change trend and key information distribution of the multimodal conversation data.

8. The multimodal conversation data analysis system based on time-axis fusion analysis according to claim 7, characterized in that, The calibration module calibrates all modal session timelines according to a preset anti-interference fitting algorithm, and the specific method for generating a unified wall clock timeline for the multimodal session data includes: For each modal session data, extract the set of event anchor points for that modal session data; Match the event anchor point sets of all the modal session data to obtain a set of associated event anchor point pairs between different modal session data; Based on the set of associated event anchor points and the preset anti-interference fitting algorithm, a global mapping relationship between the time axis of all modal sessions in the multimodal session data and the time axis of the standard reference wall clock is generated; Based on the global mapping relationship, a unified wall clock timeline for the multimodal session data is generated.

9. A multimodal conversation data analysis system based on time-axis fusion analysis, characterized in that, The system includes: Memory containing executable program code; A processor coupled to the memory; The processor calls the executable program code stored in the memory to execute the multimodal session data analysis method based on time axis fusion analysis as described in any one of claims 1-6.

10. A computer storage medium, characterized in that, The computer storage medium stores computer instructions, which, when invoked, are used to execute the multimodal session data analysis method based on time axis fusion analysis as described in any one of claims 1-6.

Images