Conversation-based multi-modal feature analysis method and electronic equipment

Abstract

The invention discloses a conversation-based multi-modal feature analysis method and electronic equipment, and relates to the technical field of language analysis, and the method comprises the following steps: collecting multi-modal evaluation data of a conversation process of a to-be-evaluated object, and carrying out the feature marking of the multi-modal evaluation data to obtain state feature points; performing statistics on occurrence frequency and feature intensity of similar features in the state feature points to obtain feature correlation parameters; extracting a historical language state feature distribution rule in different evaluation scenes from the historical evaluation data set, and calculating a language state influence factor between a language state tendency value and a feature correlation parameter in the historical evaluation data according to the historical language state feature distribution rule; the target influence factor is screened out from the language state influence factors according to the real-time language state feature distribution feature of the current conversation scene, and the effect is that clear guidance is provided for formulation of a subsequent intervention strategy.

Description

Technical Field

[0001] This invention relates to the field of language analysis technology, and more specifically, to a conversation-based multimodal feature analysis method and electronic device. Background Technology

[0002] Traditional language assessment methods have many limitations. Past assessments relied heavily on human interviews, judging the subject's state based on the content of their speech and observed facial expressions and gestures. However, human assessment is susceptible to subjective factors; different assessors may interpret the same performance differently, and the dimensions of information collected and analyzed manually are limited, making it difficult to comprehensively capture the subject's language state characteristics. Traditional methods are time-consuming and labor-intensive when handling large-scale assessment tasks, and the lack of standardized quantitative assessment criteria makes it difficult to guarantee the consistency and reliability of assessment results. Furthermore, the complex relationships between language state characteristics make it difficult to effectively uncover patterns in the distribution of language states in historical assessment data, thus failing to provide valuable guidance for current assessments. Summary of the Invention

[0003] In view of the shortcomings of the existing technology, the purpose of this invention is to provide a method and electronic device for multimodal feature analysis based on conversation.

[0004] To achieve the above objectives, the present invention provides the following technical solution: A conversation-based multimodal feature analysis method, comprising the following steps: Collect multimodal evaluation data of the conversation process of the subject to be evaluated, and perform feature labeling on the multimodal evaluation data to obtain state feature points; Feature association parameters are obtained by analyzing the frequency and intensity of similar features among statistical state feature points. Extract the distribution patterns of historical language state features under different assessment scenarios from the historical assessment dataset, and calculate the language state influence factor between the language state tendency value and the feature association parameter in the historical assessment data based on the distribution patterns of historical language state features. Based on the real-time language state feature distribution characteristics of the current conversation scenario, target influencing factors are selected from the language state influencing factors. Based on the target influencing factors, the core language state tendency value and auxiliary language state tendency value of the subject to be evaluated in the conversation process are determined, and the first language state evaluation value and the second language state evaluation value are output. The current status assessment value is obtained by judging the health status value of the assessed object based on the first language status assessment value and the second language status assessment value; The intervention strategy for the subject to be evaluated is determined based on the current status assessment value.

[0005] Preferably, the multimodal evaluation data includes voice data, facial expression data, and text response data.

[0006] Preferably, the multimodal evaluation data is characterized to obtain state feature points, specifically including the following steps: Language state features are obtained by labeling speech data with intonation and speech rate features; Facial expression data is labeled with facial expression action features to obtain facial expression language state features; Semantic and linguistic state features are used to label the text response data to obtain the text response linguistic state features. Among them, the language state features, facial expression language state features, and text response language state features constitute state feature points.

[0007] Preferably, feature association parameters are obtained by statistically analyzing the frequency and intensity of similar features among state feature points, specifically including the following steps: The frequency of occurrence of a feature is obtained by counting the frequency of occurrence of the same type of feature among state feature points within a preset time window; The feature strength value is obtained by calculating the degree of linguistic state expression of the state feature points; Feature association parameters are obtained by mapping the frequency of feature occurrence with the feature intensity value.

[0008] Preferably, the language state influence factor between the language state tendency value and the feature correlation parameter in the historical evaluation data is calculated based on the distribution pattern of historical language state characteristics, specifically including the following steps: Based on the distribution pattern of historical language state characteristics, historical language state tendency values ​​corresponding to language state tendency labels are extracted from historical assessment data. The language state difference value is obtained by calculating the language state tendency difference corresponding to the same type of feature in the historical language state tendency value based on the feature association parameter; The feature ratio value is obtained by proportionally calculating the language state difference value and the feature association parameter. The language state influence factor is obtained by weighted summation of all feature proportions.

[0009] Preferably, the target influencing factor is selected from the language state influencing factors based on the real-time language state feature distribution characteristics of the current conversation scenario, specifically including the following steps: Obtain the real-time language state feature distribution characteristics of the current conversation scene; After performing similarity matching between the real-time language state feature distribution and the historical language state feature distribution, the target influence factor is selected from the language state influence factors.

[0010] Preferably, after determining the core language state tendency value and auxiliary language state tendency value of the subject to be evaluated during the conversation based on the target influence factor, the first language state evaluation value and the second language state evaluation value are output, specifically including the following steps: Determine the core and auxiliary language state expression dimensions of the subject being evaluated during the conversation; Determine the first feature set corresponding to the core language state and the second feature set corresponding to the auxiliary language state; After determining the core language state tendency value based on the feature association parameters corresponding to the first feature set, the real-time language state strength value, and the target influence factor, the first language state evaluation value is output. Based on the feature association parameters corresponding to the second feature set, the real-time language state strength value, and the target influence factor, the auxiliary language state tendency value is determined and the second language state evaluation value is output.

[0011] Preferably, the current status assessment value is obtained by judging the health status value of the assessed object based on the first language status assessment value and the second language status assessment value, specifically including the following steps: Assign core weight values ​​to the first language status assessment values; Assign auxiliary weight values ​​to the second language status assessment values; The first evaluation factor is obtained by multiplying the first language status evaluation value by the core weight value; The second evaluation factor is obtained by multiplying the second language status assessment value by the auxiliary weight value; The comprehensive status assessment factor is obtained by summing the first assessment factor and the second assessment factor. Obtain a reference assessment database of state health values ​​corresponding to different combinations of language state tendencies and feature intensities; The current status assessment value is obtained by matching the comprehensive status assessment factors with the reference assessment database.

[0012] Preferably, determining the intervention strategy for the object to be evaluated based on the current status assessment value specifically includes the following steps: If the current status assessment value is lower than the status warning threshold, an intervention prompt message will be output; After calling the preset intervention plan library based on the intervention prompt information, candidate intervention plans that match the current language state feature distribution features are extracted; Evaluation data is obtained by assessing the implementation complexity and expected outcome of candidate intervention programs. The best-fit intervention strategy is one that extracts evaluation data from candidate intervention programs and finds that outperforms the pre-defined intervention standards.

[0013] An electronic device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the program, implements the talk-based multimodal feature analysis method.

[0014] Compared with the prior art, the present invention has the following beneficial effects: This invention, by collecting multimodal assessment data and marking state feature points, can comprehensively and meticulously capture the language state expressions of the subject being assessed during conversation. Compared to single-dimensional data collection, multimodal data fusion allows for a richer and more comprehensive mining of language state features, avoiding assessment biases caused by missing information and laying a solid foundation for subsequent accurate analysis. In the feature association and pattern mining stage, the frequency and intensity of similar features are statistically analyzed to obtain feature association parameters. These parameters are then combined with the language state feature distribution patterns extracted from historical data to calculate the language state influence factor, achieving a deep integration of historical experience and current assessment. This integration allows the assessment to move beyond the immediate characteristics of the current conversation, instead leveraging the language state and feature association patterns accumulated in historical data to provide a more valuable basis for judging the current language state tendency, improving the scientific rigor and reliability of the assessment. This application can quantify the health status of the subject being assessed, providing clear guidance for the formulation of subsequent intervention strategies. Attached Figure Description

[0015] Fig. 1 This is a schematic diagram illustrating the steps of the conversation-based multimodal feature analysis method proposed in this invention; Fig. 2 This is a schematic diagram illustrating the steps for calculating feature correlation parameters in the conversation-based multimodal feature analysis method proposed in this invention; Fig. 3 This is a schematic diagram of the structure of the electronic device provided in an embodiment of the present invention.

[0016] 610. Processor; 620. Communication interface; 630. Memory; 640. Communication bus. Detailed Implementation

[0017] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0018] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and those skilled in the art can make similar extensions without departing from the spirit of the invention. Therefore, the invention is not limited to the specific embodiments disclosed below.

[0019] Secondly, the term "an embodiment" or "embodiment" as used herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The phrase "in one embodiment" appearing in different places in this specification does not necessarily refer to the same embodiment, nor is it a single embodiment or an embodiment selectively excluded from other embodiments.

[0020] Reference Figs. 1-3 As shown.

[0021] The embodiments further illustrate the conversation-based multimodal feature analysis method and electronic device proposed in this invention.

[0022] A conversation-based multimodal feature analysis method, comprising the following steps: Collect multimodal evaluation data of the conversation process of the subject to be evaluated, and perform feature labeling on the multimodal evaluation data to obtain state feature points; Feature association parameters are obtained by analyzing the frequency and intensity of similar features among statistical state feature points. Extract the distribution patterns of historical language state features under different assessment scenarios from the historical assessment dataset, and calculate the language state influence factor between the language state tendency value and the feature association parameter in the historical assessment data based on the distribution patterns of historical language state features. Based on the real-time language state feature distribution characteristics of the current conversation scenario, target influencing factors are selected from the language state influencing factors. Based on the target influencing factors, the core language state tendency value and auxiliary language state tendency value of the subject to be evaluated in the conversation process are determined, and the first language state evaluation value and the second language state evaluation value are output. The current status assessment value is obtained by judging the health status value of the assessed object based on the first language status assessment value and the second language status assessment value; The intervention strategy for the subject to be evaluated is determined based on the current status assessment value.

[0023] First, multimodal assessment data of the conversation process of the subject to be evaluated is collected, such as voice data, facial expression data, and text response data. The modal assessment data is then labeled with features to obtain state feature points. Specifically, the voice data is labeled with intonation and speech rate features to obtain language state features, the facial expression data is labeled with facial expression action features to obtain facial expression language state features, and the text response data is labeled with semantic language state features to obtain text response language state features. These three types of features together constitute the state feature points.

[0024] The frequency and intensity of similar features within statistical state feature points are used to obtain feature association parameters. Specifically, within a preset time window, the frequency of similar features is counted, the degree of linguistic state expression is calculated to obtain feature intensity values, and then these two are correlated and mapped to obtain feature association parameters.

[0025] This study extracts the distribution patterns of historical language state features across different assessment scenarios from historical assessment datasets. Based on these patterns, it calculates the language state influence factor between language state tendency values ​​and feature association parameters in the historical assessment data. Specifically, it extracts historical language state tendency values ​​corresponding to language state tendency labels from historical data, calculates the differences in language state tendency among similar features using feature association parameters to obtain language state difference values, calculates the ratio between language state difference values ​​and feature association parameters to obtain feature ratio values, and finally, weighted sums of all feature ratio values ​​yield the language state influence factor.

[0026] A real-time scene recognition module is introduced at the data acquisition layer. Keywords from the conversation and ambient audio are extracted, and then the evaluation stage is automatically divided. In the initial stage, the voice acquisition uses a high sampling rate of 16kHz to capture subtle tremors, and facial expressions are extracted only by eye movement trajectories. In the core evaluation stage, facial micro-expression capture is activated, focusing on monitoring the related muscle groups of the zygomaticus major and corrugator supercilii muscles, and the weight of the fundamental frequency standard deviation in the voice features is increased by 40%.

[0027] Dynamic weight adjustment is designed in the feature processing layer. The information gain of each modality under the current label is calculated: when a language tendency is identified, the weight of speech features is automatically increased to 50%, text features to 30%, and facial features to 20%.

[0028] A feature fusion confidence scoring mechanism is constructed at the output layer. When the confidence level falls below a threshold, a supplementary data collection process is automatically initiated to ensure that the evaluation results maintain high accuracy during dynamic adjustments.

[0029] Based on the real-time language state feature distribution characteristics of the current conversation scenario, target influencing factors are selected from the language state influencing factors. First, the real-time language state feature distribution characteristics of the current scenario are obtained and matched with historical language state feature distribution characteristics to select target influencing factors. Using the target influencing factors, the core language state tendency value and auxiliary language state tendency value of the subject being evaluated during the conversation are determined to establish the core and auxiliary language state expression dimensions, as well as the corresponding first and second feature sets. Combining feature association parameters, real-time language state intensity values, and target influencing factors, the core language state tendency value and auxiliary language state tendency value are calculated, thus obtaining the first and second language state evaluation values.

[0030] The conversation scenarios are subdivided into core themes, and each scenario corresponds to a unique feature association model. For example, in the family conflict scenario, the correlation between specific feature combinations and actual language tendencies was discovered through training with over 100,000 samples; in the occupational stress scenario, specific feature combinations improved the accuracy of language tendency identification.

[0031] A personalized feature time series library is constructed by longitudinal tracking, recording the changes in feature associations from the first assessment to the Nth follow-up. For example, features at different stages are automatically labeled for language propensity assessment; a language inertia index is introduced to calculate the decay rate of associations of similar features in continuous assessments, providing a quantitative basis for the assessment.

[0032] The current status assessment value is obtained by determining the health status of the assessed object based on the first language status assessment value and the second language status assessment value. Core and auxiliary weight values ​​are assigned to the first language status assessment value and the second language status assessment value, respectively. Finally, a comprehensive status assessment factor is calculated and matched with a reference assessment database to obtain the current status assessment value.

[0033] Intervention strategies are determined based on the current situation assessment value. If the assessment value is lower than the situation warning threshold, a preset intervention plan library is used to extract candidate intervention plans. The implementation complexity, expected effect value, and other plan assessment data are evaluated, and suitable intervention strategies with assessment data superior to the preset standard are selected for intervention on the assessed object.

[0034] Multimodal evaluation data includes voice data, facial expression data, and text response data.

[0035] The process of labeling multimodal evaluation data to obtain state feature points includes the following steps: Language state features are obtained by labeling speech data with intonation and speech rate features; Facial expression data is labeled with facial expression action features to obtain facial expression language state features; Semantic and linguistic state features are used to label the text response data to obtain the text response linguistic state features. Among them, language state features, facial expression language state features, and text response language state features constitute state feature points.

[0036] Voice data records intonation and speech rate; for example, speech may increase in speed and pitch when excited. Facial expression data captures frowning and smiling facial expressions, such as frequent frowning. Text response data extracts the content of the language, such as responses to texts with linguistic biases, like "I'm sad." By labeling these three types of data, we can mine the language state features implied by intonation and speech rate from voice, extract the language state features corresponding to facial expressions from facial expressions, and analyze the semantically related language state features of text responses from text responses. Together, these three constitute the state feature points.

[0037] The feature association parameters are obtained by analyzing the frequency and intensity of similar features among statistical state feature points. This process includes the following steps: The frequency of occurrence of a feature is obtained by counting the frequency of occurrence of the same type of feature among state feature points within a preset time window; The feature strength value is obtained by calculating the degree of linguistic state expression of the state feature points; Feature association parameters are obtained by mapping the frequency of feature occurrence with the feature intensity value.

[0038] This application first focuses on state feature points, and statistically analyzes similar features within these state feature points within a pre-defined time window. The pre-defined time window can be a continuous 5-minute conversation of the subject being evaluated. For example, if "increased speech rate and rising intonation" is categorized as a similar excitement feature, then the frequency of occurrence of this feature during the 5-minute conversation is counted to obtain the feature frequency, which directly reflects the activity level of similar language state features during a specific time period.

[0039] The degree of linguistic state expression of state feature points is calculated. It's important to consider not only the frequency of occurrence but also the intensity of the expression. For example, the same phrase "speech speed increases and pitch rises" can vary in intensity; some instances show a slight increase or rise, while others show a sharp increase or rise. The intensity of this expression is quantified based on changes in speech waveforms and facial expression amplitude indicators to derive a feature strength value, which reflects the inherent intensity of the linguistic state feature.

[0040] This involves mapping the frequency of feature occurrences to their intensity values. This can be understood as establishing a correspondence that allows the two to work together to generate feature association parameters.

[0041] The calculation of the language state influence factor between the language state tendency value and the feature correlation parameter in historical assessment data based on the distribution pattern of historical language state characteristics includes the following steps: Based on the distribution pattern of historical language state characteristics, historical language state tendency values ​​corresponding to language state tendency labels are extracted from historical assessment data. The language state difference value is obtained by calculating the language state tendency difference corresponding to the same type of feature in the historical language state tendency value based on the feature association parameter; The feature ratio value is obtained by proportionally calculating the language state difference value and the feature association parameter. The language state influence factor is obtained by weighted summation of all feature proportions.

[0042] First, by leveraging the distribution patterns of historical language state features, we extract the historical language state tendency values ​​corresponding to language state tendency labels from historical assessment data. For example, in a tendency assessment scenario, we filter out the historical language state tendency values ​​marked as state labels from relevant assessment data for different conversation scenarios, such as workplace stress conversations and social conversations.

[0043] Feature association parameters are used to determine the differences in language state tendencies among similar features in historical language state tendency values. Taking the feature "fast speech" as an example, the language state tendency corresponding to "fast speech" may vary slightly in different historical assessment data, with some leaning more towards excitement. The language state difference value is quantified by combining feature association parameters, which include frequency and intensity, to show the fluctuations in language state tendencies of similar features in historical data.

[0044] The feature proportion value is obtained by proportionally calculating the language state difference value and the feature association parameter. Assuming that the language state difference value reflects the dispersion of the tendency of similar features, and the feature association parameter reflects the overall performance of the feature, the proportion of the two reveals the intrinsic relationship between the feature and the language state tendency difference. For example, for a certain feature, "increased intonation," the language state difference value is 0.6, the feature association parameter is 1.2, and after proportional calculation, the feature proportion value is 0.5, quantifying the contribution of this feature to the language state tendency difference.

[0045] A language state influence factor is generated by weighting and summing the proportions of all features. Different features have varying degrees of influence on language state tendencies; by assigning appropriate weights, the proportions of various features are integrated. For example, the speech rate feature has a weight of 0.3, corresponding to a proportion of 0.5; the facial expression feature has a weight of 0.4, corresponding to a proportion of 0.6. After weighted summation, the language state influence factor is obtained. This factor can effectively correlate historical data features with language state tendencies, providing a historical reference for judging language state tendencies.

[0046] Based on the real-time language state feature distribution characteristics of the current conversation scenario, target influencing factors are selected from the language state influencing factors. The specific steps include: Obtain the real-time language state feature distribution characteristics of the current conversation scene; After performing similarity matching between the real-time language state feature distribution and the historical language state feature distribution, the target influence factor is selected from the language state influence factors.

[0047] First, the real-time language state feature distribution characteristics of the current conversation scene are obtained. Multimodal data is collected and processed to obtain state feature points while the subject to be evaluated is engaged in conversation. After statistical correlation operations, the distribution of various language state features in the current scene is presented, including the proportion and combination of different language state features.

[0048] The system performs similarity matching between real-time language state feature distributions and historical language state feature distributions. Historical evaluation datasets store patterns in language state feature distributions across different past scenarios; for example, workplace stress scenarios often exhibit features such as "fast speaking speed, negative vocabulary, and high frequency of frowning." The system compares the current scenario's language state feature distribution with these historical scenario distributions one by one, calculating the similarity and identifying the historical scenario or data segment that best matches the current language state feature pattern.

[0049] The language state influence factor is a parameter calculated based on historical data, reflecting the degree of correlation between language state characteristics and language state tendencies. After identifying a historical feature distribution highly similar to the current scenario, the corresponding language state influence factor for that historical scenario is extracted and used as the target influence factor. This allows the evaluation process to fully draw on historical experience, improving the accuracy and scientific rigor of the analysis of language state in the current conversation.

[0050] After determining the core language state tendency and auxiliary language state tendency of the subject to be evaluated during the conversation based on the target impact factor, the first language state evaluation value and the second language state evaluation value are output. The specific steps include: Determine the core and auxiliary language state expression dimensions of the subject being evaluated during the conversation; Determine the first feature set corresponding to the core language state and the second feature set corresponding to the auxiliary language state; After determining the core language state tendency value based on the feature association parameters corresponding to the first feature set, the real-time language state strength value, and the target influence factor, the first language state evaluation value is output. Based on the feature association parameters corresponding to the second feature set, the real-time language state strength value, and the target influence factor, the auxiliary language state tendency value is determined and the second language state evaluation value is output.

[0051] This application first identifies the core and auxiliary language state expression dimensions. During the conversation with the subject to be evaluated, the core language state expression dimensions that have the most significant impact on the state, as well as the auxiliary language state expression dimensions that play a supplementary role, are determined.

[0052] The first feature set is formed by identifying closely related features around the core language state. Taking the core language state of negative language tendency as an example, its corresponding first feature set includes features such as "rapid speech rate, rising intonation, frowning expression, and negative semantic vocabulary," thus establishing a correspondence between language state and specific features.

[0053] For the core language state, the tendency value is comprehensively judged based on the feature association parameters corresponding to the first feature set, the real-time language state strength value, and the target influence factor. The interaction of these factors is quantitatively calculated to output a first language state evaluation value, thus presenting the degree of tendency of the core language state. Similarly, for the auxiliary language state, the tendency value is judged using the feature association parameters corresponding to the second feature set, the real-time language state strength value, and the target influence factor, and a second language state evaluation value is output, providing crucial quantitative basis for subsequent assessment of language state health.

[0054] The current status assessment value is obtained by judging the health status of the assessed object based on the first language status assessment value and the second language status assessment value. The specific steps include: Assign core weight values ​​to the first language status assessment values; Assign auxiliary weight values ​​to the second language status assessment values; The first evaluation factor is obtained by multiplying the first language status evaluation value by the core weight value; The second evaluation factor is obtained by multiplying the second language status assessment value by the auxiliary weight value; The comprehensive status assessment factor is obtained by summing the first assessment factor and the second assessment factor. Obtain a reference assessment database of state health values ​​corresponding to different combinations of language state tendencies and feature intensities; The current status assessment value is obtained by matching the comprehensive status assessment factors with the reference assessment database.

[0055] Because the first language state assessment value reflects the core language state tendency and has a greater impact on the state judgment, it is assigned a core weight value; the second language state assessment value corresponds to the auxiliary language state tendency and has a relatively minor impact, so it is assigned an auxiliary weight value. For example, the core weight is set to 0.7 and the auxiliary weight is set to 0.3.

[0056] The first evaluation factor is obtained by multiplying the first language status assessment value by the core weight value. This first evaluation factor quantifies the impact of the core language status on the overall state. The second evaluation factor is obtained by multiplying the second language status assessment value by the auxiliary weight value. This second evaluation factor reflects the role of the auxiliary language status. For example, if the first language status assessment value is 8 points and the core weight is 0.7, the first evaluation factor is 8 × 0.7 = 5.6; if the second language status assessment value is 6 points and the auxiliary weight is 0.3, the second evaluation factor is 6 × 0.3 = 1.8.

[0057] The sum of the first and second assessment factors yields the comprehensive state assessment factor, which is 5.6 + 1.8 = 7.4. This factor integrates the influence of core and auxiliary language states, and comprehensively reflects the effect of the current language state tendency on the state.

[0058] First, a reference assessment database is constructed to collect state health values ​​corresponding to different combinations of language state tendencies and feature intensities.

[0059] The calculated comprehensive status assessment factors are matched with data in the reference assessment database. The database is searched for the most matching combination of language status and corresponding health value records to determine the current health status of the subject to be assessed. The current status assessment value is then output, providing crucial quantitative basis for subsequent judgment of intervention strategies.

[0060] Determine the intervention strategy for the subject to be evaluated based on the current status assessment value, specifically including the following steps: If the current status assessment value is lower than the status warning threshold, an intervention prompt message will be output; After calling the preset intervention plan library based on the intervention prompt information, candidate intervention plans that match the current language state feature distribution features are extracted; Evaluation data is obtained by assessing the implementation complexity and expected outcome of candidate intervention programs. The best-fit intervention strategy is one that extracts evaluation data from candidate intervention programs and finds that outperforms the pre-defined intervention standards.

[0061] The current status assessment value is compared with a preset status warning threshold. If the current status assessment value is lower than the threshold, it means that the assessed object has a serious problem and intervention is required. At this time, an intervention prompt message is output. For example, if the status warning threshold is set to 60 points (assuming a full score of 100 points), and the current assessment value of the assessed object is 55 points, that is, the current status assessment value is lower than the status warning threshold, the intervention process will be triggered and a prompt message will be generated.

[0062] The system retrieves a pre-defined intervention protocol library based on the intervention prompts. This library stores diverse intervention protocols, including those adapted to different language state feature distribution scenarios. It then extracts matching candidate intervention protocols from the library based on the current language state feature distribution characteristics.

[0063] A multidimensional evaluation of candidate intervention programs is conducted, focusing on two core indicators: implementation complexity and expected outcome value. The implementation complexity assessment comprehensively considers the implementation scenario, such as online remote execution or offline face-to-face execution, and the required resources, such as the qualifications of the personnel, the type of equipment support, and the length of the implementation cycle. For example, a program requiring offline professional participation three times a week has a higher implementation complexity than a program that can be completed online once a week. The expected outcome value quantifies the degree of improvement of the program on assessment subjects with similar language state characteristics based on historical implementation data. For example, one program improves the assessment value of similar characteristic subjects by 30%, while another program improves by 15%. The quantitative scoring of these two indicators forms complete program evaluation data, providing an objective basis for subsequent selection.

[0064] First, a preset standard is set, which includes an upper limit for implementation complexity, such as no need for full-time professional follow-up, and a lower limit for expected effect value, such as a specific quantitative indicator that the improvement rate of state assessment value is not less than 20%. The assessment data of each candidate intervention plan is compared with the preset standard one by one, and the plans that do not meet the assessment data are eliminated. Finally, the plan with assessment data that is better than the standard is extracted as the appropriate intervention strategy for the current subject to be assessed, so as to ensure that the selected intervention plan is not only operable, but can also effectively promote the transformation of the subject to be assessed towards a healthier state.

[0065] Electronic devices, including memory, processor, and computer programs stored in memory and executable on the processor, wherein the processor executes the program to implement a conversation-based multimodal feature analysis method.

[0066] like Fig. 3 As shown, the electronic device may include a processor 610, a communication interface 620, a memory 630, and a communication bus 640, wherein the processor 610, the communication interface 620, and the memory 630 communicate with each other via the communication bus 640. The processor 610 can call logical instructions in the memory 630 to execute a conversation-based multimodal feature analysis method.

[0067] Furthermore, the logical instructions in the aforementioned memory 630 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory, random access memory, magnetic disks, or optical disks.

[0068] On the other hand, the present invention also provides a computer program product comprising a computer program that can be stored on a non-transitory computer-readable storage medium, wherein when the computer program is executed by a processor, the computer is capable of performing a conversation-based multimodal feature analysis method.

[0069] In another aspect, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to perform a conversation-based multimodal feature analysis method.

[0070] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0071] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0072] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A conversation-based multimodal feature analysis method, characterized in that, The method includes the following steps: Collect multimodal evaluation data of the conversation process of the subject to be evaluated, and perform feature labeling on the multimodal evaluation data to obtain state feature points; Feature association parameters are obtained by analyzing the frequency and intensity of similar features among statistical state feature points. Extract the distribution patterns of historical language state features under different assessment scenarios from the historical assessment dataset, and calculate the language state influence factor between the language state tendency value and the feature association parameter in the historical assessment data based on the distribution patterns of historical language state features. Based on the real-time language state feature distribution characteristics of the current conversation scenario, target influencing factors are selected from the language state influencing factors. Based on the target influencing factors, the core language state tendency value and auxiliary language state tendency value of the subject to be evaluated in the conversation process are determined, and the first language state evaluation value and the second language state evaluation value are output. The current status assessment value is obtained by judging the health status value of the assessed object based on the first language status assessment value and the second language status assessment value; The intervention strategy for the subject to be evaluated is determined based on the current status assessment value.

2. The conversation-based multimodal feature analysis method according to claim 1, characterized in that, The multimodal evaluation data includes voice data, facial expression data, and text response data.

3. The conversation-based multimodal feature analysis method according to claim 2, characterized in that, The process of labeling multimodal evaluation data to obtain state feature points includes the following steps: Language state features are obtained by labeling speech data with intonation and speech rate features; Facial expression data is labeled with facial expression action features to obtain facial expression language state features; Semantic and linguistic state features are used to label the text response data to obtain the text response linguistic state features. Among them, the language state features, facial expression language state features, and text response language state features constitute state feature points.

4. The conversation-based multimodal feature analysis method according to claim 1, characterized in that, The feature association parameters are obtained by analyzing the frequency and intensity of similar features among statistical state feature points. This process includes the following steps: The frequency of occurrence of a feature is obtained by counting the frequency of occurrence of the same type of feature among state feature points within a preset time window; The feature strength value is obtained by calculating the degree of linguistic state expression of the state feature points; Feature association parameters are obtained by mapping the frequency of feature occurrence with the feature intensity value.

5. The conversation-based multimodal feature analysis method according to claim 1, characterized in that, The calculation of the language state influence factor between the language state tendency value and the feature correlation parameter in historical assessment data based on the distribution pattern of historical language state characteristics includes the following steps: Based on the distribution pattern of historical language state characteristics, historical language state tendency values ​​corresponding to language state tendency labels are extracted from historical assessment data. The language state difference value is obtained by calculating the language state tendency difference corresponding to the same type of feature in the historical language state tendency value based on the feature association parameter; The feature ratio value is obtained by proportionally calculating the language state difference value and the feature association parameter. The language state influence factor is obtained by weighted summation of all feature proportions.

6. The conversation-based multimodal feature analysis method according to claim 1, characterized in that, Based on the real-time language state feature distribution characteristics of the current conversation scenario, target influencing factors are selected from the language state influencing factors. The specific steps include: Obtain the real-time language state feature distribution characteristics of the current conversation scene; After performing similarity matching between the real-time language state feature distribution and the historical language state feature distribution, the target influence factor is selected from the language state influence factors.

7. The conversation-based multimodal feature analysis method according to claim 6, characterized in that, After determining the core language state tendency and auxiliary language state tendency of the subject to be evaluated during the conversation based on the target impact factor, the first language state evaluation value and the second language state evaluation value are output. The specific steps include: Determine the core and auxiliary language state expression dimensions of the subject being evaluated during the conversation; Determine the first feature set corresponding to the core language state and the second feature set corresponding to the auxiliary language state; After determining the core language state tendency value based on the feature association parameters corresponding to the first feature set, the real-time language state strength value, and the target influence factor, the first language state evaluation value is output. Based on the feature association parameters corresponding to the second feature set, the real-time language state strength value, and the target influence factor, the auxiliary language state tendency value is determined and the second language state evaluation value is output.

8. The conversation-based multimodal feature analysis method according to claim 7, characterized in that, The current status assessment value is obtained by judging the health status of the assessed object based on the first language status assessment value and the second language status assessment value. The specific steps include: Assign core weight values ​​to the first language status assessment values; Assign auxiliary weight values ​​to the second language status assessment values; The first evaluation factor is obtained by multiplying the first language status evaluation value by the core weight value; The second evaluation factor is obtained by multiplying the second language status assessment value by the auxiliary weight value; The comprehensive status assessment factor is obtained by summing the first assessment factor and the second assessment factor. Obtain a reference assessment database of state health values ​​corresponding to different combinations of language state tendencies and feature intensities; The current status assessment value is obtained by matching the comprehensive status assessment factors with the reference assessment database.

9. The conversation-based multimodal feature analysis method according to claim 1, characterized in that, Determine the intervention strategy for the subject to be evaluated based on the current status assessment value, specifically including the following steps: If the current status assessment value is lower than the status warning threshold, an intervention prompt message will be output; After calling the preset intervention plan library based on the intervention prompt information, candidate intervention plans that match the current language state feature distribution features are extracted; Evaluation data is obtained by assessing the implementation complexity and expected outcome of candidate intervention programs. The best-fit intervention strategy is one that extracts evaluation data from candidate intervention programs and finds that outperforms the pre-defined intervention standards.

10. An electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the conversation-based multimodal feature analysis method as described in any one of claims 1 to 9.

Images