An intelligent interactive system

Abstract

The application relates to the technical field of intelligent interaction, in particular to an intelligent interaction system, which comprises: a data acquisition module, which is used for acquiring multi-modal interaction data of an interaction party and a target user in a current interaction context; an interaction state analysis module, which is used for state modeling according to multi-modal interaction data of the current interaction and historical interactions, determining a response mode of the target user in the current interaction, predicting a corresponding expression intention of the target interaction object under the response mode according to a dynamic context library, judging cognitive load and emotional state of the target user according to multi-modal interaction data of the target user, and jointly determining a tone intention of the interaction party according to at least two of text intention, spoken language attitude intention and body language intention of the interaction party; and an interaction module, which is used for determining an interaction strategy according to the response mode, the expression intention, the emotional state, the cognitive load and the tone intention. Two-way and controllable social support in a natural context interaction process is realized.

Description

Technical Field

[0001] This application relates to the field of intelligent interaction technology, and in particular to an intelligent interaction system. Background Technology

[0002] In natural communication, some users often have difficulty understanding the linguistic intentions of others, grasping changes in context, and dynamically adjusting their response strategies according to the other party's reactions. Furthermore, their communication state exhibits significant situational dependence and volatility.

[0003] To improve the communication skills of these individual users, existing technologies have proposed a variety of assistance solutions, mainly including expression assistance tools based on images, text or voice, social skills interaction systems based on fixed scenarios, and communication assistance systems that provide prompts or guidance during communication.

[0004] The aforementioned technologies have some effect in structured interaction scenarios, but they still have obvious limitations in natural social environments. For example, it is difficult to perceive the communication status and changes of such individual users in real time. Most systems only trigger prompts based on explicit behavior or single modal information. Therefore, there is an urgent need for a social communication interaction intelligent support system that can comprehensively perceive the communication status, degree of understanding of intent, and emotional risk of such users in natural social scenarios, and implement dynamic regulation and psychological protection for both parties in the communication. Summary of the Invention

[0005] The purpose of this application is to provide an intelligent interaction system that collects multimodal sensor data from both parties during the interaction process, performs state modeling, dynamically learns the interaction state and interaction intention of both parties or one party, and thus realizes two-way, controllable social support in the natural context interaction process.

[0006] In some embodiments, this application provides an intelligent interaction system, the system comprising: a data acquisition module, configured to acquire multimodal interaction data between the interacting party and the target user in the current interaction context; an interaction state analysis module, configured to perform state modeling based on the multimodal interaction data of the current round of interaction and historical rounds of interaction, determine the response pattern of the target user in the current round of interaction, predict the expressive intent of the target interactive object corresponding to the response pattern based on a dynamic context library, determine the cognitive load and emotional state of the target user based on the multimodal interaction data of the target user, and determine the tone intention of the interacting party based on at least two of the textual intent, verbal attitudinal intent, and body language intent; and an interaction module, configured to determine an interaction strategy based on at least one or more of the response pattern, the expressive intent, the emotional state, the cognitive load, and the tone intention.

[0007] In some embodiments, the interaction state analysis module further includes: a state modeling unit, used to perform sentence similarity analysis on the target user's response data in the current round and response data in historical rounds, perform similarity analysis on the target user's response data in the current round and interaction topic and historical context in the current round, classify the target user's response pattern in the current round in combination with the target user's physiological state in the current round, and predict the expressive intent of the target interaction object corresponding to the response pattern according to the dynamic context library.

[0008] In some embodiments, the interaction state analysis module further includes: a text intent calculation unit, used to perform semantic analysis on the multimodal interaction data of the interacting party in this round of interaction to obtain the text intent of the interacting party; a spoken attitude intent calculation unit, used to analyze the voice information in the multimodal interaction data to obtain the spoken attitude intent of the interacting party; a spoken consistency verification unit, used to perform consistency verification between the text intent and the spoken attitude intent; a first response unit, connected to the spoken consistency verification unit, used to, in response to the spoken consistency verification result satisfying a first preset condition, take the text intent as the tone intent of the interacting party; and a second response unit, connected to the spoken consistency verification unit, used to, in response to the spoken consistency verification result satisfying a second preset condition, determine that the interacting party is in a conflict state in this round of interaction, and determine the tone intent of the interacting party based on the body language intent, text intent, and spoken attitude intent of the interacting party.

[0009] In some embodiments, the interaction state analysis module further includes: a body language intent calculation unit, connected to the spoken language consistency verification unit, configured to analyze image data in multimodal interaction data to obtain the body language intent of the interacting party in response to the spoken language consistency verification result satisfying a second preset condition; a body consistency verification unit, configured to perform consistency verification between the text intent and the body language intent; a third response unit, connected to the body consistency verification unit, configured to use the body language intent as the tone intent of the interacting party in response to the body consistency verification result satisfying a third preset condition; and a fourth response unit, connected to the body consistency verification unit, configured to use the text intent as the tone intent of the interacting party in response to the body consistency verification result satisfying a fourth preset condition.

[0010] In some embodiments, the interaction state analysis module further includes: a psychological state assessment unit, used to compare the physiological signals in the multimodal interaction data with the resting-state baseline features to calculate the deviation, and to quantify the cognitive load and emotional state of the target user based on the deviation.

[0011] In some embodiments, the interaction state analysis module further includes: a baseline feature model construction unit, used to collect physiological sensor data of the target user, extract cognitive load feature indicators and emotional feature indicators from the physiological sensor data, and determine the degree of body muscle tension based on the cognitive load feature indicators and emotional feature indicators to obtain a resting state baseline feature model.

[0012] In some embodiments, the interaction module further includes: a joint judgment unit, configured to determine whether the target user responds in the current round; a fifth response unit, configured to calculate the cognitive load of the target user in response to the first response judgment result satisfying a fifth preset condition; a sixth response unit, configured to determine the interaction strategy as providing the target user with the tone and intent of the interacting party and a response template in response to the first cognitive load judgment result satisfying a sixth preset condition; a seventh response unit, configured to further judge the emotional state of the risky interaction in response to the second cognitive load judgment result satisfying a seventh preset condition; an eighth response unit, configured to determine the interaction strategy as providing reassurance to the target user and deciding whether to end the current dialogue in response to the emotional state judgment result satisfying an eighth preset condition; and a ninth response unit, configured to determine the interaction strategy as prompting the target user for a response in response to the emotional state judgment result satisfying a ninth preset condition, and continuing to acquire multimodal interaction data for the next round.

[0013] In some embodiments, the interaction module further includes: a tenth response unit, configured to perform echo detection to determine the target user's response pattern in response to the second response judgment result satisfying the tenth preset condition; an eleventh response unit, configured to determine the target user's cognitive load in response to the first response pattern detection result satisfying the eleventh preset condition; a twelfth response unit, configured to determine the interaction strategy as extracting the interaction party's tone intention and providing the target user with the interaction party's tone intention and response template in response to the third cognitive load judgment result satisfying the twelfth preset condition; a thirteenth response unit, configured to determine whether it is a delayed echo pattern in response to the second response pattern detection result satisfying the thirteenth preset condition; and a fourteenth response unit, configured to determine the interaction strategy as extracting the target user's response keywords and generating a target user's intent candidate set by combining historical interaction data in response to the delayed echo judgment result satisfying the fourteenth condition, selecting the target intent from the target user's intent candidate set and determining the response template and providing it to the target user, and continuing to acquire multimodal interaction data in the next round if it is not a delayed echo pattern.

[0014] In some embodiments, the system further includes: a physiological state trend calculation module, configured to determine the interaction time window corresponding to each round, acquire the physiological signal characteristics of the target user within the interaction time window, compare the physiological signal characteristics with the current resting-state baseline feature model, and generate the physiological state and physiological signal change trend of the target user in multiple rounds; a fifteenth response unit, configured to prompt that there is a risk in the interaction or stop the current interaction and update the user context database in response to the physiological signal change trend judgment result meeting the fifteenth preset condition; and a sixteenth response unit, configured to continue to analyze the multimodal interaction data according to the interaction state analysis module in response to the physiological signal change trend judgment result meeting the sixteenth preset condition.

[0015] In some embodiments, the system further includes an interaction analysis module, which is used to parse the identity information, tone of voice, and expression risk index of the interaction party, and determine the interaction strategy based on the parsed information. The interaction analysis module includes an interaction party attribute identification unit, an intent parsing unit, and an expression risk assessment unit. The interaction party attribute identification unit is used to compare the image data and / or audio data in the multimodal interaction data with a preset feature library to determine the identity information of the current interaction party. The intent parsing unit is used to parse the audio data in the multimodal interaction data using natural language processing technology to parse the tone of voice of the interaction party. The expression risk assessment unit is used to quantify and extract the multimodal expression features of the multimodal interaction data in the current interaction context to obtain the current feature vector system corresponding to the current expression mode, and obtain the physiological state level changes of the target user in similar contexts to the current interaction context based on the current feature vector system. Based on the physiological state level changes of the target user in one or more similar contexts, the potential risk score corresponding to the current expression mode is calculated.

[0016] Through the above embodiments, the intelligent interaction system provided by this application collects multimodal sensor data of both parties during the interaction process and performs state modeling to dynamically obtain information such as the interaction state, interaction intention and emotional state of both parties or one party, and then determines the next interaction strategy to achieve two-way and controllable social support in the natural situation interaction process. Attached Figure Description

[0017] The above and / or additional aspects and advantages of this application will become apparent and readily understood from the description of the embodiments taken in conjunction with the following drawings, wherein:

[0018] Figure 1 This is a schematic diagram of a module of an intelligent interactive system provided in one embodiment of this application;

[0019] Figure 2This is a flowchart illustrating the process of an intelligent interactive system prior to interaction, provided in one embodiment of this application.

[0020] Figure 3 This is a schematic diagram of the logic judgment of an intelligent interactive system provided in another embodiment of this application;

[0021] Figure 4 This is a schematic diagram of the structure of a computer device provided in one embodiment of this application. Detailed Implementation

[0022] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0023] The technical solutions of the various embodiments of this application can be combined with each other, but only if they are based on the ability of a person skilled in the art to implement them. When the combination of technical solutions is contradictory or cannot be implemented, it should be considered that such combination of technical solutions does not exist and is not within the scope of protection claimed by this application.

[0024] This solution does not aim to obtain disease diagnosis results or health status. Instead, it processes the multimodal interaction data generated by the interactive objects during the interaction process to achieve a more intelligent, efficient, and secure interactive system. All steps are information processing methods implemented by computers and other devices.

[0025] It should be fully understood that the user information involved in this application (including but not limited to user interaction data, such as image data, behavioral data, physiological sensor data, and user personal information) is all information and data authorized by the user or fully authorized by all parties. The use of user information shall comply with the privacy policies and practices of the industry that are generally considered to meet or exceed the requirements for maintaining user privacy. The collection, use and processing of related data shall comply with relevant laws, regulations and standards, and provide corresponding operation access points for users to choose to authorize or refuse.

[0026] It should be noted that the target user in this application is a user who needs external assistance during the interaction process. For example, in natural communication, the target user often exhibits difficulty in understanding the intentions of others' language, difficulty in grasping changes in context, and difficulty in dynamically adjusting response strategies based on the other party's reactions. Furthermore, their communication state shows significant situational dependence and volatility. In the following embodiments, the target user can be a user with Autism Spectrum Disorder (ASD). However, ASD users are only used as one embodiment for illustration. In other embodiments, other types of users can also be used, such as normal users who are younger, etc., and this is not a limitation. The embodiments provided in this application can ensure a safe interaction process between the target user and the interacting party.

[0027] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, specific embodiments of this application will be described in detail below with reference to the accompanying drawings.

[0028] In some embodiments, see Figure 1 This application provides an intelligent interaction system 100, which includes a data acquisition module 101, an interaction state analysis module 102, and an interaction module 103.

[0029] The data acquisition module 101 is used to acquire multimodal interaction data between the interacting party and the target user in the current interaction context. The interaction state analysis module 102 is used to perform state-based modeling based on the multimodal interaction data of the current and historical interactions, determine the target user's response pattern in the current interaction, predict the target user's expressive intent corresponding to the response pattern based on a dynamic context database, determine the target user's cognitive load and emotional state based on the target user's multimodal interaction data, and determine the interacting party's tone intention based on at least two of the interacting party's textual intent, verbal attitudinal intent, and body language intent. The interaction module 103 is used to determine an interaction strategy based on at least one or more of the response pattern, expressive intent, emotional state, cognitive load, and tone intention through joint analysis.

[0030] In some embodiments, if the target user's interaction capabilities are insufficient, external intervention is required to ensure their safety during the interaction process.

[0031] The interaction context can be a dialogue scenario, which includes two parties interacting, specifically the interacting party and the target user. In some embodiments of this application, the interaction state is obtained by analyzing the interaction data of the two parties, and the interaction strategy between the interacting party and the target user is dynamically adjusted according to the interaction state, so that the two parties can continue to interact and have dialogue in a relatively safe interaction environment.

[0032] Specifically, the intelligent interaction system 100 provided in this application includes a data acquisition module 101, which is used to synchronously acquire multimodal interaction data generated when the communication object (interaction party) and the target user conduct dialogue interaction in the current interaction context (e.g., a natural communication scenario), wherein the multimodal interaction data can be multi-source data.

[0033] In some embodiments, the multimodal interaction data collected by the data acquisition module 101 may include image data, audio data, motion state data, and physiological sensor data. The collected image data is primarily used to obtain the identity attribute information of the interacting parties (e.g., communication partners), to determine the relationship between the interacting parties and the target user (e.g., whether the interacting party is an acquaintance or stranger to the target user), and to determine the body posture and facial expressions of both parties, providing a data foundation for subsequent analysis of the expressive methods, emotional intensity, and potential communication risks of the interacting parties. Audio data is primarily used to collect the speech content and speech features of both parties to support dialogue parsing, echo language detection, and semantic intent analysis. Motion state data is primarily used to reflect changes in the target user's physical activity and avoidance behavior characteristics. Physiological sensor data mainly includes electroencephalography (EEG), functional near-infrared spectroscopy (FIR), electrocardiography (ECG), electromyography (EMG), and skin conductance, used to characterize the target user's cognitive load, emotional fluctuations, and stress levels.

[0034] The interaction state analysis module 102, as a data analysis module, is the core module of this application. It analyzes the multimodal interaction data collected by the data acquisition module 101 to obtain the interaction state of the two parties during the interaction process. Specifically, it can perform channel-specific modeling and joint analysis on the collected multimodal interaction data to achieve the identification of the communication state of the two parties, intent parsing, and risk assessment.

[0035] In some embodiments, the interaction state analysis module 102 includes multiple functional sub-modules for processing the interaction state information of both parties and making matching and interaction strategy decisions based on the obtained interaction state information. For example, in some embodiments, the system further includes an interaction party analysis module, which is used to parse the identity information, tone of voice, intent, and expression mode risk index of the interaction party, and determine the interaction strategy based on the parsed information. The interaction party analysis module may further include an interaction party attribute identification unit, an intent parsing unit, and an expression mode risk assessment unit. The interaction attribute identification unit is used to compare the image data and / or audio data in the multimodal interaction data with a preset feature library to determine the identity information of the current interaction party; the intent parsing unit is used to parse the audio data in the multimodal interaction data using natural language processing technology to parse the tone and intent of the interaction party; the expression mode risk assessment unit is used to quantify and extract the multimodal expression features of the multimodal interaction data in the current interaction context to obtain the current feature vector system corresponding to the current expression mode, and obtain the physiological state level changes of the target user in similar situations to the current interaction context based on the current feature vector system, and calculate the potential risk score corresponding to the current expression mode based on the physiological state level changes of the target user in one or more similar situations.

[0036] Specifically, the interaction attribute identification unit is used to identify the attributes of the interaction party, such as whether the interaction party is an acquaintance or a stranger of the target object. The intent parsing unit is used to parse the interaction party's expressive intent. The expression method risk assessment unit is used to assess the risk of the interaction party's expression method.

[0037] In a specific embodiment, the interaction attribute identification unit is used to identify the identity attributes of the interaction party. Specifically, it can compare image data (e.g., facial recognition data) or audio data (e.g., voiceprint recognition data) from the multimodal interaction data collected by the data acquisition module with a pre-stored database of acquaintance features of the target object to determine whether the current communication object (interaction party) is an acquaintance of the target user. It should be noted that this identification result is the primary key parameter for the system's personalized adaptation. The intent parsing unit can specifically use natural language processing technology to parse the semantic intent of the other party's statements. The expression mode risk assessment unit is used to assess the potential emotional stimulation or cognitive stress risk that the current expression mode of the interaction party (i.e., the target user's communication object) may cause to the target user. This unit first quantifies and extracts the multimodal expression features of the interaction party, i.e., the communication object, including speech rate, changes in voice tone, facial expression tension, changes in body posture, and gesture amplitude, forming a feature vector of the current expression mode. Based on the feature vector, the system retrieves historical context records similar to the current expression mode from the user context database and obtains the changes in the target user's physiological state level under the corresponding context. By statistically analyzing the magnitude of changes in the target user's state in multiple similar scenarios, the system calculates the risk score of emotional or cognitive stress that the current expression may trigger, and maps this score to a preset risk level. When the risk level is detected to exceed the set threshold, the system outputs prompt information corresponding to the specific risk type to the target user or the interacting party, in order to guide the adjustment of the dialogue rhythm or expression, thereby reducing the risk of negative stimuli in the communication process, such as "Please slow down your speech, I can't understand."

[0038] In some embodiments, the system further includes a target user state modeling module, which is used to identify the target user's communication behavior state and internal psychological state in real time. Specifically, the target user state modeling module includes a behavior state classification unit, a psychological state assessment unit, and an echo language decoding unit. The behavior state classification unit is used to detect response patterns based on the audio stream and classify the target user's behavior state into normal response, immediate echo, delayed echo, silence, and overexpression. The psychological state assessment unit is used to compare the target user's physiological signals (EEG frequency band energy, heart rate variability, skin conductance response) with baseline values ​​in the target user's individual baseline feature library (e.g., the resting state baseline feature library baseline_rest during rest and relaxation and the baseline baseline_social during interactive states), calculate the deviation, and quantitatively assess the target user's cognitive load, stress level, and emotional valence (positive or negative). The echo language decoding unit is used to decode the immediate and delayed echo categories in conjunction with a user context library. Immediate echoes may indicate "confusion" or "being processed"; delayed echoes infer the underlying intent that the current user may be expressing (such as "I want to do something I did before") by matching keywords with historical successful interaction scenarios in a context library.

[0039] The system also includes an interaction module 103, which has feedback and intervention functions. Specifically, the interaction module 103 is used for matching analysis and decision-making. Based on the communication status, intent matching results, and risk level output by the data analysis module, this module dynamically provides corresponding prompts or interventions to the target user or interacting party. In some embodiments, the interaction module can be a rule engine or a lightweight decision model. Based on the analysis results obtained by the above module, the interaction strategy is determined according to the decision logic designed by the interaction module. The decision logic can be simplified to the following judgment matrix.

[0040] Condition A (Intent Understanding Match): Whether the target user's response and behavior are consistent with the current dialogue intent. Condition B (Psychological State Safety): Whether the target user's cognitive load and stress level are within a safe threshold. Condition C (Expression Mode Safety): Whether the risk level of the interacting party's expression mode is within an acceptable range. Furthermore, different strategies in the feedback / intervention module are triggered based on different Boolean value combinations of (A, B, C).

[0041] When the system determines that the target user has the willingness to continue communicating but has difficulty communicating, it provides them with intent prompts, response suggestions, or structured social script support. When the system detects communication risks or that the target user is not suitable to continue communicating, it triggers reassurance prompts, pace adjustments, content neutralization, or outputs suggestions for adjusting the expression style to the communication object, so as to achieve two-way control of social interaction.

[0042] In some embodiments, this application also constructs a user-related context database, which may include historical interaction context information as well as dynamically updated interaction context information. Specifically, the target user's context database is used to store and manage context information related to the target user's past communication experiences, providing a reference for the system to understand the current communication state and infer the individual's true intentions. This context database uses "communication context" as the basic recording unit to structurally preserve the interaction process that has occurred. Each historical context record includes at least: the type of communication scenario, characteristics of the dialogue content, characteristics of the communication partner's expression style, the target user's state performance in the corresponding turn, and the actual reaction to the dialogue result or subsequent confirmation.

[0043] The above information collectively describes the correspondence between external expressions and the target user's internal reactions in a specific communication context. When the system analyzes the current dialogue, it searches the user's historical context database for historical records similar to the current situation in terms of scenario type, expression, or interaction form to obtain the target user's typical reaction patterns in similar situations. Based on these historical correspondences, the system can interpret echolalia, simplified responses, or ambiguous expressions appearing in the current turn and infer their possible true intentions or understanding states, thus providing a basis for subsequent prompts or communication adjustments. Because different target users exhibit significant differences in social sensitivity, stress tolerance, and expression habits, the user historical context database is established and continuously updated on an individual basis, enabling the system to make judgments based on the individual's own historical reaction characteristics, rather than relying on general rules. This context database can be configured by the target user or their familiar caregiver during initial system use and gradually supplemented and improved during actual use.

[0044] In some embodiments, the interaction state analysis module further includes: a baseline feature model construction unit, used to collect physiological sensor data of the target user, extract cognitive load feature indicators and emotional feature indicators from the physiological sensor data, and determine the degree of body muscle tension based on the cognitive load feature indicators and emotional feature indicators to obtain a resting state baseline feature model.

[0045] In a specific embodiment, the individual baseline acquisition and processing process for the target user includes the following steps: Under relaxed conditions, physiological sensor data such as electroencephalogram (EEG), electrocardiogram (ECG), and skin conductance (SC) are collected. After preprocessing the EEG, relevant indicators such as cognitive load and emotion are extracted, including alpha band power and the theta / alpha ratio. Bandpass filtering is used to retain high-frequency components (30-150Hz) of the EEG, and the total power, root mean square, and variance of the high-frequency components are calculated to indirectly characterize the degree of muscle tension. The HRV index is calculated using ECG to form a resting-state baseline feature library (baseline_rest), thereby establishing a resting-state baseline feature model.

[0046] See Figure 3 , Figure 3 This is a schematic diagram of the logic judgment of an intelligent interactive system provided in one embodiment of this application. The following embodiments of this application will incorporate... Figure 3 This application provides a detailed description of the logical judgment process of the intelligent interactive system. It should be noted that the first to sixteenth preset conditions disclosed in the following embodiments of this application are coordinated through a clear decision-making process. Their relationship is not isolated, but is specifically reflected in the following three levels: First, there is a dependency relationship in the judgment order. Higher-level conditions are prerequisites for triggering lower-level judgments. Second, there is an input-output relationship in information transmission. The judgment result of the preceding condition will serve as a key parameter, directly inputting into the evaluation of subsequent conditions. Multiple preset conditions jointly influence the subsequent comprehensive evaluation of cognitive load and emotional state. Finally, there is a shared decision-making relationship in functional output. The judgment results of all relevant conditions will converge along the path in the flowchart, ultimately generating a unified control command. This command is the result of all conditions coordinating according to the aforementioned established logical process; its technical effect stems from the structured combination of conditions, rather than any single condition. This complete collaborative logic is detailed in the appendix to the specification. Figure 3 The decision-making flowchart shown below clearly illustrates this. Figure 3 The interaction logic, including the first to the sixteenth preset conditions, is explained in detail.

[0047] In some embodiments, the system further includes: a physiological state trend calculation module, configured to determine the interaction time window corresponding to each round, acquire the physiological signal characteristics of the target user within the interaction time window, compare the physiological signal characteristics with the current resting-state baseline feature model, and generate the physiological state and physiological signal change trend of the target user in multiple rounds; a fifteenth response unit, configured to prompt that there is a risk in the interaction or stop the current interaction and update the user context database in response to the physiological signal change trend judgment result meeting the fifteenth preset condition; and a sixteenth response unit, configured to continue to analyze the multimodal interaction data according to the interaction state analysis module in response to the physiological signal change trend judgment result meeting the sixteenth preset condition.

[0048] See Figure 3 Specifically, the fifteenth preset condition could be: if the physiological signal shows a worsening trend and exceeds a preset number of rounds, then a risk is detected in the interaction, the current interaction is stopped, and the user context database is updated. The sixteenth preset condition could be: if the physiological signal shows a worsening trend but has not exceeded a preset number of rounds, then the multimodal interaction data continues to be analyzed according to the data analysis module. It should be noted that the fifteenth and sixteenth preset conditions are two different branches generated by the intelligent interaction system when determining whether the worsening trend of the physiological signal has reached a preset number of rounds, and there is a mutual exclusion relationship between the fifteenth and sixteenth preset conditions.

[0049] In one embodiment, the process of collecting and modeling target user social state data includes the following steps: When the target user engages in a structured dialogue with others (interacting parties), the target user's physiological sensor data is collected synchronously and recorded using tags, including the start or end of the dialogue, topic switching, and successful response times. The degree of change in physiological characteristics during the dialogue compared to the resting-state baseline feature model (resting baseline) is calculated, and a normal fluctuation range threshold is set as t1 based on the mean and standard deviation of the social baseline feature distribution. During each turn management process, after initiating a social dialogue, the system calls the dialogue management service module to implement dialogue start detection and turn sequence management. The dialogue management service module can be an existing large-scale model dialogue framework or voice interaction system, used to output structured turn information turn_i={turn_id,speaker,text_content,semantic_embedding,prosody_features,start_time,end_time}, where turn_id is the turn index, speaker indicates the speaker type of the current turn (target user, communication object such as the interacting party, or system), text_content is the speech recognition or text input content, semantic_embedding is the text semantic representation generated by the large model, and prosody_features includes information such as pitch, speech rate, pauses, and emotion category, used for subsequent confirmation of speech intent.

[0050] Furthermore, in some embodiments, the response time of the target user can be calculated based on the turn sequence: response_latency_i = start_time(turn_i_ASD) - end_time(turn_{i-1}_other). The response time is used for subsequent silence detection. Next, the physiological state of the turn is calculated. In some embodiments, if the physiological threshold is exceeded by two turns, the interaction process for that turn is stopped. For each turn Turn_i, the system determines the start and end times of that turn and constructs a physiological signal window for the target user before and after this time period: phys_window_i = [start_time_i - δ_pre, end_time_i + δ_post], where δ_pre represents the prefetch time before the turn begins, and δ_post represents the physiological response delay after the turn ends. Features are extracted from physiological signals within the window and compared with the current resting-state baseline feature model to generate the corresponding physiological state information Phys_State_i={cognitive_load,stress_index,emotion_valence,muscle_tension,baseline_deviation_score} for that turn. The system continuously monitors the physiological signal trends of the target user for each turn to see if they worsen. If the worsening trend exceeds a set threshold t2 turns, the system prompts the ASD individual to express discomfort or terminates the conversation, and updates the user context database. If the physiological threshold of t2 turns is not exceeded in this turn, the Partner_Intent_i tone intent is calculated based on a combination of surface text intent, verbal attitude, and body language.

[0051] In some embodiments, the interaction state analysis module of this application further includes a state modeling unit, which is used to perform sentence similarity analysis on the response data of the target user in the current round and the response data in historical rounds, perform similarity analysis on the response data of the target user in the current round and the interaction topic and historical context in the current round, classify the response pattern of the target user in the current round in combination with the physiological state of the target user in the current round, and predict the expressive intent of the target interaction object corresponding to the response pattern according to the dynamic context library.

[0052] Specifically, the system detects the target user's response behavior patterns and echo language. Response patterns are categorized as normal, silent, immediate echo, delayed echo, and overexpression. For each target user's turn, the system detects the similarity between the target user's response and the statements of the previous N turns, the similarity between the target user's response and the current topic, and the similarity to historical context. Combined with the physiological state (Phys_State_i), the system classifies the response category for this turn as Behavior_i∈{NORMAL,SILENCE,IMMEDIATE_ECHO,DELAYED_ECHO,OVER_EXPRESSION}, where NORMAL represents the normal category, SILENCE represents the silent category, IMMEDIATE_ECHO represents the immediate echo category, DELAYED_ECHO represents the delayed echo category, and OVER_EXPRESSION represents the overexpression category.

[0053] In some embodiments, the interaction state analysis module further includes a method for determining the interaction intent of the interacting party. Specifically, a text intent calculation unit is used to perform semantic analysis on the multimodal interaction data of the interacting party in this round of interaction to obtain the text intent of the interacting party; a spoken attitude intent calculation unit is used to analyze the voice information in the multimodal interaction data to obtain the spoken attitude intent of the interacting party; a spoken consistency verification unit is used to perform consistency verification between the text intent and the spoken attitude intent; a first response unit, connected to the spoken consistency verification unit, is used to take the text intent as the tone intent of the interacting party in response to the spoken consistency verification result meeting a first preset condition; a second response unit, connected to the spoken consistency verification unit, is used to determine that the interacting party is in a conflict state in this round of interaction in response to the spoken consistency verification result meeting a second preset condition, and to determine the tone intent of the interacting party based on the body language intent, text intent, and spoken attitude intent of the interacting party.

[0054] Specifically, the first preset condition may be: the consistency between the textual intent and the spoken attitude intent passes the verification. The second preset condition may be: the consistency between the textual intent and the spoken attitude intent fails the verification. It should be noted that the first and second preset conditions are two different branches generated by the intelligent interaction system when judging the consistency between the textual intent and the spoken attitude intent, and there is a mutual exclusion relationship between the first and second preset conditions.

[0055] In some embodiments, the interaction state analysis module further includes a body language intent calculation unit connected to the spoken language consistency verification unit, used to analyze image data in multimodal interaction data to obtain the body language intent of the interacting party in response to the spoken language consistency verification result meeting a second preset condition; a body consistency verification unit used to perform consistency verification between the text intent and the body language intent; a third response unit connected to the body consistency verification unit used to take the body language intent as the tone intent of the interacting party in response to the body consistency verification result meeting a third preset condition; and a fourth response unit connected to the body consistency verification unit used to take the text intent as the tone intent of the interacting party in response to the body consistency verification result meeting a fourth preset condition.

[0056] Specifically, the third preset condition may be that the consistency between the body language intention and the textual intention is higher than the consistency between the spoken attitude intention and the textual intention. The fourth preset condition may be that the consistency between the body language intention and the textual intention is lower than the consistency between the spoken attitude intention and the textual intention. It should be noted that the third and fourth preset conditions are determined by the intelligent interaction system when judging body language intention, textual intention, and spoken attitude intention. Figure 3 The two different branches generated when the parties are consistent, and there is a mutual exclusion relationship between the third and fourth preset conditions.

[0057] In one specific embodiment, the interactive text and tone intention of the interacting party are parsed. Specifically, if the physiological threshold of two rounds has not been exceeded in the current round, the tone intention Partner_Intent_i of the interacting party is calculated by comprehensively considering the surface text intention, spoken attitude, and body language. Specifically, within the Turn_i alignment window, the system calls the background large model to parse the language text and speech signals of the interacting party (communication object). The system first performs semantic analysis on the text data Turn_i.text_content (which is the content of speech recognition or text input) collected by the interacting party in this round, generates semantic embedding through the large model, and infers the surface text intention Text_Intent_i of the interacting party, i.e., the communication object. For example, the text intention may specifically be affirmation, rejection, questioning, or suggestion. Simultaneously, the system analyzes the spoken attitude of the interacting party (i.e., the communication partner) by combining the prosody features of the speech signal, including pitch, speech rate, stress, pauses, and prolonged ending sounds, generating the spoken attitude Oral_Attitude_i and calculating the confidence score Conf_Oral_i. The system further evaluates the consistency between the text intent and the spoken attitude intent, calculating the consistency confidence score Conf_LM_i. When the text intent and spoken attitude are highly consistent, the system directly outputs Text_Intent_i as Partner_Intent_i. If there is a significant conflict between the text intent and the spoken attitude, the turn is marked as conflicting, and the system proceeds to the next step of further analysis combining the interacting party's body language to understand their actual intent.

[0058] In some embodiments, the step of confirming the intent of an interacting party using body language includes the following: Specifically, the system invokes a body language parsing module to assist in determining the true intent of the interacting party, i.e., the communication target. The system extracts head posture, gaze direction, body orientation, gestures, distance changes, and facial expression features from image or depth data, and generates a nonverbal attitudinal intent, i.e., body language intent Nonverbal_AttitudeIntent_i, and its confidence level Conf_Nonverbal_i. The system determines whether the body language supports the text intent or the verbal intent by comparing the consistency between the body language intent Nonverbal_AttitudeIntent_i and the text intent Text_Intent_i and the oral attitudinal intent Oral_Attitude_i. If the consistency between the body language intent and the text intent is high, the system sets the interacting party's intent Partner_Intent_i to the text intent Text_Intent_i and calculates the comprehensive confidence level Conf_Partner_i=f(Conf_LM_i, Conf_Nonverbal_i); otherwise, the system outputs the text intent along with a confidence level indicator.

[0059] See Figure 3 In some embodiments, the interaction module further includes a joint judgment unit, which is used to determine whether the target user responds in the current round; a fifth response unit, which is used to calculate the cognitive load of the target user in response to the first response judgment result satisfying the fifth preset condition; a sixth response unit, which is used to determine the interaction strategy of providing the target user with the tone and intent of the interacting party and a response template in response to the first cognitive load judgment result satisfying the sixth preset condition; a seventh response unit, which is used to further judge the emotional state of the risky interaction in response to the second cognitive load judgment result satisfying the seventh preset condition; an eighth response unit, which is used to determine the interaction strategy of providing reassurance to the target user and deciding whether to end the current dialogue in response to the emotional state judgment result satisfying the eighth preset condition; and a ninth response unit, which is used to determine the interaction strategy of prompting the target user for a response in response to the emotional state judgment result satisfying the ninth preset condition, and continuing to acquire multimodal interaction data for the next round.

[0060] Specifically, the fifth preset condition may be the absence of a response. The sixth preset condition may be that the cognitive load is higher than a preset value. The seventh preset condition may be that the cognitive load is not higher than a preset value. The eighth preset condition may be that the emotional state corresponds to an abnormal emotional state. The ninth preset condition may be that the emotional state corresponds to a normal emotional state. It should be noted that the fifth preset condition is a prerequisite for the sixth and seventh preset conditions. The judgment of the sixth and seventh preset conditions continues only if the fifth preset condition is met. The sixth and seventh preset conditions are two different branches generated by the intelligent interaction system when judging the magnitude of the cognitive load, and there is a mutual exclusion relationship between the sixth and seventh preset conditions. The seventh and eighth preset conditions are two different branches generated by the intelligent interaction system when judging the emotional state, and there is a mutual exclusion relationship between the eighth and ninth preset conditions.

[0061] See Figure 3 In some embodiments, the interaction module further includes a tenth response unit, which is used to perform echo detection to determine the target user's response pattern in response to the second response judgment result meeting the tenth preset condition; an eleventh response unit, which is used to determine the target user's cognitive load in response to the first response pattern detection result meeting the eleventh preset condition; a twelfth response unit, which is used to determine the interaction strategy as extracting the interaction party's tone intention and providing the target user with the interaction party's tone intention and response template in response to the third cognitive load judgment result meeting the twelfth preset condition; a thirteenth response unit, which is used to determine whether it is a delayed echo mode in response to the second response pattern detection result meeting the thirteenth preset condition; and a fourteenth response unit, which is used to determine the interaction strategy as extracting the target user's response keywords and generating a target user's intent candidate set by combining historical interaction data in response to the delayed echo judgment result meeting the fourteenth condition, selecting the target intent and determining the response template from the target user's intent candidate set and providing it to the target user, and continuing to acquire multimodal interaction data in the next round if it is not a delayed echo mode.

[0062] Specifically, the tenth preset condition could be: the target user responds. The eleventh preset condition could be: the target user's response pattern is an immediate echo pattern. The twelfth preset condition could be: cognitive load is higher than a preset value. The thirteenth preset condition could be: the target user's response pattern is not an immediate echo pattern. The fourteenth preset condition could be: the target user's response pattern is a delayed echo pattern. It should be noted that the tenth preset condition is used by the intelligent interaction system to determine whether the target user responds, and when a response is found, it continues to determine the target user's response pattern based on subsequent preset conditions. The eleventh and thirteenth preset conditions are two different branches generated by the intelligent interaction system when determining the target user's response pattern. Furthermore, the eleventh and thirteenth preset conditions are mutually exclusive, the twelfth and eleventh preset conditions are progressively dependent, and the fourteenth and thirteenth preset conditions are progressively dependent. In the fourteenth response unit, if the response type of this turn is silence or immediate response, continue to judge cognitive load. If it exceeds the threshold, simplify the communication object's intent according to the tone intent of Partner_Intent_i and give ASD prompt. If it does not exceed the threshold, maintain the dialogue without any operation.

[0063] See Figure 3 In one specific embodiment, if the target user's response mode in this round of interaction is either silence or immediate echo, the process further includes determining the target user's cognitive load and emotional state, and further determining the current interaction strategy based on the cognitive load and emotional state. Specifically, when the target user's Behavior_i = IMMEDIATE_ECHO or SILENCE, it is further determined whether the target user's cognitive load exceeds a threshold. If the cognitive load exceeds the threshold, it is determined that the target user has failed to understand the intention of the other party, and the interaction intention Partner_Intent_i of the other party is extracted. The interaction intention of the other party is simplified and made explicit, and a prompt is given to the target user. The interaction mode inference_Mode_i is set to the prompt "Prompt". If the target user's cognitive load does not exceed the threshold, it is determined that the target user is currently maintaining a conversational immediate echo, and the interaction mode inference_Mode_i is set to no operation "No_Operation".

[0064] Specifically, in the delayed echo speech intent parsing process provided in this application, if the current turn response type is a delayed response, the target user's response keywords are extracted, and historical events are retrieved to generate an intent candidate set. The true intent is then recovered through a confirmation strategy. Specifically, when Behavior_i = DELAYED_ECHO, the target user's response keywords are extracted, historical events are retrieved from the context database, an intent candidate set is generated, and the true intent is recovered through a confirmation strategy. Recovered_Intent_i, with inference_Mode_i set to "Confirm_Intent".

[0065] In some embodiments, the interaction state analysis module further includes a psychological state assessment unit, which is used to compare the physiological signals in the multimodal interaction data with the resting state baseline features to calculate the deviation, and to quantify the cognitive load and emotional state of the target user based on the deviation.

[0066] In some contexts, autism spectrum disorder (ASD) is a neurodevelopmental disorder characterized by impairments in social communication and social interaction. Individuals with ASD often exhibit difficulty understanding the linguistic intentions of others, struggling to grasp changes in context, and failing to dynamically adjust their response strategies based on the other person's reaction during natural communication. Furthermore, their communication style is significantly situation-dependent and volatile.

[0067] In real-world social situations, individuals with ASD not only experience difficulties in expression and comprehension, but also generally exhibit high sensitivity to language content, tone changes, and nonverbal cues. When the pace of a conversation is too fast, the semantics are complex, the emotional intensity is too high, or inappropriate expressions occur, individuals with ASD are prone to reactions such as anxiety, increased stress, emotional outbursts, or communication avoidance, leading to communication breakdowns or even negative social experiences.

[0068] To improve the communication skills of individuals with ASD, existing technologies have proposed various assistive solutions, mainly including image-, text-, or voice-based expression aids, social skills training systems based on fixed scenarios, and communication assistance systems that provide prompts or guidance during communication. While these technologies have shown some effectiveness in structured training scenarios, they still have significant limitations in natural social environments.

[0069] First, existing technologies struggle to perceive the communication state and its changes in individuals with ASD in real time. Most systems trigger prompts based solely on overt behavior or single-modal information, failing to distinguish between normal responses, silence, overexpression, and the echo language state unique to individuals with ASD, nor can they determine whether the individual truly understands the dialogue's intent. Traditionally, echo language in individuals with ASD is defined as "pathological, parrot-like, and clearly meaningless repetition." However, recent research indicates that 96% of echo language contains the true intentions of individuals with ASD and is not entirely meaningless; it should be considered an atypical communication pattern, a compensatory strategy for the individual's potential communication function.

[0070] Secondly, existing solutions lack a comprehensive assessment mechanism for communication willingness, cognitive load, and emotional risk. When individuals with ASD are under increased stress or emotional instability, the system may continue to push for communication tasks, leading to compulsive interactions and increasing psychological burden.

[0071] Furthermore, current technology lacks the ability to assess and protect the security of communication content. When the other party in the communication makes irritating, inappropriate, or emotionally risky expressions, the system cannot identify, buffer, or filter them, leaving individuals with ASD vulnerable to continuous negative stimulation without protection.

[0072] In addition, most existing assistive communication systems only provide one-way support to individuals with ASD. The other party in the communication cannot know the individual's current level of understanding, willingness to communicate, or psychological state, which leads to an imbalance in the communication rhythm, accumulation of misunderstandings, and seriously affects the quality of two-way interaction.

[0073] Therefore, there is an urgent need for a social communication support system that can comprehensively perceive the communication status, understanding of intent, and emotional risk of individuals with ASD in natural social scenarios, and implement dynamic regulation and psychological protection for both parties in the communication, so as to ensure the psychological safety and communication sustainability of individuals with ASD in real communication.

[0074] This invention aims to solve the technical problem of how to dynamically perceive the communication status and analyze the semantic intent of individuals with autism spectrum disorder in natural social scenarios, while ensuring their willingness to communicate and their psychological safety, so as to achieve two-way and controllable social communication support for both individuals with ASD and the other party in the communication.

[0075] To address the aforementioned technical issues, this invention proposes a social communication support system for autism spectrum disorder based on matching communication status with dialogue intent. Through multimodal interaction data collection and state-based modeling, it achieves bidirectional and controllable social support during natural communication.

[0076] The core technical idea of ​​this invention lies in state-based modeling of the communication behavior of individuals with ASD. Through audio, image, motion state, and physiological signal data, the response patterns of individuals with ASD are identified and classified, distinguishing their communication states into normal responses, immediate echo language, delayed echo language, silence, and overexpression. Cognitive load and stress levels are assessed in conjunction with changes in physiological signals. Intent parsing and contextual modeling of dialogue content are performed. By semantically parsing the statements of the other party in the communication, a structured representation of dialogue intent is constructed. Combined with dialogue turns, historical context, and a user intent context database, the current communication intent is modeled. A communication state and intent matching analysis mechanism is constructed. The response characteristics of individuals with ASD are matched and analyzed with dialogue intent to determine their level of understanding and communication readiness, thereby providing a basis for cue control decisions. A two-way cue control mechanism based on safety conditions is introduced. Intent cueing, response suggestions, or social script support are provided to individuals with ASD only when preset communication safety conditions are met. When it is determined that the individual is temporarily unsuitable to continue communicating, the system automatically provides the other party with status prompts or delay prompts, enabling two-way control of the interaction pace for individuals with ASD. When the dialogue content is determined to contain risky expressions, the system provides reassurance to the individual with ASD or suggests terminating the dialogue and alerts the other party to the risk. Through the synergistic effect of the above technical features, this invention constructs a closed-loop social communication support system centered on communication status perception, intent matching, and prompt control, effectively improving the quality of natural communication between individuals with ASD and others while ensuring their willingness to communicate and their psychological safety.

[0077] See Figure 2 and Figure 3 This is a flowchart of the logical judgment process of the intelligent interactive system provided in this application.

[0078] Step 1: Individual baseline collection and processing for target users (ASD users are used as an example in the following examples).

[0079] See Figure 2 Under relaxed conditions, extreme multimodal interaction data of individuals with ASD were collected, including physiological sensor data such as electroencephalogram (EEG), electrocardiogram (ECG), and skin conductance (SC). After preprocessing the EEG, relevant indicators such as cognitive load and emotion, such as alpha band power and theta / alpha ratio, were extracted. Bandpass filtering was used to retain high-frequency components (30-150Hz) of the EEG, and the total power, root mean square, and variance of the high-frequency components were calculated to indirectly characterize the degree of muscle tension. The HRV index was calculated through ECG, and a resting-state baseline feature library, baseline_rest, was formed.

[0080] Step 2: ASD social status data collection and modeling.

[0081] See Figure 2 In a structured dialogue between an individual with ASD and others, data on the normal social state of ASD were collected, and their physiological sensory data were collected simultaneously. The labels included the start / end of the dialogue, topic switching, and the time of successful response. The degree of change of physiological characteristics during the dialogue compared with the resting baseline was calculated. Based on the mean and standard deviation of the distribution of social baseline characteristics, the threshold of normal fluctuation range was set as t1.

[0082] Step 3: Turntable management.

[0083] After initiating a social conversation, see Figure 3 The system calls the dialogue management service module to realize the start detection of dialogue and the management of the turn sequence.

[0084] The dialogue management service module can be an existing large-scale model dialogue framework or voice interaction system. It is used to output structured turn information turn_i={turn_id, speaker, text_content, semantic_embedding, prosody_features, start_time, end_time}, where turn_id is the turn index, speaker indicates the speaker type of the current turn (ASD individual / communication object / system), text_content is the speech recognition or text input content, semantic_embedding is the text semantic representation generated by the large model, and prosody_features includes information such as pitch, speech rate, pauses, and emotion category, which are used to confirm subsequent speech intent.

[0085] The response time of an individual with ASD is calculated based on the turn sequence: response_latency_i = start_time(turn_i_ASD)-end_time(turn_{i-1}_other). The response time is used for subsequent silence detection.

[0086] Step 4: Calculate the physiological state of the conversation.

[0087] See Figure 3 After the social conversation begins, for each turn Turn_i, the system determines the start and end times of that turn and constructs a physiological signal window for the ASD individual before and after this time period: phys_window_i = [start_time_i - δ_pre ,end_time_i + δ_post], where δ_pre represents the prefetch time before the turn begins and δ_post represents the physiological response delay after the turn ends.

[0088] Features are extracted from physiological signals within the window and compared with the current baseline model to generate the physiological state corresponding to the turn: Phys_State_i={cognitive_load, stress_index, emotion_valence, muscle_tension, baseline_deviation_score}. The physiological signal change trend of each ASD individual is continuously monitored to see if it worsens. If the worsening trend exceeds the set threshold t2 turns, the ASD individual is prompted to express discomfort or the conversation is terminated, and the user context database is updated.

[0089] Step 5: Text and tone intention analysis.

[0090] Within the Turn_i alignment window, the system invokes a large background model to parse the language text and speech signals of the communication subject. The system first performs semantic analysis on Turn_i.text_content (the content input for speech recognition or text), generates semantic embeddings through the large model, and infers the surface textual intent (Text_Intent_i) of the communication subject, such as affirmation, rejection, questioning, or suggestion.

[0091] At the same time, the system combines the prosody features of the speech signal, including pitch, speech rate, stress, pauses and prolongation of the last syllable, to analyze the oral attitude of the communication subject, generate the oral attitude Oral_Attitude_i, and calculate the confidence level of the attitude Conf_Oral_i.

[0092] The system further assesses the consistency between textual intent and spoken attitude, calculating the consistency confidence score Conf_LM_i. When the textual intent and spoken attitude are highly consistent, the system directly outputs Text_Intent_i as Partner_Intent_i. If there is a significant conflict between the textual intent and spoken attitude, the turn is marked as conflicting, and the process proceeds to step 6, body language analysis, for further confirmation.

[0093] Body language-assisted intent confirmation: The system invokes the body language parsing module to help determine the true intent of the communication partner. The system extracts head posture, gaze direction, body orientation, gestures, distance changes, and facial expression features from image or depth data, and generates nonverbal attitude intent (Nonverbal_AttitudeIntent_i) and its confidence level (Conf_Nonverbal_i).

[0094] The system determines whether body language supports textual intent or spoken attitude by comparing the consistency of Nonverbal_AttitudeIntent_i with Text_Intent_i and Oral_Attitude_i. Figure 1 If the consistency is high, the system will set Partner_Intent_i to Text_Intent_i and calculate the overall confidence level Conf_Partner_i=f(Conf_LM_i,Conf_Nonverbal_i). Otherwise, the system will output the text intent with a confidence level hint.

[0095] Step 6: ASD response behavior patterns and echo language detection.

[0096] For each turn of an ASD individual, the system detects the similarity between the ASD response and the statements of the other party in the previous N rounds, the similarity between the ASD response and the current topic, and the similarity between the historical context. Combined with Phys_State_i, the system classifies the response of this turn into Behavior_i∈{NORMAL,SILENCE,IMMEDIATE_ECHO,DELAYED_ECHO,OVER_EXPRESSION}.

[0097] Step 7: Analyzing the linguistic intent of silence and immediate echo.

[0098] When Behavior_i = IMMEDIATE_ECHO or SILENCE, determine whether the cognitive load exceeds the threshold. If it exceeds the threshold, it is determined that the ASD individual has failed to understand the other party's intention. Extract Partner_Intent_i, simplify and make explicit the intention of the communication partner, and give the ASD a prompt. Set inference_Mode_i to "Prompt". If it does not exceed the threshold, it is determined that the ASD individual is maintaining a conversational immediate echo, and set inference_Mode_i to "No_Operation".

[0099] Step 8: Delayed echo language intent analysis.

[0100] When Behavior_i=DELAYED_ECHO, extract ASD response keywords, retrieve historical events from the context library, generate an intent candidate set, and recover the true intent Recovered_Intent_i through a confirmation strategy, with inference_Mode_i set to "Confirm_Intent".

[0101] Step 9: Controlled large model invocation.

[0102] The system constructs a large model input LLM_Input_i = {recent_dialogue, final_intent, Phys_State_i, Inference_Mode_i}, which is used to set the output complexity and type.

[0103] Step 10: Generate dual-channel output.

[0104] Provide simplified response options, visual cues, or read-aloud text to individuals with ASD; provide explanations of ASD intent, suggested response methods, or waiting prompts to the communication recipient.

[0105] Step 11: Risk assessment and contextual memory update.

[0106] The system continuously monitors the Phys_State sequence:

[0107] If the situation worsens, the dialogue is marked as risky, the dialogue topic, physiological trajectory, intention recovery result are saved, and the context library is updated based on the above results.

[0108] Step 12: Session termination and reassurance control.

[0109] When the conversation becomes unsustainable, prompt the person with ASD to end the conversation, guide them to breathe or pause, and suggest an understanding end.

[0110] It should be noted that, in the technical solutions provided in this application, natural communication scenarios do not refer to completely unstructured, highly socially pressured, or highly unfamiliar random interaction environments. Rather, they refer to actual communication scenarios in daily life with clear interaction goals, relatively clear dialogue roles, and no mandatory social requirements. Examples include, but are not limited to: daily communication with acquaintances or semi-acquaintances, functional dialogues related to learning or work, and language interactions in short-term service or collaborative scenarios. Individuals with autism spectrum disorder (ASD) do not exhibit significant avoidance behavior in all natural communication scenarios. Some experimental results indicate that a considerable number of ASD individuals possess the ability to actively participate in the aforementioned types of communication scenarios. Their main difficulty lies not in a lack of willingness to communicate, but in a misunderstanding of implied intentions, nonverbal cues, and the rhythm of interaction during the dialogue, leading to communication interruptions, misunderstandings, or increased emotional burden. For some ASD individuals who possess the ability and willingness to participate in social production and daily affairs, the main limitations they face in real-world environments do not stem from insufficient cognitive or operational abilities, but rather from differences in social interaction patterns compared to typical populations, limiting their participation in application scenarios requiring frequent communication and collaboration. Due to the aforementioned social communication barriers, individuals with ASD are currently able to participate stably in a relatively limited range of social activities and work scenarios, thus affecting their continued participation and ability to perform effectively in a wider range of real-world situations. The support system proposed in this application, through auxiliary analysis and prompting of multimodal information during communication, reduces the risk of communication misunderstandings and interaction interruptions without replacing individual autonomous decision-making. This enables individuals with ASD who possess the corresponding abilities and needs to achieve stable participation in a wider range of social production and daily collaborative scenarios.

[0111] It is understood that the computer device used to implement the user interaction system solution provided in this application can be a server, and its internal structure diagram can be as follows: Figure 4 As shown, the computer device includes a processor, memory, and a network interface connected via a system bus. The processor provides computing and control capabilities. The memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores an operating system, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage medium. The database stores relevant data. The network interface communicates with external terminals via a network connection. When the computer program is executed by the processor, it implements the method provided in this application.

[0112] Those skilled in the art will understand that Figure 4The structures shown are merely block diagrams of a portion of the structures related to the present application and do not constitute a limitation on the computer device to which the present application is applied. The computer device may include more or fewer components than shown in the figures, or combine certain components, or have different component arrangements. Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program may include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application may include at least one of non-volatile and volatile memory. Non-volatile memory may include read-only memory (ROM), magnetic tape, floppy disk, flash memory, or optical storage, etc. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM).

[0113] It should be understood that the processor mentioned in the embodiments of this application can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor can be a microprocessor or any conventional processor.

[0114] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0115] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. An intelligent interactive system, characterized in that, The system includes: The data acquisition module is used to acquire multimodal interaction data between the interacting parties and the target user in the current interaction context; The interaction state analysis module is used to perform state modeling based on multimodal interaction data from the current round of interaction and historical rounds of interaction, determine the response pattern of the target user in the current round of interaction, predict the expressive intent of the target interaction object corresponding to the response pattern based on the dynamic context library, determine the cognitive load and emotional state of the target user based on the multimodal interaction data of the target user, and determine the tone intent of the interaction party based on at least two of the textual intent, verbal attitudinal intent and body language intent of the interaction party. An interaction module is used to determine an interaction strategy based on at least one or more of the response pattern, the expressive intent, the emotional state, the cognitive load, and the tone intent through joint analysis.

2. The system according to claim 1, characterized in that, The interaction state analysis module also includes: The state modeling unit is used to perform sentence similarity analysis on the target user's response data in the current round and response data in historical rounds, perform similarity analysis on the target user's response data in the current round and interaction topic and historical context in the current round, classify the target user's response pattern in the current round in combination with the target user's physiological state in the current round, and predict the expressive intent of the target interaction object in the response pattern according to the dynamic context library.

3. The system according to claim 1, characterized in that, The interaction state analysis module also includes: The text intent calculation unit is used to perform semantic analysis on the multimodal interaction data of the interacting party in this round of interaction to obtain the text intent of the interacting party. The spoken attitude and intent calculation unit is used to analyze the speech information in multimodal interaction data to obtain the spoken attitude and intent of the interacting parties. A spoken language consistency verification unit is used to verify the consistency between the text intent and the spoken language attitude intent. The first response unit, connected to the spoken language consistency verification unit, is used to respond to the spoken language consistency verification result satisfying the first preset condition by taking the text intent as the tone intent of the interacting party. The second response unit, connected to the spoken language consistency verification unit, is used to determine that the interacting party is in a conflict state in this round of interaction in response to the spoken language consistency verification result meeting the second preset condition, and to determine the interacting party's tone intention based on the interacting party's body language intention, text intention, and spoken attitude intention.

4. The system according to claim 3, characterized in that, The interaction state analysis module also includes: A body language intent calculation unit, connected to the spoken language consistency verification unit, is used to analyze the image data in the multimodal interaction data to obtain the body language intent of the interacting party in response to the spoken language consistency verification result meeting the second preset condition. A body consistency verification unit is used to verify the consistency between the text intent and the body language intent. The third response unit, connected to the body consistency verification unit, is used to respond to the body consistency verification result satisfying the third preset condition and to take the body language intention as the tone intention of the interacting party. The fourth response unit, connected to the body consistency verification unit, is used to take the text intent as the tone intent of the interacting party in response to the body consistency verification result satisfying the fourth preset condition.

5. The system according to claim 1, characterized in that, The interaction state analysis module also includes: The psychological state assessment unit is used to compare the physiological signals in the multimodal interaction data with the resting-state baseline features to calculate the deviation, and to quantify the cognitive load and emotional state of the target user based on the deviation.

6. The system according to claim 5, characterized in that, The interaction state analysis module also includes: The baseline feature model construction unit is used to collect physiological sensor data of the target user, extract cognitive load feature indicators and emotional feature indicators from the physiological sensor data, and determine the degree of muscle tension based on the cognitive load feature indicators and emotional feature indicators to obtain the resting state baseline feature model.

7. The system according to claim 1, characterized in that, The interaction module also includes: The joint judgment unit is used to determine whether the target user has responded in this round; The fifth response unit is used to calculate the cognitive load of the target user in response to the first response judgment result satisfying the fifth preset condition; The sixth response unit is used to determine the interaction strategy as providing the target user with the tone and intent of the interacting party and the response template in response to the first cognitive load judgment result satisfying the sixth preset condition; The seventh response unit is used to further determine the emotional state of risk interaction in response to the second cognitive load judgment result meeting the seventh preset condition; The eighth response unit is used to respond to the emotional state judgment result satisfying the eighth preset condition, determine the interaction strategy as providing comfort to the target user and decide whether to end the current dialogue; The ninth response unit is used to respond to the emotional state judgment result meeting the ninth preset condition, determine the interaction strategy as prompting the target user to respond, and continue to acquire the next round of multimodal interaction data.

8. The system according to claim 7, characterized in that, The interaction module also includes: The tenth response unit is used to perform echo detection to determine the target user's response mode in response to the second response judgment result meeting the tenth preset condition; The eleventh response unit is used to determine the cognitive load of the target user in response to the detection result of the first response mode meeting the eleventh preset condition. The twelfth response unit is used to respond to the third cognitive load judgment result satisfying the twelfth preset condition, and to determine that the target user has failed to recognize the tone and intent of the interacting party, and then to determine that the interaction strategy is to extract the tone and intent of the interacting party and provide the target user with the tone and intent of the interacting party and the response template. The thirteenth response unit is used to determine whether it is a delayed echo mode in response to the second response mode detection result meeting the thirteenth preset condition. The fourteenth response unit is used to respond to the delayed echo judgment result satisfying the fourteenth condition, determine the interaction strategy as follows: extract the target user's response keywords and combine them with historical interaction data to generate a target user's intent candidate set, select the target intent from the target user's intent candidate set and determine the response template and provide it to the target user; if it is not a delayed echo mode, it continues to acquire the next round of multimodal interaction data.

9. The system according to claim 1, characterized in that, The system also includes: The physiological state trend calculation module is used to determine the interaction time window corresponding to each round, obtain the physiological signal characteristics of the target user within the interaction time window, compare the physiological signal characteristics with the current resting state baseline feature model, and generate the physiological state and physiological signal change trend of the target user in multiple rounds. The fifteenth response unit is used to respond to the physiological signal change trend judgment result meeting the fifteenth preset condition, prompting that there is a risk in the interaction or stopping the current interaction, and updating the user context library. The sixteenth response unit is used to continue analyzing the multimodal interaction data according to the interaction state analysis module in response to the physiological signal change trend judgment result meeting the sixteenth preset condition.

10. The system according to claim 1, characterized in that, The system also includes an interaction analysis module, which is used to analyze the identity information, tone of voice, intention and expression risk index of the interaction party, and determine the interaction strategy based on the information obtained from the analysis. The interaction analysis module includes an interaction attribute identification unit, an intent parsing unit, and an expression mode risk assessment unit. The interaction party attribute recognition unit is used to compare the image data and / or audio data in the multimodal interaction data with a preset feature library to determine the identity information of the current interaction party. The intent parsing unit is used to parse the audio data in the multimodal interaction data using natural language processing technology, and to parse the tone and intent of the interacting party. The expression mode risk assessment unit is used to quantify and extract the multimodal expression features of multimodal interaction data in the current interaction context to obtain the current feature vector system corresponding to the current expression mode, and obtain the physiological state level changes of target users in similar contexts to the current interaction context based on the current feature vector system. Based on the physiological state level changes of target users in one or more similar contexts, the unit calculates the potential risk score corresponding to the current expression mode.

Images