Intelligent conversation interaction system and method based on facial recognition and electroencephalogram and galvanic skin

Abstract

The invention relates to the technical field of intelligent conversation interaction, and discloses an intelligent conversation interaction system and method based on facial recognition and electroencephalogram and galvanic galvanis.The modal information of a user is obtained in real time through a modal obtaining module via a sensor network, and the modal information of the user is corrected according to environment changes; meanwhile, data synchronization is carried out on the corrected modal information, and user state changes are comprehensively reflected; the state recognition module extracts the features of the modal information and recognizes the user state through a state recognition strategy, the state recognition strategy comprises the steps of calculating a user reaction value and comparing the user reaction value with a user reaction threshold value to obtain the user state, and the current state of the user can be accurately recognized; the voice recognition module recognizes the voice of the user and judges the requirements of the user, so that the requirements of the user in different states can be deeply understood; and the conversation interaction module generates an interaction response according to the user state and the user demand, and collects the feedback of the user on the interaction response to optimize the interaction response and the state recognition strategy.

Description

Technical Field

[0001] The present invention relates to the field of intelligent conversation interaction technology, and in particular to an intelligent conversation interaction system and method based on facial recognition and electroencephalogram (EEG). Background Art

[0002] At present, researchers are committed to integrating facial recognition and multiple physiological signals such as electroencephalography to improve the accuracy of physiological recognition, and with the help of advanced algorithms and hardware equipment, they can achieve real-time monitoring and recognition of physiological signals to provide users with timely feedback and interaction. However, most of them have not solved the problem of how to obtain the user's modal information to identify the user's state, recognize the user's voice to determine the user's needs, and generate interactive responses and continuously optimize them in combination with the user's state and user needs.

[0003] For example, the Chinese patent with the authorization announcement number CN113488057B discloses a method and system for implementing conversations for health care. The technical problem to be solved is how to simulate the unique way of speaking and tone of voice of an individual as much as possible through voice interaction technology, improve the quality of life of widowed elderly people, and alleviate the pain of children losing their loved ones. The technical solution is as follows: The method is as follows: S1. Record the conversation between the two parties with a collection device; S2. Convert the recorded voice into text and proofread it; S3. Input the sorted corpus into the conversation model for training, and output a personalized conversation model; S4. Use the existing sound corpus to create a speech synthesis model with personal accent characteristics. The system includes a conversation model generation unit and a personalized speech synthesis unit; the conversation model generation unit includes a conversation recording collection module, a speech-to-text module, a sorting and proofreading module, and a model training module; the personalized speech synthesis unit includes a voiceprint encoder, a speech synthesizer, and a sound generator.

[0004] For example, a Chinese patent with authorization announcement number CN108021703B discloses a conversational intelligent teaching system, including interconnected knowledge base units and functional units, the functional units including input preprocessing module, answer reasoning module, evaluation module and dialogue management module, the input preprocessing module, answer reasoning module, evaluation module and dialogue management module are interconnected; wherein the knowledge base unit includes a domain ontology, an interaction template, a semantic dictionary and a student model. The conversational teaching system uses a domain ontology with good semantic relations and hierarchical structures to model the domain knowledge of the system, and can provide the system with reasoning-based knowledge through the hierarchical structure and semantic relations of concepts, and proposes an ontology-driven dialogue management mechanism, which can quickly form dialogue content and dialogue sequences.

[0005] The above patents have the problems raised by this background technology: the above dialogue implementation method records the conversation sound of both parties through the acquisition device, converts the recorded sound into text and proofreads it, and inputs the sorted corpus into the dialogue model for training, outputs a personalized dialogue model, and uses the existing sound corpus to create a speech synthesis model with personal accent characteristics to simulate the unique personal speaking style and tone; the above conversational intelligent teaching system models the domain knowledge of the system through a domain ontology with good semantic relationships and hierarchical structures, and provides the system with reasoning-based knowledge through the hierarchical structure of concepts and voice relationships. The above two patents do not solve the problem of how to obtain the user's modal information to identify the user's state, and identify the user's voice to judge the user's needs, and combine the user's state and user needs to generate interactive responses and continuously optimize. To solve this problem, the present invention proposes an intelligent conversation interaction system and method based on facial recognition and electroencephalography. Summary of the invention

[0006] The purpose of this section is to summarize some aspects of the embodiments of the present invention and briefly introduce some preferred embodiments. Some simplifications or omissions may be made in this section and the specification abstract and the invention title of this application to avoid blurring the purpose of this section, the specification abstract and the invention title, and such simplifications or omissions cannot be used to limit the scope of the present invention.

[0007] In view of the above problems existing in the existing intelligent conversation interaction system and method based on facial recognition and electroencephalography, the present invention is proposed.

[0008] Therefore, the object of the present invention is to provide an intelligent conversation interaction system and method based on facial recognition and electroencephalography.

[0009] In order to solve the above technical problems, the present invention provides an intelligent conversation interaction system based on facial recognition and electroencephalography, comprising: a modality acquisition module, a state recognition module, a speech recognition module and a conversation interaction module;

[0010] The modality acquisition module is used to acquire the user's modality information in real time through the sensor network, and correct the user's modality information according to environmental changes, and synchronize the corrected modality information;

[0011] The state recognition module is used to extract features of modal information and recognize the user state through a state recognition strategy, wherein the state recognition strategy includes calculating a user reaction value and comparing the user reaction value with a user reaction threshold to obtain the user state;

[0012] The speech recognition module is used to recognize the user's speech and determine the user's needs;

[0013] The conversation interaction module is used to generate an interactive response according to the user status and user needs, and collect user feedback on the interactive response to optimize the interactive response and status recognition strategy. The logic of optimizing the status recognition strategy includes comparing the head posture angle with the angle threshold to obtain the user's motion state, and dynamically adjusting the coefficient according to the user's motion state.

[0014] As a preferred solution of the intelligent conversation interaction system based on facial recognition and electroencephalogram (EEG) of the present invention, the features of the modal information are extracted to generate user features, the user features include detection point distance, head posture angle, frequency band ratio and electrodermal amplitude, and the user state is identified through a state recognition strategy, the state recognition strategy includes:

[0015] Calculate the user reaction value v according to the user characteristics, configure the user reaction threshold, which includes the reaction base value v1 and the reaction extreme value v2. Compare the reaction base value and the reaction extreme value with the user reaction value to obtain the user state. If v<v1, identify the user state as the state base value.

[0016] If v1≤v≤v2, the user status is identified as a high status value;

[0017] If v>v2, the user state is identified as a state extreme.

[0018] As a preferred solution of the intelligent conversation interaction system based on facial recognition and electroencephalography of the present invention, the calculation formula of the user reaction value is as follows:

[0019] v=w1×d+w2×θ+w3×r+w4×p;

[0020] Where v represents the user response value, w1 represents the distance coefficient, d represents the detection point distance, w2 represents the angle coefficient, θ represents the head posture angle, w3 represents the ratio coefficient, r represents the frequency band ratio, w4 represents the amplitude coefficient, and p represents the skin electrical amplitude.

[0021] As a preferred solution of the intelligent conversation interaction system based on facial recognition and electroencephalography of the present invention, the logic of the optimization state recognition strategy includes:

[0022] Configure the angle threshold, which includes the base angle value θ1 and the extreme angle value θ2. Compare the base angle value and the extreme angle value with the head posture angle to obtain the user's motion state. If |θ|≤θ1, the user's motion state is no motion.

[0023] If θ1<|θ|<θ2, the user's motion state is the motion base value;

[0024] If |θ|≥θ2, the user's motion state is the motion extreme;

[0025] The coefficients are dynamically adjusted according to the user's motion state, and the coefficients include distance coefficient, angle coefficient, ratio coefficient and amplitude coefficient.

[0026] As a preferred solution of the intelligent conversation interaction system based on facial recognition and electroencephalography of the present invention, the logic of the dynamic adjustment coefficient includes:

[0027] Obtain the coefficient before adjustment, determine the smoothing factor and the target coefficient according to the user's motion state, subtract the target coefficient from the coefficient before adjustment to obtain the coefficient difference, and process the smoothing factor and the coefficient difference, and then combine them with the coefficient before adjustment to obtain the adjusted coefficient to complete the dynamic adjustment coefficient.

[0028] As a preferred solution of the intelligent conversation interaction system based on facial recognition and electroencephalogram and electrocutaneous conduction of the present invention, wherein: the modal information includes facial information, electroencephalogram information and electrocutaneous conduction information;

[0029] Strategies for obtaining users' modal information in real time through sensor networks include:

[0030] The user's modal information is corrected according to environmental changes, including adaptive image enhancement of facial information, comprehensive artifact control of EEG information and comprehensive environmental compensation of electrical skin information, and the corrected modal information is synchronized.

[0031] As a preferred solution of the intelligent conversation interaction system based on facial recognition and electroencephalographic information, the logic of comprehensive environmental compensation of electrodermal information includes:

[0032] Obtaining raw skin electrical information, measuring current skin temperature, configuring standard skin temperature, processing raw skin electrical information with current skin temperature and standard skin temperature, and obtaining temperature-compensated skin electrical information;

[0033] Measure the current ambient humidity, configure the standard ambient humidity, process the temperature-compensated skin electrical information with the current ambient humidity and the standard ambient humidity to obtain the temperature-compensated skin electrical information;

[0034] The average value of the temperature and humidity compensated skin electrical information is calculated to perform baseline correction on the temperature and humidity compensated skin electrical information.

[0035] As a preferred solution of the intelligent conversation interaction system based on facial recognition and electroencephalography of the present invention, the step of synchronizing the corrected modal information includes:

[0036] The target synchronization time point is determined, and two time points close to the target synchronization time point and the values ​​of facial information, EEG information and skin electrical information at these two time points are found to interpolate the target synchronization time point.

[0037] The intelligent conversation interaction method based on facial recognition and electroencephalography includes:

[0038] S1. Obtain the user's modal information in real time through the sensor network, correct the user's modal information according to environmental changes, and synchronize the corrected modal information;

[0039] S2, extracting features of modal information, and identifying the user state through a state recognition strategy, wherein the state recognition strategy includes calculating a user reaction value, and comparing the user reaction value with a user reaction threshold to obtain the user state;

[0040] S3, recognize the user's voice and determine the user's needs;

[0041] S4. Generate interactive responses based on user status and user needs, and collect user feedback on the interactive responses to optimize the interactive responses and state recognition strategies. The logic of optimizing the state recognition strategy includes comparing the head posture angle with the angle threshold to obtain the user's motion state, and dynamically adjusting the coefficient according to the user's motion state.

[0042] A computer device includes a memory for storing instructions and a processor for executing the instructions, so that the computer device implements an intelligent conversation interaction method based on facial recognition and electroencephalography.

[0043] A computer-readable storage medium stores a computer program, which, when executed, implements an intelligent conversation interaction method based on facial recognition and electroencephalogram (EEG).

[0044] Beneficial effects of the present invention: The present invention obtains the user's modal information in real time through the sensor network through the modal acquisition module, corrects the user's modal information according to environmental changes, and synchronizes the corrected modal information, which can fully reflect the user's state changes; the state recognition module extracts the features of the modal information, and identifies the user's state through the state recognition strategy, which includes calculating the user reaction value, comparing the user reaction value with the user reaction threshold to obtain the user state, and can accurately identify the user's current state; the voice recognition module recognizes the user's voice and judges the user's needs, and can more deeply understand the user's real needs and intentions in a specific state; the conversation interaction module generates an interactive response according to the user's state and user needs, and collects user feedback on the interactive response to optimize the interactive response and state recognition strategy, and the logic of the optimized state recognition strategy includes comparing the head posture angle with the angle threshold to obtain the user's motion state, and dynamically adjusting the coefficient according to the user's motion state, generating multiple forms of interactive responses, which can improve the user's participation and experience, and continuously improve the intelligence level and user experience of the conversation interaction system. The intelligent conversation interaction system can be applied in multiple scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for describing the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative labor. Among them:

[0046] Figure 1 The system structure diagram of the intelligent conversation interaction system based on facial recognition and electroencephalography of the present invention;

[0047] Figure 2 It is a state recognition strategy flow chart of the intelligent conversation interaction system based on facial recognition and electroencephalography of the present invention;

[0048] Figure 3 This is a flow chart of the optimization state recognition strategy of the intelligent conversation interaction system based on facial recognition and electroencephalography in the present invention;

[0049] Figure 4 The present invention is a method flow chart of the intelligent conversation interaction method based on facial recognition and electroencephalography. DETAILED DESCRIPTION

[0050] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are described in detail below in conjunction with the accompanying drawings.

[0051] In the following description, many specific details are set forth to facilitate a full understanding of the present invention, but the present invention can also be implemented in other ways different from those described herein, and those skilled in the art can make similar generalizations without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0052] Secondly, the term "one embodiment" or "embodiment" as used herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The term "in one embodiment" that appears in different places in this specification does not necessarily refer to the same embodiment, nor does it refer to a separate or selective embodiment that is mutually exclusive with other embodiments.

[0053] Example 1

[0054] In this embodiment, a system structure diagram of an intelligent conversation interaction system based on facial recognition and electroencephalography is provided, such as Figure 1 As shown, the intelligent conversation interaction system based on facial recognition and electroencephalography includes a modality acquisition module, a state recognition module, a speech recognition module and a conversation interaction module.

[0055] The modality acquisition module is used to acquire the user's modality information in real time through the sensor network, correct the user's modality information according to environmental changes, and synchronize the corrected modality information.

[0056] Modal information includes facial information, EEG information and electrodermal information.

[0057] Strategies for obtaining users' modal information in real time through sensor networks include:

[0058] The user's modal information is corrected according to environmental changes, including adaptive image enhancement of facial information, comprehensive artifact control of EEG information and comprehensive environmental compensation of electrical skin information, and the corrected modal information is synchronized.

[0059] It should be understood that adaptive image enhancement of facial information is used to adjust the contrast of facial images, comprehensive artifact control of EEG information is used to remove artifacts in EEG information through ICA technology and control the intensity of motion artifact suppression, and comprehensive environmental compensation of skin electrode information is used to correct skin electrode information and comprehensively consider the influence of skin temperature and ambient humidity.

[0060] Exemplarily, the function expression of adaptive image enhancement of facial information is as follows:

[0061] I(a,b)=λ×log(1+I0(a,b));

[0062] Where I(a, b) represents the pixel value of the enhanced facial image at the coordinates (a, b), λ represents the enhancement factor, and I0(a, b) represents the pixel value of the original facial image at the coordinates (a, b).

[0063] It should be explained that the enhancement factor λ is used to control the strength of the logarithmic transformation to adjust the scaling factor of the facial image contrast. The enhancement factor λ is usually a positive real number with a value range of 0 to 5. A larger enhancement factor λ will lead to a more obvious facial image enhancement effect, and the dark details of the facial image will be stretched, while a smaller enhancement factor λ will lead to a weaker facial image enhancement effect. If λ = 1, it means that the facial image is only subjected to a standard logarithmic transformation; the pixel value I0(a, b) of the original facial image at the coordinates (a, b) refers to the pixel intensity value of the facial image at the coordinates (a, b), and the value range is between 0 and 255; log(·) is a logarithmic function used to adjust the contrast of the facial image to enhance the visibility of low-brightness pixels and reduce the difference in high-brightness areas, thereby enhancing the contrast. The above formula log(1+I0(a, b)) ensures that the input value is at least 1 to avoid the logarithmic function from being undefined when the input is 0.

[0064] Exemplarily, the functional expression of comprehensive artifact control of EEG information is as follows:

[0065] b(t)=β×ICA(b0(t));

[0066] Where b(t) represents the observed value of EEG information at time t after ICA technology artifact removal and motion artifact suppression, β represents the motion suppression coefficient, ICA(·) represents the artifact removal of the original EEG information by ICA technology, and b0(t) represents the observed value of the original EEG information at time t.

[0067] It needs to be explained that: the observed value b0(t) of the original EEG information at time t usually has an amplitude range of -100μV to 100μV; the motion suppression coefficient β is a weight coefficient used to control the strength of motion artifact suppression. The motion suppression coefficient β is usually a positive number, ranging from 0 to 1. The larger the value of the motion suppression coefficient β, the greater the strength of motion artifact suppression, and the smaller the value of the motion suppression coefficient β, the smaller the strength of motion artifact suppression; the goal of ICA is to retain the useful information of EEG information to the maximum extent while removing artifacts.

[0068] The logic of comprehensive environmental compensation of skin electrical information includes:

[0069] Obtaining raw skin electrical information, measuring current skin temperature, configuring standard skin temperature, processing raw skin electrical information with current skin temperature and standard skin temperature, and obtaining temperature-compensated skin electrical information;

[0070] Measure the current ambient humidity, configure the standard ambient humidity, process the temperature-compensated skin electrical information with the current ambient humidity and the standard ambient humidity to obtain the temperature-compensated skin electrical information;

[0071] The average value of the temperature and humidity compensated skin electrical information is calculated to perform baseline correction on the temperature and humidity compensated skin electrical information.

[0072] Exemplarily, the functional expression of the comprehensive environmental compensation of skin electrical information is as follows:

[0073]

[0074] Where s represents the skin electrical information after comprehensive environmental compensation, s0 represents the original skin electrical information, γ represents the temperature coefficient, T represents the current skin temperature, T0 represents the standard skin temperature, h0 represents the standard ambient humidity, h represents the current ambient humidity, and mean(·) represents the mean processing.

[0075] It needs to be explained that: the skin electrical information s after comprehensive environmental compensation refers to the elimination of the influence caused by skin temperature, environmental humidity and baseline drift during long-term measurement, so that the skin electrical information can reflect more actual physiological activities, and is usually a positive number; the value of the original skin electrical information s0 reflects the change in the conductivity of the skin surface, and is usually a positive number; the current skin temperature T refers to the temperature of the skin surface during measurement, which usually fluctuates between 28°C and 34°C; the standard skin temperature T0 refers to the skin temperature when the ambient temperature is stable; the temperature coefficient γ refers to the degree of influence of skin temperature changes on skin electrical information, and the value range here is between 0.01 and 0.05; the standard ambient humidity h0 is usually the humidity during the experimental setting, and the value is usually between 30% and 60%; the mean processing mean(·) is used for baseline drift correction, which eliminates the influence caused by baseline drift during long-term measurement by subtracting the mean value in brackets; the calculation of the above formula needs to be dimensionless.

[0076] The steps of data synchronization of the corrected modal information include:

[0077] The target synchronization time point is determined, and two time points close to the target synchronization time point and the values ​​of facial information, EEG information and skin electrical information at these two time points are found to interpolate the target synchronization time point.

[0078] Exemplarily, the function expression for data synchronization of the corrected modal information is as follows:

[0079]

[0080] In the formula, Represents facial information, EEG information, and skin electrical information at time point t i The value after data synchronization, X i-1 represents the value of facial information, EEG information and skin electrical information at the i-1th time point, X i+1 represents the value of facial information, EEG information and skin electrical information at the i+1th time point, t i+1 Indicates the timestamp of the i+1th time point, t i-1 Indicates the timestamp of the i-1th time point, t * Indicates the target synchronization time point.

[0081] What needs to be explained is: It refers to the facial information at time point t i The value at time t i The value and skin electrical information at time point t i The value at needs to be dimensionless; and X i-1 and X i+1 The value of refers to the time point t near the target synchronization * The values ​​corresponding to the two time points; the target synchronization time point t * The value of is usually within the range of all timestamps, and time points with smaller intervals are usually selected for alignment.

[0082] The sensor network includes RGB cameras, EEG sensors (such as head-mounted electroencephalographs) and skin electrical sensors (such as GSR sensors). The facial information captured by the RGB camera is a facial image. The changes in the user's facial information are usually closely related to the user's status. The EEG sensor obtains the EEG signal, which can provide rich information related to brain activity and can be used to analyze the user's status. The skin electrical information obtained by the skin electrical sensor is the skin electrical signal, which can detect the changes in the user's status through the changes in the skin electrical signal. In real applications, environmental factors may affect the data acquisition effect of the sensor network, so the sensor network The network needs to consider environmental interference correction during data acquisition, including adaptive image enhancement of facial images, comprehensive artifact control of EEG signals, and comprehensive environmental compensation of skin electrodermal signals, to improve the quality of facial images under low light conditions, remove artifacts caused by muscle activity or eye movement, and compensate according to changes in skin temperature and ambient humidity after measuring the user's skin electrodermal response; since the sampling rates of different sensors may be different, it is necessary to synchronize the modal information of all users to the same time point for subsequent analysis and processing. The above strategy can acquire and process the user's modal information in real time and accurately, providing support for subsequent interactive responses.

[0083] The state recognition module is used to extract the features of modal information and identify the user state through a state recognition strategy. The state recognition strategy includes calculating the user reaction value and comparing the user reaction value with the user reaction threshold to obtain the user state.

[0084] Extract the features of modal information and generate user features, which include detection point distance, head posture angle, frequency band ratio and skin electrical amplitude. The user state is identified through the state recognition strategy. The state recognition strategy is as follows: Figure 2 As shown, specifically including:

[0085] Calculate the user reaction value v according to the user characteristics, configure the user reaction threshold, which includes the reaction base value v1 and the reaction extreme value v2. Compare the reaction base value and the reaction extreme value with the user reaction value to obtain the user state. If v<v1, identify the user state as the state base value.

[0086] If v1≤v≤v2, the user status is identified as a high status value;

[0087] If v>v2, the user state is identified as a state extreme.

[0088] Exemplarily, the function expression of the detection point distance is as follows:

[0089]

[0090] Where d represents the distance between the detection points, (a1, b1) and (a2, b2) are the coordinates of the two detection points.

[0091] It needs to be explained that the detection point distance d refers to the distance between the key points of the face. If you want to measure the distance between the eyes, the two detection points are the inner corner and the outer corner of one eye respectively; if you want to measure the relative distance between the lips and the nose, the two detection points are the tip of the nose and the center of the lower lip respectively; the value range of the detection point distance d depends on the position of the detection point in the facial image. A detection point distance of 0 means that the two detection points coincide. Generally speaking, the larger the pixel value, the larger the detection point distance.

[0092] Exemplarily, the function expression of the head posture angle is as follows:

[0093]

[0094] Where θ represents the head posture angle, (a1, b1) and (a2, b2) are the coordinates of the two detection points.

[0095] It needs to be explained that: the head posture angle θ is the angle that describes the direction of the head, which refers to the direction angle from the first detection point (a1, b1) to the second detection point (a2, b2), usually the angle relative to the horizontal line (x-axis), and the value range of the head posture angle θ is (-π / 2, π / 2); arctan(·) is the inverse tangent function, which is used to calculate the head posture angle from the slope, and converts the ratio of the vertical and horizontal distances between the two detection points into an angle value; when the pitch angle (that is, the up and down rotation of the head) is calculated, the two detection points are the tip of the nose and the chin; when the yaw angle (that is, the left and right rotation angle of the head) is calculated, the two detection points are the tip of the nose and the center of the two eyes; when the roll angle (that is, the rotation of the head around the central axis, that is, the tilt of the head) is calculated, the two detection points are the center of the two eyes and the center of the lips.

[0096] The frequency band ratio refers to the power ratio of theta waves and beta waves calculated from the acquired EEG information, which can indicate the user's status.

[0097] Exemplarily, the functional expression of the skin electrical amplitude is as follows:

[0098] p = max(s) - min(s);

[0099] Where p represents the skin electrical amplitude, and s represents the skin electrical information after comprehensive environmental compensation.

[0100] It needs to be explained that the skin electrical amplitude p refers to the difference between the maximum and minimum values ​​of the skin electrical information after stimulation. It is used to quantify the intensity of the skin electrical response and can reflect the degree of user reaction. The higher the skin electrical amplitude, the stronger the user reaction.

[0101] The calculation formula of user reaction value is as follows:

[0102] v=w1×d+w2×θ+w3×r+w4×p;

[0103] Where v represents the user response value, w1 represents the distance coefficient, d represents the detection point distance, w2 represents the angle coefficient, θ represents the head posture angle, w3 represents the ratio coefficient, r represents the frequency band ratio, w4 represents the amplitude coefficient, and p represents the skin electrical amplitude.

[0104] It needs to be explained that: the distance coefficient w1 refers to the influence of the detection point distance on the user reaction value, and the value of the distance coefficient w1 is usually a non-negative number; the angle coefficient w2 refers to the influence of the head posture angle on the user reaction value, and the value of the angle coefficient w2 is usually a non-negative number; the ratio coefficient w3 refers to the influence of the frequency band ratio on the user reaction value, and the value of the ratio coefficient w3 is usually a non-negative number; the value range of the frequency band ratio r is between 0 and 1 or greater than 1, depending on the different states of the brain. When the brain is in a concentrated state, Beta waves are dominant, and the frequency band ratio is lower. When the brain is in a relaxed state, Theta waves are dominant, and the frequency band ratio is higher; the amplitude coefficient w4 refers to the influence of the skin electrode amplitude on the user reaction value, and the value of the amplitude coefficient w4 is usually a non-negative number; the values ​​of the above distance coefficient w1, angle coefficient w2, ratio coefficient w3 and amplitude coefficient w4 need to be obtained through experimental optimization, and satisfy w1+w2+w3+w4=1.

[0105] In specific applications, for facial images, each frame is analyzed through a deep learning model to extract the features of facial information, including the detection point distance and the head posture angle, where the detection point distance refers to the distance between facial key points (including the distance between eyes, lips and nose), and the head posture angle refers to the pitch angle, yaw angle and roll angle of the head calculated based on the facial key points to judge the user's facial expressions and movements; the features of the extracted EEG information, including the frequency band ratio, are used to evaluate the user's status; the features of the extracted skin electricity information, including the skin electricity amplitude, are used to judge The intensity of the user's reaction is determined; the user state is identified according to the comparison results of the user reaction value with the reaction base value and the reaction extreme value, wherein when the user state is the state base value, the facial expression is usually relaxed, the skin electrical reaction is low, and the Theta wave is dominant; when the user state is the state high value, the facial expression is usually strong, the skin electrical reaction is strong, and the frequency band ratio is high; when the user state is the state extreme value, the facial expression is usually tense, the head posture changes significantly, and the skin electrical reaction is strong. According to the above steps, the user's state changes can be monitored and responded to in real time, providing the user with a more personalized and adaptable interactive experience.

[0106] The speech recognition module is used to recognize the user's speech and determine the user's needs.

[0107] Exemplarily, when a user inputs voice, the user's voice is recognized and converted into text information, the converted text is cleaned and segmented to remove noise and irrelevant information, the text is analyzed through a semantic understanding model, the demand intent is identified, and the demand intent is further refined in combination with facial images to generate user needs, determine the priority of user needs and refine the response.

[0108] The conversation interaction module is used to generate interactive responses based on user status and user needs, and collect user feedback on the interactive responses to optimize the interactive responses and state recognition strategies. The optimized state recognition strategy includes comparing the head posture angle with the angle threshold to obtain the user's motion state, and dynamically adjusting the coefficient according to the user's motion state.

[0109] According to the user status and user needs, select the appropriate response template from the response template library to generate an interactive response. For example, when it is recognized that the user is querying weather information, the conversation interaction system automatically generates weather-related interactions, where the response template is: Today's weather is 25℃, and tomorrow's temperature is expected to be 20℃, please remember to put on more clothes; when it is recognized that the user status is an extreme state, the conversation interaction system automatically makes corresponding acceleration and care, where the response template is: I will query for you as soon as possible, please wait a moment; after each interactive response with the user, collect the user's feedback on the interactive response, including whether the user chooses satisfaction or dissatisfaction, configure the frequency threshold, and obtain the optimization strategy based on the comparison of the user's dissatisfaction frequency with the frequency threshold. If the user's dissatisfaction frequency is less than or equal to the frequency threshold, the optimization strategy is to optimize the interactive response;

[0110] If the user's dissatisfaction frequency is greater than the frequency threshold, the optimization strategy is the optimization state identification strategy.

[0111] Exemplarily, optimizing interactive responses includes simplifying the language in response templates, speeding up the response speed of conversational interaction systems, and enhancing accuracy. For example, for some complex or ambiguous queries, multiple rounds of dialogue should be used to accurately determine the user's real needs, and the accuracy of the response should be enhanced through the user's historical behavior. At the same time, the response should be completed within a reasonable time. For example, the response time for simple queries should not exceed 3 seconds. For more complex queries, the response time should be appropriately extended and the user should be informed of the progress simultaneously.

[0112] The logic of optimizing the state recognition strategy is as follows Figure 3 As shown, specifically including:

[0113] Configure the angle threshold, which includes the base angle θ1 and the extreme angle θ2. Compare the base angle and the extreme angle with the head posture angle to get the user's motion state. If |θ|≤θ1, the user's motion state is no motion, that is, the user is not moving and may be in a static state.

[0114] If θ1<|θ|<θ2, the user's motion state is the motion base value, that is, the user is performing slight exercise;

[0115] If |θ|≥θ2, the user's motion state is the motion extreme, that is, the user is performing strenuous exercise.

[0116] The coefficients are dynamically adjusted according to the user's motion state. The coefficients include distance coefficient, angle coefficient, ratio coefficient and amplitude coefficient. The logic of dynamically adjusting the coefficients includes:

[0117] Obtain the coefficient before adjustment, determine the smoothing factor and the target coefficient according to the user's motion state, subtract the target coefficient from the coefficient before adjustment to obtain the coefficient difference, and process the smoothing factor and the coefficient difference, and then combine them with the coefficient before adjustment to obtain the adjusted coefficient to complete the dynamic adjustment coefficient.

[0118] Exemplarily, the function expression of the dynamic adjustment coefficient is as follows:

[0119]

[0120] In the formula, w j (m) represents the coefficient adjusted under the user's motion state m, w j represents the coefficient before adjustment, κ(m) represents the smoothing factor under motion state m, Represents the target coefficient.

[0121] Exemplarily, the value logic of the smoothing factor includes: if the user's motion state is no motion, then κ(m)=0;

[0122] If the user's motion state is the base value, then κ(m) = 0.5;

[0123] If the user's motion state is a motion extreme, κ(m)=1.

[0124] What needs to be explained is that the coefficient w adjusted under the user's motion state m j (m) where j represents an index. When j=1, it refers to the distance coefficient adjusted under the user's motion state m. When j=2, it refers to the angle coefficient adjusted under the user's motion state m. When j=3, it refers to the ratio coefficient adjusted under the user's motion state m. When j=4, it refers to the amplitude coefficient adjusted under the user's motion state m. The smoothing factor κ(m) is a factor used to adjust the coefficient according to the current user's motion state, which controls the amplitude of the coefficient adjustment. The value range of the smoothing factor κ(m) is between 0 and 1. The target coefficient It refers to the target coefficient that the user hopes to adjust to when in a certain state of motion. The value range is between 0 and 1.

[0125] In practical applications, when the user's motion state is no motion, the values ​​of the distance coefficient and the amplitude coefficient are usually higher, because the detection point distance and the skin conductance amplitude can more accurately reflect the user's state under static conditions, while the values ​​of the angle coefficient and the ratio coefficient are usually lower, because the head posture and brain activity changes relatively little in a static state; when the user's motion state is the base value of motion, the values ​​of the angle coefficient and the amplitude coefficient should be appropriately increased, especially the motion state has a certain influence on the user's state, while the values ​​of the distance coefficient and the ratio coefficient may remain relatively stable; when the user's motion state is the extreme value of motion, the values ​​of the angle coefficient and the amplitude coefficient are usually higher, because the head posture The changes in state and skin conduction response are closely related to the user state, while the values ​​of distance coefficient and ratio coefficient are usually relatively low, because facial expressions may be greatly affected by movement, and the changes in brain activity are not as obvious as head posture; the response speed and response template of the conversation interaction system are adjusted according to the adjusted coefficients combined with the user state. For example, if the user is in the extreme state, the conversation interaction system should slow down the response speed, lower the tone and provide encouraging interaction responses. If the user is in the base state, the conversation interaction system should speed up the response speed, increase the tone and provide more interaction response information, so as to provide users with a more personalized and adaptable interaction experience.

[0126] Example 2

[0127] In this embodiment, a method flow chart of an intelligent conversation interaction method based on facial recognition and electroencephalography is provided, such as Figure 4 As shown, the intelligent conversation interaction method based on facial recognition and electroencephalographic electrocorticography includes:

[0128] S1. Obtain the user's modal information in real time through the sensor network, correct the user's modal information according to environmental changes, and synchronize the corrected modal information;

[0129] S2, extracting features of modal information, and identifying the user state through a state recognition strategy, wherein the state recognition strategy includes calculating a user reaction value, and comparing the user reaction value with a user reaction threshold to obtain the user state;

[0130] S3, recognize the user's voice and determine the user's needs;

[0131] S4. Generate interactive responses based on user status and user needs, and collect user feedback on the interactive responses to optimize the interactive responses and state recognition strategies. The logic of optimizing the state recognition strategy includes comparing the head posture angle with the angle threshold to obtain the user's motion state, and dynamically adjusting the coefficient according to the user's motion state.

[0132] For details of the intelligent conversation interaction method based on facial recognition and electroencephalogram (EEG), please refer to the intelligent conversation interaction system based on facial recognition and EEG, which will not be described in detail here.

[0133] Example 3

[0134] In this embodiment, a computer device is provided, including a memory and a processor, the memory is used to store instructions, and the processor is used to execute the instructions, so that the computer device executes the steps of implementing the above-mentioned intelligent conversation interaction method based on facial recognition and electroencephalography.

[0135] Example 4

[0136] In this embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed, the steps of the above-mentioned intelligent conversation interaction method based on facial recognition and electroencephalography are implemented.

[0137] The computer-readable storage medium includes: a U disk, a mobile hard disk, a read-only memory, a random access memory, a magnetic disk or an optical disk, and other media for storing program codes.

[0138] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention can be modified or replaced by equivalents without departing from the spirit and scope of the technical solutions of the present invention, which should be included in the scope of the claims of the present invention.

Claims

1. An intelligent conversation interaction system based on facial recognition and electroencephalography, characterized by: include: Modality acquisition module, state recognition module, speech recognition module and conversation interaction module; The modality acquisition module is used to acquire the user's modality information in real time through the sensor network, and correct the user's modality information according to environmental changes, and synchronize the corrected modality information; The state recognition module is used to extract features of modal information and recognize the user state through a state recognition strategy, wherein the state recognition strategy includes calculating a user reaction value and comparing the user reaction value with a user reaction threshold to obtain the user state; The speech recognition module is used to recognize the user's speech and determine the user's needs; The conversation interaction module is used to generate an interactive response according to the user status and user needs, and collect user feedback on the interactive response to optimize the interactive response and status recognition strategy. The logic of optimizing the status recognition strategy includes comparing the head posture angle with the angle threshold to obtain the user's motion state, and dynamically adjusting the coefficient according to the user's motion state.

2. The intelligent conversation interaction system based on facial recognition and electroencephalography as claimed in claim 1, characterized in that: The features of the modal information are extracted to generate user features, wherein the user features include detection point distance, head posture angle, frequency band ratio and skin electrical amplitude, and the user state is identified through a state recognition strategy, wherein the state recognition strategy includes: Calculate the user reaction value v according to the user characteristics, configure the user reaction threshold, which includes the reaction base value v1 and the reaction extreme value v2. Compare the reaction base value and the reaction extreme value with the user reaction value to obtain the user state. If v<v1, identify the user state as the state base value. If v1≤v≤v2, the user status is identified as a high status value; If v>v2, the user state is identified as a state extreme.

3. The intelligent conversation interaction system based on facial recognition and electroencephalography as claimed in claim 2, characterized in that: The calculation formula of the user reaction value is as follows: v=w1×d+w2×θ+w3×r+w4×p; Where v represents the user response value, w1 represents the distance coefficient, d represents the detection point distance, w2 represents the angle coefficient, θ represents the head posture angle, w3 represents the ratio coefficient, r represents the frequency band ratio, w4 represents the amplitude coefficient, and p represents the skin electrical amplitude.

4. The intelligent conversation interaction system based on facial recognition and electroencephalography as claimed in claim 3, characterized in that: The logic of the optimization state identification strategy includes: Configure the angle threshold, which includes the base angle value θ1 and the extreme angle value θ2. Compare the base angle value and the extreme angle value with the head posture angle to obtain the user's motion state. If |θ≤θ1, the user's motion state is no motion. If θ1<|θ<θ2, the user's motion state is the motion base value; If |θ≥θ2, the user's motion state is the motion extreme; The coefficients are dynamically adjusted according to the user's motion state. The coefficients include distance coefficient, angle coefficient, ratio coefficient and amplitude coefficient.

5. The intelligent conversation interaction system based on facial recognition and electroencephalography as claimed in claim 4, characterized in that: The logic of the dynamic adjustment coefficient includes: Obtain the coefficient before adjustment, determine the smoothing factor and the target coefficient according to the user's motion state, subtract the target coefficient from the coefficient before adjustment to obtain the coefficient difference, and process the smoothing factor and the coefficient difference, and then combine them with the coefficient before adjustment to obtain the adjusted coefficient to complete the dynamic adjustment coefficient.

6. The intelligent conversation interaction system based on facial recognition and electroencephalography as claimed in claim 5, characterized in that: The modality information includes facial information, electroencephalogram information and electrodermal information; Strategies for obtaining users' modal information in real time through sensor networks include: The user's modal information is corrected according to environmental changes, including adaptive image enhancement of facial information, comprehensive artifact control of EEG information and comprehensive environmental compensation of electrical skin information, and the corrected modal information is synchronized.

7. The intelligent conversation interaction system based on facial recognition and electroencephalography as claimed in claim 6, characterized in that: The logic of the comprehensive environmental compensation of the electrical skin information includes: Obtaining raw skin electrical information, measuring current skin temperature, configuring standard skin temperature, processing raw skin electrical information with current skin temperature and standard skin temperature, and obtaining temperature-compensated skin electrical information; Measure the current ambient humidity, configure the standard ambient humidity, process the temperature-compensated skin electrical information with the current ambient humidity and the standard ambient humidity to obtain the temperature-compensated skin electrical information; The average value of the temperature and humidity compensated skin electrical information is calculated to perform baseline correction on the temperature and humidity compensated skin electrical information.

8. The intelligent conversation interaction system based on facial recognition and electroencephalography as claimed in claim 7, characterized in that: The step of performing data synchronization on the corrected modal information comprises: The target synchronization time point is determined, and two time points close to the target synchronization time point and the values ​​of facial information, EEG information and skin electrical information at these two time points are found to interpolate the target synchronization time point.

9. An intelligent conversation interaction method based on facial recognition and electroencephalogram (EEG), implemented based on an intelligent conversation interaction system based on facial recognition and EEG according to any one of claims 1 to 8, characterized in that: include: S1. Obtain the user's modal information in real time through the sensor network, correct the user's modal information according to environmental changes, and synchronize the corrected modal information; S2, extracting features of modal information, and identifying the user state through a state recognition strategy, wherein the state recognition strategy includes calculating a user reaction value, and comparing the user reaction value with a user reaction threshold to obtain the user state; S3, recognize the user's voice and determine the user's needs; S4. Generate interactive responses based on user status and user needs, and collect user feedback on the interactive responses to optimize the interactive responses and state recognition strategies. The logic of optimizing the state recognition strategy includes comparing the head posture angle with the angle threshold to obtain the user's motion state, and dynamically adjusting the coefficient according to the user's motion state.

10. A computer device, characterized in that: include: A memory for storing instructions; The processor is used to execute the instructions so that the computer device implements the intelligent conversation interaction method based on facial recognition and electroencephalography as claimed in claim 9.

11. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed, the intelligent conversation interaction method based on facial recognition and electroencephalography as claimed in claim 9 is implemented.

Images