4590results about "Acquiring/recognising facial features" patented technology

A method and system are introduced for recognizing cognitive engagement in classrooms by utilizing multimodal data. Associated modalities of cognitive engagement are learned from visual and audio data. This approach involves constructing a multidimensional representation model for cognitive engagement that includes behaviors, emotions, and speech. To identify engagement, three distinct deep learning models are employed: You Only Look Once version 8 (Yolov8) for analyzing body posture, Efficient Network (EfficientNet) for facial expressions, and Text Convolution Neural Network (TextCNN) for speech text. These models are trained and refined with the aid of student engagement surveys, leading to a decision-making process that integrates the results from the different modalities. Additionally, a dataset and a data annotation system are developed for engagement recognition. This innovative method aims to achieve detailed engagement recognition, addressing various perception needs in real-world applications.

In addition to an original prompt that is manually provided by a user, contextual information is sent to a generative AI to elicit a higher quality response. Sensors collect audio, video, physiological, cognitive, environmental, and digital data from the user. Machine-learning models evaluate the sensor data to infer the emotional state of the user. The emotional state is used to augment the original prompt with contextual information. The augmented prompt is fed into the generative AI to make it context-aware. Accordingly, the generative AI can automatically pick up on non-verbal cues that the user did not manually articulate in the original prompt. Just as a human-to-human conversation involves a combination of verbal and non-verbal communications, the present concepts enable the generative AI to also leverage non-verbal communication when interacting with human users.

The invention provides a private domain live broadcast data storage and visitor authentication method and system based on a block chain, and relates to the technical field of data storage and access control, and the method comprises the steps: collecting multi-mode biological characteristic data of a visitor, carrying out the feature extraction and fusion through a deep neural network of a space-time attention mechanism, and carrying out the authentication of the visitor; authenticity verification is carried out in combination with an adversarial generative network, a dynamic identity token is generated, and a hash value of the dynamic identity token is written into the block chain. The method comprises the following steps: performing semantic analysis on live broadcast data flow through a self-adaptive content perception model, constructing a double-layer access control matrix according to a data sensitivity level and a credibility score, performing self-adaptive fragmentation and hierarchical encryption based on federated learning, and then storing. After an access request is subjected to identity verification, an optimal edge node is selected according to access authority and network performance parameters to construct multi-path transmission, and a transmission strategy is generated based on a service quality score and a data sensitivity level. And when the service quality is lower than a threshold value, triggering the smart contract to perform fault migration.

A driving state monitoring and feedback method and system based on multi-modal human factors intelligent data analysis, and an edge computing terminal device. The method comprises: receiving multi-modal human factors data of a tested driver that is collected in real time (S110); pre-processing the multi-modal human factors data, wherein the pre-processing comprises denoising processing and data normalization processing (S120); sending the pre-processed multi-modal human factors data into a pre-trained first state recognition model, so as to obtain a real-time recognized driver state, wherein driver states include a normal state and abnormal states, and the types of the abnormal states include a plurality of states such as a fatigue state, a distracted state and an angry state (S130); and when it is recognized that the driver state is an abnormal state, generating, for different categories of abnormal states, driving state feedback instructions to a driving intervention system, such that the driving intervention system performs state feedback adjustments on the driver on the basis of the received driving state feedback instructions (S140). The method and system can recognize different driving states of a driver in real time and then perform processing on the basis of different driving states, thereby avoiding the occurrence of traffic accidents.

Disclosed are an emotional evolution method and terminal for a virtual avatar in educational metaverse. By collecting expression data and audio data of the virtual avatar, and performing emotional feature extraction based on the expression data and the audio data; fusing an extracted sound emotional feature with an extracted expression emotional feature by using an emotional feature fusion model, and performing emotion recognition on a multi-modal emotional feature fusion result obtained by fusion to obtain an emotional category corresponding to the multi-modal emotional feature fusion result; labeling the multi-modal emotional feature fusion result based on a semantic vector of the emotional category to generate an emotional evolution sequence; and driving the virtual avatar to perform emotional expression according to a target emotional evolution pattern extracted from the emotional evolution sequence, a unified and united multi-modal emotional feature fusion result is formed, the emotional self-evolution of the virtual avatar is achieved.

An emotional engagement detection method based on positive emotional perception is provided, which relates to a field of data analysis technologies. The emotional engagement detection method is to extract features from data of different dimensions during student classroom learning, to achieve information recognition of each dimension, and perform decision fusion and result analysis and application. The emotional engagement detection method is to extract the features from the data of different dimensions during the student classroom learning, applies a fusion strategy based on a deep learning network for subsequent supervision classification, to construct a positive emotional engagement model of a student. Different modalities information such as cognition, emotion and thinking of the student obtained from different means is integrated to a frame, to comprehensively reflect a positive emotion of the student.

In some embodiments, a computer system displays user interfaces for enrolling one or more features of a user of a computer system. In some embodiments, a computer system displays visual effects associated with a virtual avatar in an XR environment. In some embodiments, a computer system displays objects having different visual characteristics in an XR environment. In some embodiments, a computer system switches between different presentation modes associated with a user represented in an XR environment. In some embodiments, a computer system displays a virtual avatar in an XR environment.

Systems, methods and non-transitory computer readable media for generating and operating artificial entities are provided. Some disclosed embodiments may involve receiving information related to a source individual; generating an artificial entity associated with the source individual based on the received information; receiving data reflecting an interaction with the artificial entity; and determining a manner for the artificial entity to respond to the interaction based on the collected information.

The invention belongs to but not limited to the technical field of emotion recognition, and discloses a multi-modal classroom emotion recognition method and system based on modal adaptive learning, and the method comprises the steps: building a multi-modal data collection system, collecting the multi-modal data, such as texts, audios and videos, in a classroom teaching process in real time, and carrying out the real-time collection of the multi-modal data; and preprocessing and feature extraction are carried out on data of different modes. For emotion expression feature differences of different classroom scenes, a multi-modal data fusion method based on adaptive modal scoring (AMS) is provided, and a time sequence modeling and emotion classification model is constructed. The multi-modal fusion and time sequence modeling mechanism simulates the multi-dimensional perception of the teacher on the emotional states of the students in the class, and enhances the interpretability of the emotion recognition system. According to the system, weighted fusion is carried out on data features from texts, audios and videos through fusion of multi-modal data, and complete emotion feature representation is formed.

The invention provides a digital human interaction control method and device fusing emotion semantics and logical reasoning and a storage medium. The method comprises the following steps: analyzing multi-modal input data of a user, constructing emotion-semantics joint representation, and generating a logic decision path; and through a cognitive fusion module, emotion-semantic representation and a logic decision path are fused, and an interaction response adapting to emotion and logic consistency is generated. The system optimizes an emotion semantic model and a logical reasoning rule on line according to user feedback and interaction history, and real-time interaction of emotion dynamic and logical rules is achieved. The system can dynamically adjust the logic decision path based on the multi-mode emotional state of the user, and improves the naturalness and situation adaptability of interaction. A dynamic time warping algorithm and a factorization machine are introduced to process a multi-modal feature fusion problem, and the accuracy and robustness of emotional state recognition are improved. The online optimization mechanism enables the model and the rule to be evolved continuously, and reasoning errors are corrected automatically through user feedback, so that error circulation is avoided.

The invention relates to the field of data processing, in particular to an emotion recognition method based on a large model and a related device. The method comprises the following steps: firstly, acquiring to-be-processed multi-modal data; the multi-modal data at least comprises text information, audio information and image information input by a user; the image information at least comprises a facial expression image and / or a limb action image. And inputting the multi-modal data into an emotion recognition model for emotion state analysis to obtain a predicted emotion state corresponding to the user. And finally, performing confidence evaluation based on the predicted emotional state, and generating a user emotion recognition result based on the predicted emotional state and a corresponding evaluation result. According to the method, richer emotion details are captured from the multi-modal data through the emotion recognition model, cross-modal emotion intelligent recognition is realized, the accuracy of emotion recognition is further improved by combining confidence evaluation, the emotion recognition efficiency is improved, and the application range of emotion recognition is expanded.

The invention relates to the technical field of man-machine interaction, in particular to a robot behavior mode dynamic adjustment method based on multi-mode perception, which comprises the following steps: S1, collecting a visual image, a voice signal and an environment parameter of a target scene in real time; s2, extracting each modal feature; s3, dynamically allocating weights, and generating a fusion feature vector; s4, identifying a current scene type and a user attribute; s5, matching a corresponding interaction mode from a preset strategy library based on the scene type and the user attribute identified in the step S4; and S6, executing the matched interaction mode in the S5, and updating the weight distribution rule according to the feedback data. According to the method, vision, voice and environment characteristics are fused through a multi-mode perception technology, the behavior mode of the robot is dynamically adjusted based on a self-adaptive weighted fusion algorithm, and the interaction strategy is optimized in combination with user feedback, so that the interaction accuracy and the intelligent level of the robot under different scenes and user attributes are improved.

The invention relates to the technical field of intelligent robots, in particular to a service robot intelligent emotion interaction method, which comprises the following steps: acquiring emotion related data of a user through a multi-modal sensor on a service robot, analyzing the acquired data based on a deep learning model, and extracting emotion features of the user; future emotion changes are predicted through a time sequence analysis algorithm, a service robot generates a personalized interaction strategy through an emotion decision-making system according to the current and predicted emotion states, and the system adopts a self-adaptive algorithm based on reinforcement learning, so that the interaction strategy can be continuously optimized along with the emotion changes of a user; the robot performs modeling and optimization on long-term interaction data by utilizing a self-learning mechanism in combination with emotional feedback of the user, so that the precision and interaction effect of emotional response are gradually improved; according to the method, the problems of inaccurate emotion recognition, inflexible interaction strategy and lack of adaptive optimization in the prior art are effectively solved, and the interaction experience of the service robot and the user is remarkably improved.

Systems, methods, and computer program products are disclosed for selectively employing a processing mode based on a continuous gesture. Selectively employing a processing mode may include detecting an existence of a mode-selection gesture by an individual. Upon detecting the existence of the mode-selection gesture, a first mode for processing facial micromovements of the individual may be continuously implemented while continuously detecting the mode-selection gesture. Following the continuous detection of the mode-selection gesture, a cessation of the mode-selection gesture may be detected. In response, implementation of the first mode may cease, and implementation of a second mode different from the first mode may be initiated for processing facial micromovements of the individual.

Systems, methods and non-transitory computer readable media for image analysis for personal interaction are provided. Audio data may be received. The audio data may include an input from an entity in a natural language. The input may include at least a first part and a second part. Further, image data may be received. The image data may depict a particular movement. The particular movement may be a movement of a particular portion of a particular body. The particular movement and the first part may be concurrent. The particular body may be associated with the entity. Further, a conversational artificial intelligence model may be used to analyze the audio data and the image data to generate a response in the natural language to the input. The response may be based on the input and the particular movement. Further, the generated response may be provided to the entity.

The invention discloses a self-adaptive education method based on artificial intelligence, and relates to the technical field of education, and the method comprises the following components: intelligent emotion monitoring, learning performance evaluation, self-adaptive teaching strategy adjustment, personalized learning path planning and a feedback and optimization mechanism. According to the method, by introducing a learning performance evaluation and prediction module and a prediction and adaptive teaching strategy adjustment module, individual differences, including learning styles, interest preferences and ability levels, of students are fully considered, so that teaching strategies are intelligently recommended and adjusted, and personalized learning paths are dynamically generated; the personalized teaching mode effectively overcomes the defects of a traditional'one-cut 'teaching mode, the teaching effect is remarkably improved, each student can obtain the most suitable learning experience, and the learning enthusiasm and the learning effect of the students are greatly improved.

The invention relates to a psychological counseling robot based on multi-modal data and a large vertical model. The robot is used for providing psychological counseling services for scenes such as families, schools and enterprises, and the front end depends on an intelligent robot terminal, adopts a main mode of virtual digital man-machine interaction and is assisted by multi-modal emotion recognition to enhance the psychological counseling effect; the rear end takes an intelligent agent as a control core and takes a psychological big model as a computing platform. The strong semantic understanding and dialogue interaction capability of the large model is utilized, and the multi-modal mode recognition of expressions and voices is combined, so that the psychological assessment and psychological healing functions are realized, and meanwhile, the knowledge supplementing and checking capability of an external knowledge base is realized; according to the method, a strategy of combining structured prompt with multi-modal emotion data enhancement is used, performance advantages of a large model are fully utilized by means of a retrieval enhancement generation and knowledge graph technology, meanwhile, occurrence of illusion problems is reduced, a digital human interaction mode is adopted, interpersonal barriers are reduced, and consultation quality is improved.

The invention provides a multi-mode-based man-machine interaction method and system, and relates to the technical field of man-machine interaction. Voice input of a user is received, environment information is collected, facial expressions of the user are captured, and voice features, environment features and facial features are obtained based on the input; utilizing a pre-trained emotion recognition model to generate emotion features of the user, and utilizing a pre-trained user behavior model to generate behavior features according to the current operation information of the user; after the features are obtained, the first weights and the second weights of the features are dynamically adjusted, feature weighted fusion is carried out, and comprehensive features are obtained; the multiple candidate responses are generated based on the comprehensive characteristics, the final response is screened out through a preset scoring mechanism and output, comprehensive understanding of the intention and the emotional state of the user is achieved through comprehensive utilization of multiple data sources, and meanwhile the weight of each characteristic can be dynamically adjusted so that more intelligent and humanized interaction experience can be generated.

The invention discloses an intelligent advertisement putting processing method and system based on user data, and relates to the technical field of data processing, and the method comprises the steps: putting an advertisement in an advertisement pool based on an edge device, collecting the video information during the advertisement playing period through the edge device, analyzing the stored video information, extracting the user feature information, and storing the user feature information in the advertisement pool; the user feature information is integrated into user feature vectors, the user feature vectors stored in the edge devices are uploaded to the cloud platform, the cloud platform analyzes the user feature vectors uploaded by the edge devices to generate user portrait information, and the user portrait information is displayed on the basis of the user portrait information generated by the cloud platform. And putting corresponding advertisements to the user groups with similar user portrait information, collecting feedback data, and dynamically optimizing the advertisements put by the edge devices and the user groups based on the feedback data. The method has the advantages that the data transmission delay and the cloud computing pressure are reduced through the cloud edge collaboration method.

Systems, apparatuses, methods, and computer program products are disclosed for predicting an emotion in real-time based on a multimodal input including at least (i) an amount of keystrokes over a period of time and (ii) text. An example method may include receiving, by a communications circuitry, a multimodal input from a user including at least (i) an amount of keystrokes over a period of time and (ii) text. The example method may further include generating, by a trained machine learning model of an emotion prediction circuitry and using the multimodal input, an EmotionPrint for the user. The example method may finally include determining, by the emotion prediction circuitry and using the EmotionPrint, a next action.

Disclosed in the present invention is a person concentration recognition and evaluation system based on machine vision. The technical method belongs to the field of computer vision and human-machine interaction in the field of artificial intelligence. During an assembly activity, the states of listeners and audiences are important links for feeding back the assembly effect, but there is no accurate scoring standard, and therefore the links are referred to as fuzzy links. Taking a party, a speech or a conference as an example, the evaluation system designed herein innovatively performs listener (audience) behavior discrimination across four dimensions, which are respectively expression screening, concentration recognition, facial distance monitoring and facial position monitoring; fuzzy evaluation can be better performed on the real-time states of the listeners (audiences) by means of a fuzzy algorithm; a particle swarm fuzzy algorithm scheme is provided to enable the state discrimination to break through the polarization limitation; and a membership function is set to define a fuzzy area, the degree of behaviors of the minority being different from behaviors of the majority is discriminated, and boxes of different colors are provided for marking.

Provided are a conversational artificial intelligence (AI) system and method based on real-time multimodal emotion recognition. The system includes a model server configured to provide a machine learning-based conversational model, a terminal configured to perform a conversation with the machine learning-based conversational model through the model server, display a virtual human responding to a user during a conversation with the user, and capture a facial image of the user during the conversation, and a multimodal empathetic conversation-generation system configured to access the model server and receive a response to a question of the user from the terminal, and assess an emotion of the user from the facial image of the user and control, based on the assessed emotion, an expression of the virtual human displayed on the terminal.

The invention discloses an emotion-enhanced digital human driving and presenting system and method, and belongs to the technical field of computers.The system comprises an audio decoupling module, a potential diffusion model module, a human motion prior module, a training module and a rendered image generation module; predicting and reconstructing the facial expression and gesture action sequence to obtain a final action sequence; and generating a neural rendering image based on the final action sequence. According to the method, through voice input, emotion and style features in voice are captured, a cross attention feature fusion mechanism is introduced to enhance expression of emotion and personal style features, a voice-driven gesture generation task is expanded to a facial expression and gesture action synchronous generation task, the integrity of generated digital human actions is realized, and the user experience is improved. The emotional expressive force of facial expressions and gestures of the digital human is greatly improved.

An autonomous environmental regulation and behavioral monitoring system is disclosed, configured to adapt temperature, lighting, and acoustic conditions based on real-time emotional and physiological data. The system includes a dual-redundant central processor, hierarchical communication networks, multi-angle visual acquisition units, infrared thermometers, and modular environmental subsystems. It detects posture, gestures, facial expressions, and thermal signals to classify user states and apply individualized airflow, light, and sound modulation without relying on external internet connectivity. The system also monitors connected appliances using voltage-based pressure analysis to forecast device degradation. With integrated gesture recognition, privacy-preserving data handling, and predictive adaptation, the invention enables multi-user personalization, long-term learning, and uninterrupted operation within residential, administrative, or healthcare infrastructures.

An AI-driven system for personalized advertising in real time, where: ◯ an analytics unit to monitor and analyze user behavior in real time across multiple digital platforms such as websites, mobile applications, social media, and smart devices; the unit collects data on user engagement, browsing patterns, time spent, and content preferences to enable targeted, personalized advertising; ◯ a prediction module that predicts preferences and interests of a user, operatively connected to the user interaction data collection unit, wherein the prediction module uses machine learning models such as deep learning, recurrent neural networks (RNNs) and transformer-based architectures to predict interests of users based on historical interactions and inferred preferences; ◯ an emotion and sentiment analysis unit that assesses the user's mood in real time through computer vision, natural language processing (NLP) and voice analysis, whereby the analysis of facial expressions, voice pitch and linguistic mood is used to determine emotional states and receptivity to advertising content; ◯ an embodiment of a context awareness component in operational communication with the emotion analysis and mood unit, in some cases further augmented by various environmental and situational data such as the device type and its physical location, date and time, and the content processed in the device, processing methods, etc., in establishing adaptability and automatic ad placement to be as relevant as possible to the user and their status as prescribed; ◯ Use reinforcement learning algorithms and generative AI models to drive advertising with personalization engines. Creative elements, messaging, and presentations are dynamically adjusted in real time based on user responses to ensure advertising is personalized and always optimized for best performance; o a privacy-focused AI system with federated learning, differential privacy methods, and on-device AI processing to reduce targeted advertising while complying with international data protection laws such as the General Data Protection Regulation (GDPR) and California consumer privacy laws; o an operationally adapted ad delivery mechanism to engage with real-time bidding (RTB) networks, programmatic advertising exchanges and demand-side platforms (DSPs) and place advertisements through digital advertising networks, which guarantees the delivery of tailored advertising to the most relevant audience in real time; and o a contextual feedback loop in which the machine learning models used in the preference and interest prediction module and in advertising personalization The engine is continuously updated to reflect the latest user engagement data, improving personalization over time and optimizing advertising performance.

An apparatus in one embodiment comprises at least one wearable device, with the at least one wearable device comprising at least one of a transmitter and a receiver of a wireless transmitter-receiver pair. The transmitter of the wireless transmitter-receiver pair transmits an acoustic signal, and the receiver of the wireless transmitter-receiver pair receives the acoustic signal. The received acoustic signal is processed utilizing a machine learning system to detect at least one characteristic of a user of the at least one wearable device. In some embodiments, a given wearable device comprises at least first and second wireless transmitter-receiver pairs, with the transmitter of each of the first and second wireless transmitter-receiver pairs transmitting an acoustic signal having a different carrier frequency. In such embodiments, a multi-channel echo profile may be generated using the multiple acoustic signals and classified by the machine learning system to detect the at least one characteristic.

The invention relates to the field of multi-modal interaction analysis, in particular to a multi-modal interaction method and system of a digital human agent. The method comprises the following steps: acquiring a real-time face image and a voice signal input stream of an interactive user based on an intelligent agent; performing real-time micro-expression recognition and deep emotion analysis based on the real-time facial image to obtain real-time emotion features of the user; performing time sequence evolution analysis on the real-time emotion characteristics of the user, performing holographic user emotion deep mining, and constructing a user emotion holographic characteristic spectrum; carrying out adaptive acoustic gain processing on the voice signal input stream, and carrying out voice-emotion association analysis based on the user emotion holographic characteristic spectrum to generate a voice-emotion linkage mapping spectrum; and carrying out eyeball fixation point migration tracking based on the user emotion holographic feature map and the real-time face image, and generating a user interaction depth intention signal. Through the real-time deep semantic understanding and emotion perception ability, the intelligent agent interaction intelligence and response accuracy are improved.

The invention provides an examination room abnormal behavior analysis method and system for examinees in a computer examination, and the method comprises the steps: extracting the facial expression, body posture information, keyboard tapping frequency, mouse movement track and other behavior data of the examinees, and forming a comprehensive behavior feature set; a time sequence prediction algorithm based on a long and short term memory network is combined with an attention mechanism to analyze a behavior pattern in a time dimension, and abnormal behavior information is obtained; creating a target examination room environment graph through a graph neural network algorithm, analyzing abnormal behavior information, and identifying collaborative cheating behaviors; introducing a Q learning reinforcement learning framework, and dynamically adjusting a detection strategy; performing weighted fusion on the abnormal behavior information, the collaborative cheating recognition result and the detection strategy by using an extreme gradient lifting learning algorithm, calculating a comprehensive score, and generating an abnormal behavior judgment report based on the comprehensive score; according to the method, the detection capability and efficiency of the abnormal behaviors and the cheating behaviors in the online examination are improved, and the fairness and fairness of the examination can be maintained.

The invention relates to the technical field of video behavior recognition, and discloses a behavior recognition and early warning system for a real-time video stream. The system comprises a spatio-temporal feature modeling module, a behavior fragment extraction module, an anomaly propagation modeling module, a risk area positioning module and an early warning strategy generation module. The spatial-temporal feature modeling module builds a dynamic model based on historical data, captures a skeleton key point three-dimensional coordinate sequence, a motion optical flow vector field and a micro-expression intensity spectrum, and outputs a theoretical behavior mode vector; the behavior fragment extraction module generates a multi-modal difference feature tensor through cross-modal difference analysis; the exception propagation modeling module generates an exception propagation path risk probability distribution cloud picture in combination with spatial constraint and trajectory information; the risk area positioning module identifies a high-risk area and marks a boundary; and the early warning strategy generation module dynamically configures monitoring parameters, starts high-frame-rate micro-expression capture for a high-risk area, and performs a track disturbance test on an adjacent area.

The invention discloses an AI digital human interaction system based on emotion recognition, relates to the technical field of emotion calculation, and improves the accuracy of emotion recognition through multi-modal emotion data acquisition and processing of an emotion primary recognition module and a secondary recognition module. The system effectively distinguishes complex emotions, such as fatigue and depression, of the trainee in the virtual education task by calculating the first emotion coefficient and the second emotion coefficient; secondary emotion recognition is carried out, so that misjudgment is reduced, and the emotion state recognition accuracy is ensured; through an emotion feedback module, the system dynamically adjusts the expression, intonation and interaction frequency of the AI digital human according to a second emotion coefficient, so that the emotion feedback is more natural, and the emotion recognition sense of the student is enhanced; besides, the culture adaptation analysis module collects cross-culture emotion data based on the culture background of the trainee, calculates and evaluates a culture adaptation coefficient, and automatically adjusts an emotion recognition and feedback strategy, thereby improving the adaptability of the system in a multi-culture environment and the accuracy of emotion expression.