A robot facial expression control system and method based on internal projection technology

By using a robot facial expression control system based on internal projection technology, facial expressions can be captured and analyzed in real time. This solves the problem that traditional robot systems cannot accurately interpret non-verbal expressions, enabling more natural and accurate human-computer interaction, improving user satisfaction and interactive experience, and making it suitable for a variety of application scenarios.

CN118700178BActive Publication Date: 2024-11-01SHENZHEN KINTO EXHIBITION
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411195463.8
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-08-29
Publication Date
2024-11-01
Estimated Expiration
2044-08-29

AI Technical Summary

Technical Problem

Traditional robotic systems cannot accurately capture and interpret human nonverbal expressions, resulting in a lack of naturalness and fluency in interactions, making them difficult to apply widely in various fields.

Method used

The robot facial expression control system based on internal projection technology includes modules for facial expression recognition, dialogue content matching, digital expression generation, and synchronous display. It combines high-resolution cameras, morphological filtering, and differential evolution algorithms to capture and analyze facial expressions in real time, and synchronously display expressions that match the interlocutor through internal projection technology.

Benefits of technology

It improves the naturalness and accuracy of human-computer interaction, enhances the robot's ability to understand and respond to user emotions, and improves user satisfaction and interactive experience. It is suitable for various scenarios such as virtual assistants, online education, and intelligent tour guides for exhibition halls and trade shows.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118700178B_ABST
    Figure CN118700178B_ABST
Patent Text Reader

Abstract

The present invention discloses a robot facial expression control system and method based on internal projection technology, which is used in the field of image processing technology. The robot facial expression control system includes: a facial expression recognition module, a dialogue content matching module, a digital expression generation module, a facial expression synchronization display module and a control module. The present invention captures and analyzes the facial expressions of the interlocutor in real time, so that the robot can respond to human emotions and reactions more naturally and accurately. The combination of the evolutionary algorithm and the LM algorithm optimizes the feature combination and classification model, improves the calculation efficiency, and ensures a faster processing speed. Through accurate and timely expression recognition and response, the robot can better understand and meet the user's emotions and needs, thereby improving the user's overall satisfaction and interactive experience.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to a robot facial expression control system and method based on internal projection technology. Background Art

[0002] A robot is an automatic machine that performs tasks by simulating human behavior or thinking patterns. They can be designed to perform a wide range of tasks from simple physical tasks to complex intelligent behaviors. The core of a robot lies in its automaticity and adaptability, and its ability to work independently without direct human operation. In modern technology, robots are widely used in manufacturing, medical care, service industry, education, home automation, and smart guides for exhibition halls and trade shows.

[0003] Intelligent tour guide robots can recognize visitors' facial expressions and non-verbal signals, thereby understanding their interests and emotional states; for example, when visitors appear curious or interested, the robot can automatically adjust its explanation content, provide more in-depth information or guide visitors to other related exhibits.

[0004] By simulating human facial expressions, robots can communicate with people more naturally, enhancing users' comfort and trust. Facial expressions are an important non-verbal information channel in human communication, which can convey emotions and intentions. When robots can understand and simulate these expressions, their interactions with people are more vivid and natural. Facial expression regulation makes robots not just cold execution machines, but can show emotional reactions and respond appropriately to human emotions. This ability is particularly important in the fields of service industry, medical care, education and exhibition hall intelligent guidance, because these fields require robots to show a higher level of emotional intelligence to better serve humans. Facial expression regulation can also help robots better adapt to complex social environments and understand human behavior and reactions; when interacting with people, robots can better understand the needs and feelings of the other party by observing and simulating human expressions, so as to make more appropriate responses.

[0005] Traditional robot systems are often unable to accurately capture and interpret human non-verbal expressions, such as facial expressions, when interacting with people, resulting in a lack of naturalness and fluency in the interaction. Previous facial expression recognition and processing systems often have slow processing speeds in complex situations and cannot meet the needs of real-time interaction. When generating dialogue content, traditional systems often cannot generate matching feedback based on the user's actual emotions, making the interaction lack personalization and realism. Many existing human-computer interaction technologies are difficult to be widely used in different fields, such as smart guided tours in exhibition halls, due to limited technical applicability and scalability.

[0006] Currently, no effective solution has been proposed for the problems in the related technologies. Summary of the invention

[0007] In order to overcome the above problems, the present invention aims to propose a robot facial expression control system and method based on internal projection technology, with the aim of solving the problem that traditional systems often cannot generate matching feedback according to the user's actual emotions when generating dialogue content, making the interaction lack of personalization and realism. Many existing human-computer interaction technologies are difficult to be widely used in different fields, such as smart guided tours in exhibition halls, due to limited technical applicability and scalability.

[0008] To this end, the specific technical solution adopted by the present invention is as follows:

[0009] According to one aspect of the present invention, a robot facial expression control system based on internal projection technology is provided, the robot facial expression control system comprising: a facial expression recognition module, a dialogue content matching module, a digital expression generation module, a facial expression synchronization display module and a control module;

[0010] A facial expression recognition module is used to collect facial expression images of the interlocutor using a camera pre-installed on the robot's eyes, and to extract features and classify the facial expression images using a facial expression classification model;

[0011] The dialogue content matching module is used to combine the facial expression features after feature extraction and classification with natural language processing technology to generate dialogue content that matches the facial expression features;

[0012] A digital expression generation module is used to convert the collected facial expression features into digital animations and present the interlocutor's facial expressions;

[0013] A facial expression synchronization display module is used to project digital animation onto the robot's facial screen using internal projection technology, and synchronously display facial expressions that match those of the interlocutor;

[0014] The control module is used to receive and update the latest facial expression features and conversation content in real time through connection with the cloud, and control the robot's facial expressions and voice output;

[0015] The facial expression recognition module uses a camera pre-installed on the robot's eyes to collect the interlocutor's facial expression images, and extracts and classifies the facial expression images through a facial expression classification model, including:

[0016] Using the camera pre-installed in the robot's eyes, the facial expression images of the interlocutor are captured in real time;

[0017] The facial expression features of facial expressions are extracted by using the mean of consecutive image frames as a dynamic threshold and combining it with multi-structural element morphological filtering technology;

[0018] The extracted facial expression features are optimized using differential evolution algorithm to determine the best feature combination and parameters;

[0019] A facial expression classification model is constructed based on the optimal feature combination and parameters, and the facial expression classification model is trained using the LM algorithm;

[0020] The performance of the trained facial expression classification model is evaluated, and each facial expression is recognized and classified.

[0021] Optionally, by using the mean of consecutive image frames as a dynamic threshold and combining the multi-structural element morphological filtering technique, the facial expression features of the facial expression are extracted including:

[0022] Obtaining the interlocutor's facial expression image and converting the facial expression image into a grayscale image;

[0023] Use basic morphological filtering to remove noise from grayscale images;

[0024] Calculate the global average grayscale value of each frame of grayscale image, and use the global average grayscale value of the previous frame as the adaptive threshold for processing the current frame;

[0025] According to the preset goal of facial expression recognition, the structural elements of facial expression features are defined, and morphological filtering is performed on the structural elements;

[0026] Based on the adaptive threshold, the structural elements after morphological filtering are binarized to identify and mark the facial feature areas;

[0027] By comparing the changes in position, size and shape of facial feature areas in consecutive frames, changes in facial expressions are detected and continuous tracking of facial expression changes is achieved.

[0028] Optionally, removing noise from the grayscale image using basic morphological filtering includes:

[0029] Perform visual or automated noise assessment on grayscale images to determine the type of noise;

[0030] According to the type of noise, the window size of the median filter is determined, and the window of the median filter is slid pixel by pixel on the grayscale image;

[0031] In each window, collect the gray values ​​of all pixels and calculate the median of the gray values;

[0032] Replace the pixel value at the center of the window with the median of all pixel grayscale values ​​in the current window;

[0033] The steps of calculating the median and replacing the median of the pixel grayscale value are repeated until each pixel of the grayscale image is processed, and whether the noise is effectively removed is evaluated.

[0034] Optionally, the formula for calculating the global average grayscale value of each frame of grayscale image is:

[0035] ;

[0036] In the formula, Represents the grayscale image in i Line j The global average gray value of the column pixels;

[0037] m Indicates the number of rows of the grayscale image;

[0038] n Represents the number of columns of the grayscale image;

[0039] Represents the grayscale image in i Line j Grayscale value of the column pixels.

[0040] Optionally, defining a structural element of a facial expression feature and performing morphological filtering on the structural element includes:

[0041] Defining structural elements a 1 and structural elements a 2 ;

[0042] Using structural elements a 1 For opening operations, using structural elements a 2 Used for closing operation;

[0043] Select structural elements of different sizes or shapes through opening and closing operations;

[0044] Depending on the result, adjust the shape, size, or order of operations of the structure elements.

[0045] Optionally, the definition of selecting structure elements of different sizes or shapes through opening and closing operations is:

[0046] ;

[0047] In the formula, , They represent different structural elements respectively;

[0048] Indicates the opening operation;

[0049] represents the closing operation;

[0050] Represents a grayscale image;

[0051] Represents the position of a pixel in a grayscale image;

[0052] Indicates that the image after the morphological filtering operation is at position x The pixel value of .

[0053] Optionally, the extracted facial expression features are optimized using a differential evolution algorithm to determine the best feature combination and parameters including:

[0054] The goal is to improve the accuracy of facial expression classification, and the classification accuracy is used as the fitness function;

[0055] Randomly generate an initial population containing a variety of feature combinations and parameter settings;

[0056] In the process of differential evolution, individuals in the initial population are iteratively optimized through mutation, crossover, and selection operations;

[0057] The differential evolution process is repeated until the maximum number of iterations is reached, and the individual with the highest fitness is selected from the optimized initial population to obtain the optimal feature combination and parameters.

[0058] Optionally, constructing a facial expression classification model based on the best feature combination and parameters, and training the facial expression classification model using the LM algorithm includes:

[0059] Construct the architecture of the facial expression classification model and determine the type, number and parameters of each layer in the architecture;

[0060] The best feature combination determined by the differential evolution algorithm is divided and standardized;

[0061] Set the hyperparameters of the LM algorithm;

[0062] The processed features are used to train the facial expression classification model, the prediction results of the model are calculated through forward propagation, and the hyperparameters of the LM algorithm are used to back propagate the errors and update the model parameters;

[0063] Evaluate model performance on the validation set and adjust hyperparameters or model architecture based on the evaluation results;

[0064] Test the performance of the facial expression classification model on the test set to evaluate its accuracy in real-world applications.

[0065] Optionally, the conversation content matching module combines the facial expression features after feature extraction and classification with natural language processing technology to generate conversation content matching the facial expression features, including:

[0066] Convert the recognized facial expression features into a format for natural language processing;

[0067] Use sentiment analysis in natural language processing technology to make sentiment judgments on the converted facial expression features and determine the emotional state of the interlocutor;

[0068] Combining the analyzed emotion and facial expression features, the preset rules are used to generate dialogue content that matches the current emotional state.

[0069] According to another aspect of the present invention, a robot facial expression control method based on internal projection technology is also provided, and the robot facial expression control method comprises the following steps:

[0070] S1, using the camera pre-installed on the robot's eyes to collect the interlocutor's facial expression images, and extracting and classifying the facial expression images through a facial expression classification model;

[0071] S2, combining the facial expression features extracted and classified with natural language processing technology to generate dialogue content matching the facial expression features;

[0072] S3, converting the collected facial expression features into digital animations and presenting the interlocutor's facial expressions;

[0073] S4, using internal projection technology to project digital animation onto the robot's facial screen, synchronously displaying facial expressions that match those of the interlocutor;

[0074] S5. Through the connection with the cloud, the robot receives and updates the latest facial expression features and conversation content in real time, and regulates the robot's facial expressions and voice output.

[0075] Compared with the prior art, this application has the following beneficial effects:

[0076] 1. By capturing and analyzing the facial expressions of the interlocutor in real time, the present invention enables the robot to respond to human emotions and reactions more naturally and accurately, thereby enhancing the naturalness and effect of human-computer interaction. High-resolution cameras and advanced image processing technologies (such as morphological filtering and median filtering) are used to ensure accurate extraction of facial features. The optimized use of dynamic thresholds and structural elements can adapt to different environmental changes, thereby maintaining a high recognition accuracy rate. By using the average grayscale value of the previous frame as the dynamic threshold of the current frame, it can adapt to different lighting and background conditions and improve the performance stability in complex environments. The combination of differential evolution algorithm and LM algorithm optimizes the feature combination and classification model, improves computing efficiency, and ensures a faster processing speed. Through accurate and timely expression recognition and response, the robot can better understand and meet the user's emotions and needs, thereby improving the user's overall satisfaction and interactive experience.

[0077] 2. The present invention combines facial expression features with natural language processing technology to generate dialogue content that matches the current emotional state, which increases the personalization and realism of the interaction. The digital expression generation and synchronous display module further enhances this authenticity, allowing the robot to present the interlocutor's expression in an almost real way. The synchronous display of dialogue content and expression can greatly improve the user's sense of participation and satisfaction. This tight feedback loop makes users feel that the robot can truly understand and reflect their emotions and expressions, thereby establishing a stronger emotional connection.

[0078] 3. The present invention ensures smooth display of animation and efficient performance through advanced graphics rendering technology and real-time data processing, which is crucial for real-time interactive environments that require quick response. It is not limited to face-to-face interaction scenarios, but can also be applied to virtual assistants, online education, telemedicine, smart guides in exhibition halls and exhibitions, and other scenarios. The wide applicability of the technology provides huge market potential. BRIEF DESCRIPTION OF THE DRAWINGS

[0079] The above characteristics, features and advantages of the present invention and their implementation methods and methods will become more clearly understood in conjunction with the following description of the embodiments, which are described in detail in conjunction with the accompanying drawings. Herein, a schematic diagram is shown:

[0080] Figure 1 is a principle block diagram of a robot facial expression control system based on internal projection technology according to an embodiment of the present invention;

[0081] Figure 2 This is a flow chart of a robot facial expression control system based on internal projection technology according to an embodiment of the present invention.

[0082] In the figure:

[0083] 1. Facial expression recognition module; 2. Dialogue content matching module; 3. Digital expression generation module; 4. Facial expression synchronization display module; 5. Control module. DETAILED DESCRIPTION

[0084] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0085] According to an embodiment of the present invention, a robot facial expression control system and method based on internal projection technology are provided.

[0086] The present invention is further described with reference to the accompanying drawings and specific embodiments. Figure 1 As shown, according to one embodiment of the present invention, a robot facial expression control system based on internal projection technology is provided, and the robot facial expression control system comprises: a facial expression recognition module 1, a dialogue content matching module 2, a digital expression generation module 3, a facial expression synchronization display module 4 and a control module 5;

[0087] The facial expression recognition module 1 is used to collect the facial expression image of the interlocutor by using the camera pre-installed in the robot's eye, and to extract features and classify the facial expression image by using the facial expression classification model.

[0088] The facial expression recognition module 1 collects the facial expression image of the interlocutor by using the camera pre-installed on the robot eye, and extracts and classifies the facial expression image by the facial expression classification model, including:

[0089] Using the camera pre-installed in the robot's eyes, the facial expression images of the interlocutor are captured in real time.

[0090] It should be explained that one or more high-resolution cameras are installed at the "eye" position of the robot to ensure that the camera can cover a sufficient field of view to capture the facial expressions of all interlocutors interacting with the robot, and at the same time adjust the focus and angle of the camera to optimize the image quality and ensure that facial features are clearly visible; perform real-time processing of the video stream, such as adjusting the brightness and contrast, to adapt to different ambient light conditions.

[0091] The facial expression features of facial expressions are extracted by utilizing the mean of consecutive image frames as a dynamic threshold and combining it with multi-structural element morphological filtering technology.

[0092] Preferably, by using the mean of consecutive image frames as a dynamic threshold and combining the multi-structural element morphological filtering technology, the facial expression features of the facial expression are extracted, including:

[0093] Obtain the interlocutor's facial expression image and convert the facial expression image into a grayscale image.

[0094] Remove noise from grayscale images using basic morphological filtering.

[0095] Preferably, removing noise in a grayscale image by using basic morphological filtering comprises:

[0096] Perform visual or automated noise assessment on grayscale images to determine the type of noise;

[0097] According to the type of noise, the window size of the median filter is determined, and the window of the median filter is slid pixel by pixel on the grayscale image;

[0098] In each window, collect the gray values ​​of all pixels and calculate the median of the gray values;

[0099] Replace the pixel value at the center of the window with the median of all pixel grayscale values ​​in the current window;

[0100] The steps of calculating the median and replacing the median of the pixel grayscale value are repeated until each pixel of the grayscale image is processed, and whether the noise is effectively removed is evaluated.

[0101] Preferably, the formula for calculating the global average grayscale value of each frame of grayscale image is:

[0102] ;

[0103] In the formula, Represents the grayscale image in i Line j The global average gray value of the column pixels;

[0104] m Indicates the number of rows of the grayscale image;

[0105] n Represents the number of columns of the grayscale image;

[0106] Represents the grayscale image in i Line j Grayscale value of the column pixels.

[0107] It should be explained that converting the captured color image into a grayscale image reduces the complexity of processing the data because the grayscale image only contains brightness information, which is sufficient for subsequent noise assessment and filtering processing. Visual or automated noise assessment to determine the noise type (such as random noise, salt and pepper noise, etc.) is crucial to selecting the appropriate filtering method.

[0108] According to the type and degree of noise, select a median filter window of appropriate size. The window size affects the filtering effect and the degree of detail retention. Slide the median filter window pixel by pixel on the image, covering each pixel and the pixels around it. In each window, calculate the median of the grayscale values ​​of all covered pixels, and replace the pixel value at the center of the window with this median. Repeat the above calculation and replacement steps until the entire image processing is completed, ensuring that the noise is effectively removed while retaining the necessary image details.

[0109] The global average grayscale value of the previous frame is used as the dynamic threshold for processing the current frame. This can adapt to changes in different lighting and background conditions, making the threshold setting more flexible and adaptable. According to the specific characteristics of facial expressions, structural elements of different shapes and sizes are selected for morphological filtering. The image is processed using opening and closing operations in turn, and different structural elements are used to optimize feature extraction. Positive and negative pulse noises are effectively processed while maintaining key facial features.

[0110] Calculate the global average grayscale value of each frame of grayscale image, and use the global average grayscale value of the previous frame as the adaptive threshold for processing the current frame.

[0111] According to the preset goal of facial expression recognition, the structural elements of facial expression features are defined, and morphological filtering is performed on the structural elements.

[0112] Preferably, defining the structural elements of the facial expression features and performing morphological filtering on the structural elements comprises:

[0113] Defining structural elements a 1 and structural elements a 2 ;

[0114] Using structural elements a 1 For opening operations, using structural elements a 2 Used for closing operation;

[0115] Select structural elements of different sizes or shapes through opening and closing operations;

[0116] Depending on the result, adjust the shape, size, or order of operations of the structure elements.

[0117] Preferably, the definition of selecting structural elements of different sizes or shapes by opening and closing operations is:

[0118] ;

[0119] In the formula, , They represent different structural elements respectively;

[0120] Indicates the opening operation;

[0121] represents the closing operation;

[0122] Represents a grayscale image;

[0123] Represents the position of a pixel in a grayscale image;

[0124] Indicates that the image after the morphological filtering operation is at position x The pixel value of .

[0125] It should be explained that in the process of calculating the global average grayscale value of each frame of grayscale image, first, ensure that the input is a grayscale image, each pixel value represents brightness, traverse all pixels in the image, sum them up and divide them by the total number of pixels in the image (that is, the width of the image multiplied by the height) to get the global average grayscale value of the frame. Before processing the current frame, save the average grayscale value of the previous frame, use the average grayscale value of the previous frame as the threshold for processing the current frame, and perform binarization or other threshold-related operations.

[0126] Structural element a 1 : Selects smaller or specifically shaped structural elements, suitable for opening operations, used to remove small noise or details and smooth the edges of larger objects.

[0127] Structural element a 2 : Choose larger or with a 1 Structural elements of different shapes are used for closing operations to fill small holes and cracks that may be produced by opening operations and maintain the integrity of features.

[0128] Open operation (using the structural element a 1 ) includes the following steps:

[0129] 1) Corrosion: First use the structural element a 1 Perform an erosion operation on the image to reduce the bright areas in the image.

[0130] 2) Dilation: The eroded image is then dilated using the same structural element to restore the size of the main features and smooth the edges.

[0131] Closing operation (using the structural element a 2 ) includes the following steps:

[0132] 1) Expansion: First use the structural element a 2 Dilate the image to increase the bright areas in the image.

[0133] 2) Erosion: The dilated image is then eroded using the same structuring element to fill small holes and cracks, preserving an approximate original shape.

[0134] Observe the effects of opening and closing operations, and check whether the target facial features, such as the edges of the eyes, nose, and mouth, are effectively extracted. Depending on the quality of the extraction, you may need to adjust the size or shape of the structural elements or replace different structural elements. Consider adjusting the order of operations, such as performing closing operations before opening operations, especially in different feature extraction scenarios where different strategies may be required.

[0135] Based on the adaptive threshold, the structural elements after morphological filtering are binarized to identify and mark the facial feature areas.

[0136] By comparing the changes in position, size and shape of facial feature areas in consecutive frames, changes in facial expressions are detected and continuous tracking of facial expression changes is achieved.

[0137] It should be explained that after completing the morphological filtering (opening and closing operations), the main features in the image have been enhanced, while noise and unnecessary details have been suppressed; the global average grayscale value calculated from the previous frame is used as the threshold of the current frame. This threshold is dynamically adjusted to adapt to different lighting and background conditions; the image after morphological filtering is binarized, that is, for each pixel, if its grayscale value is higher than the threshold, it is set to white (or 1), otherwise it is set to black (or 0). In the binary image, since the main facial features such as eyes, nose and mouth have been enhanced by morphological filtering, these areas will appear as more obvious white areas; image processing techniques such as connected component analysis are used to identify and mark these white facial feature areas.

[0138] Store the position, size and shape information of each facial feature area in consecutive frames; compare the changes of the corresponding feature area in the current frame with the previous frame. This includes analyzing the geometric changes of the feature area (such as area size, shape deformation, etc.) and the offset of the position, and applying tracking algorithms, such as Kalman Filter or Optical Flow, to predict and track the movement trajectory of facial features. The algorithm can predict the future state based on past data, thereby achieving continuous tracking of facial features, especially when the interlocutor moves faster, it can still maintain the accuracy of tracking.

[0139] The extracted facial expression features are optimized using differential evolution algorithm to determine the best feature combination and parameters.

[0140] Preferably, the extracted facial expression features are optimized using a differential evolution algorithm to determine the best feature combination and parameters including:

[0141] The goal is to improve the accuracy of facial expression classification, and the classification accuracy is used as the fitness function;

[0142] Randomly generate an initial population containing a variety of feature combinations and parameter settings;

[0143] In the process of differential evolution, individuals in the initial population are iteratively optimized through mutation, crossover, and selection operations;

[0144] The differential evolution process is repeated until the maximum number of iterations is reached, and the individual with the highest fitness is selected from the optimized initial population to obtain the optimal feature combination and parameters.

[0145] It should be explained that the accuracy of facial expression classification is improved. In this step, the goal is to find the feature combination and parameters that can maximize the classification accuracy. The fitness function is used to evaluate the effect of each feature combination. The fitness function is defined as the classification accuracy, that is, the accuracy of the feature combination in classifying facial expressions. An initial population containing multiple feature combinations and parameter settings is randomly generated. Each individual represents a specific set of features and parameters. In order to explore the search space, the individuals in the current population are mutated. Usually three different individuals are selected, and their difference is added to another individual to generate a new candidate individual. The crossover operation is used to increase the diversity of the population. The mutated candidate individual is combined with an individual in the current population, and their features are exchanged with a certain probability to generate the final candidate individual. The newly generated candidate individual is compared with the original individual in the current population. The individual with better performance is retained according to the fitness function (classification accuracy). The mutation, crossover and selection operations are repeated, and each iteration attempts to find a better solution. This process usually sets a maximum number of iterations, or stops when a certain stopping condition is reached (such as fitness no longer significantly improving). After the maximum number of iterations is reached, the individual with the highest fitness is selected from the optimized population, that is, the feature combination and parameter setting with the highest classification accuracy.

[0146] A facial expression classification model is constructed based on the optimal feature combination and parameters, and the facial expression classification model is trained using the LM algorithm.

[0147] Preferably, constructing a facial expression classification model based on an optimal feature combination and parameters, and training the facial expression classification model using the LM algorithm includes:

[0148] Construct the architecture of the facial expression classification model and determine the type, number and parameters of each layer in the architecture;

[0149] The best feature combination determined by the differential evolution algorithm is divided and standardized;

[0150] Set the hyperparameters of the LM algorithm;

[0151] The processed features are used to train the facial expression classification model, the prediction results of the model are calculated through forward propagation, and the hyperparameters of the LM algorithm are used to back propagate the errors and update the model parameters;

[0152] Evaluate model performance on the validation set and adjust hyperparameters or model architecture based on the evaluation results;

[0153] Test the performance of the facial expression classification model on the test set to evaluate its accuracy in real-world applications.

[0154] The performance of the trained facial expression classification model is evaluated, and each facial expression is recognized and classified.

[0155] It should be explained that the facial expression classification model is a convolutional neural network model. The number of layers of the facial expression classification model, the type of each layer (such as convolutional layer, pooling layer, fully connected layer) and the corresponding parameters (such as filter size, step size, activation function, etc.) are designed. The feature combination selected by the differential evolution algorithm is appropriately divided into training set, validation set and test set. The features are standardized to ensure that the scale of input features is consistent during model training, which helps to accelerate the convergence speed of the model training process and select appropriate hyperparameters for the LM algorithm (Levenberg-Marquardt algorithm), such as learning rate, error tolerance, etc. The LM algorithm is a An effective nonlinear minimization (optimization) method, commonly used for training neural networks, especially for small to medium-sized problems. It uses the current model parameters to calculate the prediction results of the training data, calculates the error between the prediction results and the actual labels, and uses the LM algorithm to back propagate the error, update the model parameters, and test the performance of the model on the validation set, which helps to monitor the generalization ability of the model on unseen data. It also adjusts the hyperparameters or model architecture based on the results, runs the model on the test set to evaluate its performance in actual application scenarios, and performs a final performance evaluation on the trained model, including recognizing different facial expressions and classifying and counting the recognition accuracy of various expressions.

[0156] Defining layers and structures includes:

[0157] 1) Input layer: Set up the input layer to receive facial images. The size of the image needs to be pre-defined, such as 64x64 or 128x128 pixels.

[0158] 2) Convolutional layers: Add multiple convolutional layers. Each convolutional layer is usually followed by an activation function, such as ReLU. The convolutional layer is used to extract features from the image. The number and size of filters are determined in this step.

[0159] 3) Pooling layer: Adding a pooling layer, such as max pooling, after the convolutional layer is used to reduce the feature dimension and enhance the generalization ability of the model;

[0160] 4) Fully connected layer: A fully connected layer is added at the end of the convolutional neural network model to map the learned features to the classification results;

[0161] 5) Output layer: The number of neurons in the output layer should match the number of expression categories in the classification task, and the softmax function is used as the activation function to output the probability of each category.

[0162] The dialogue content matching module 2 is used to combine the facial expression features after feature extraction and classification with natural language processing technology to generate dialogue content that matches the facial expression features.

[0163] Preferably, the dialogue content matching module combines the facial expression features after feature extraction and classification with natural language processing technology to generate dialogue content matching the facial expression features, including:

[0164] Convert the recognized facial expression features into a format for natural language processing;

[0165] Use sentiment analysis in natural language processing technology to make sentiment judgments on the converted facial expression features and determine the emotional state of the interlocutor;

[0166] Combining the analyzed emotion and facial expression features, the preset rules are used to generate dialogue content that matches the current emotional state.

[0167] It should be explained that, first, the output results of the facial expression classification model (such as labels or probabilities of different expressions) are converted into a format suitable for NLP processing; for example, the expression recognition results can be converted into labels or descriptive texts, such as "happy", "sad", etc., and these data are organized so that they can be used as input to the sentiment analysis module, including encoding conversion or adaptation to the format required by a specific NLP library. The converted facial expression data are analyzed using sentiment analysis tools in natural language processing technology. The tools are usually based on pre-trained models and can determine the emotional tendency expressed by the text. The possible emotional state of the interlocutor is determined based on the sentiment analysis results, which will directly affect the emotional color and style of the generated dialogue content. According to the emotional state, rules for generating dialogue content are set. The rules can be template-based or more complex machine learning-based generation models. According to preset rules or models, dialogue content that matches the current emotional state is generated. For example, if the emotional state is "happy", the generated dialogue content may be encouraging or positive; if the emotional state is "sad", the generated content may be comforting or sympathetic.

[0168] The specific example is as follows: Using sentiment analysis in natural language processing technology to make sentiment judgments on the converted facial expression features and determine the emotional state of the interlocutor includes:

[0169] Ensure that each recognized facial expression and its possible expression labels, such as "happy", "sad", "surprised", etc., are obtained from the facial expression recognition system;

[0170] Convert the labels of facial expressions into text descriptions. For example, if the facial expression model outputs the expression "happy", then convert it into the text "This person looks happy";

[0171] Use sentiment analysis tools in natural language processing libraries (such as NLTK, TextBlob, or more advanced deep learning models such as BERT, ChatMindAi, etc.), which can usually analyze the sentiment tendency of text data;

[0172] Enter the text description of the facial expression into the sentiment analysis tool. These tools will analyze the text based on the trained model to determine its emotional tendency;

[0173] Sentiment analysis tools usually output sentiment polarity (positive, negative) and sentiment intensity or confidence. For example, for the text "This person has a happy expression", the tool may judge it as a positive sentiment;

[0174] Based on the result of sentiment analysis, the emotional state of the interlocutor is determined. For example, if the result of sentiment analysis is positive, the emotional state of the interlocutor may be determined to be happy or satisfied.

[0175] Integrate the emotional state obtained from sentiment analysis into the dialogue system to influence the generated dialogue content to make it more consistent with the user's emotional state;

[0176] Adjust the dialogue generation strategy based on the emotional state. For example, if a happy emotion is detected, generate more positive and encouraging dialogue content.

[0177] The digital expression generation module 3 is used to convert the collected facial expression features into digital animations and present the facial expressions of the interlocutor.

[0178] It should be explained that facial recognition technology (such as OpenCV, dlib or more advanced deep learning methods) is used to capture the facial feature points of the interlocutor in the real-time video. The facial feature points include key areas such as eyes, mouth, eyebrows, etc. The captured facial data is converted into a set of standardized facial expression parameters, such as the degree of opening and closing of the eyes, the degree of opening and closing of the mouth, etc., and the complex facial dynamics are simplified into numerical data that can be used for animation. A digital facial model is designed or selected. Usually this model is a pre-created 3D model with a certain range of expression dynamics. According to the extracted facial expression parameters, the corresponding feature points of the animation model are adjusted. For example, if the captured facial expression parameters are If the data obtained shows that the interlocutor is smiling, the degree of upward movement of the animation model's mouth corners will be increased, and the adjusted animation model will be rendered into real-time video output using graphics rendering technology (such as OpenGL or DirectX), including texture, lighting, and shadow processing, to make the animation as realistic as possible, and ensure performance optimization of the rendering process so that the animation can be presented in real time, including adjusting rendering settings, optimizing data processing procedures, etc., integrating the digital expression generation module with other systems (such as the dialogue system or sentiment analysis module) to ensure smooth data transmission between modules, adjusting module parameters based on user feedback and system usage, and optimizing the naturalness and reaction speed of the expression.

[0179] The facial expression synchronous display module 4 is used to project the digital animation onto the robot's facial screen using the internal projection technology, and synchronously display the facial expression that matches the interlocutor.

[0180] It needs to be explained that the pre-generated animations are received from the digital expression generation module, which are based on the facial expression features captured in real time and have been converted and rendered into digital form, ensuring that the format and resolution of the animations are suitable for the specifications of the robot's facial screen, converting the video format or adjusting the image size and resolution to fit the screen, and selecting a suitable facial screen technology, usually a flexible OLED or LED panel, which can fit the curve of the robot's face and provide high-quality image display.

[0181] Mount the screen on the robot's head to ensure that it can be precisely aligned and fixed to maintain the stability and continuity of the image, configure internal projection, involving micro-projectors and mirror reflection components to ensure that the image can be correctly mapped onto the facial screen, adjust the alignment and focus of the projection to ensure that the animated image accurately covers the corresponding position of the robot's facial screen without distortion and correct proportions, achieve real-time data processing and transmission, ensure that there is no delay in the transmission of animation from the digital expression generation module to the robot's facial screen, dynamically adjust the image's brightness, contrast, and color settings based on the observed display effect and user feedback to improve the naturalness and expressiveness of the expression, conduct comprehensive tests to verify the accuracy and effect of the synchronous display of facial expressions, and ensure the seamless operation of all components and modules.

[0182] Example scenario

[0183] Consider a robot whose task is to interact with customers in a store. The robot is equipped with various modules, including facial expression recognition, digital expression generation, facial expression synchronization display, and dialogue content matching modules.

[0184] The expression recognition module includes the following steps:

[0185] 1) Camera setup: The robot’s “eyes” are equipped with high-resolution cameras that cover the entire reception area, ensuring that every customer’s facial expression can be clearly captured.

[0186] 2) Real-time image processing: The camera captures images in real time and automatically adjusts brightness and contrast to suit the lighting in the store. The image is converted to grayscale, morphological filtering technology is used to remove noise, and then structural elements are used to optimize expression feature extraction.

[0187] The digital expression generation module includes the following steps:

[0188] 1) Feature extraction: Extract key parameters (such as eye openness and mouth openness) from the recognized facial expressions.

[0189] 2) Animation Mapping: Mapping these parameters onto a preset 3D facial model, adjusting the model to match the captured expressions.

[0190] 3) Rendering output: Use graphics processing technology to render animation and output high-quality digital animation in real time.

[0191] The facial expression synchronization display module includes the following steps:

[0192] 1) Internal projection setup: Using the internal projection technology, the animation is projected onto the flexible OLED screen on the robot’s face.

[0193] 2) Image adjustment and synchronization: Ensure that the animated image is perfectly aligned with the robot’s facial screen, adjust the focus and scale, and synchronously display the animation to match the customer’s expression.

[0194] The conversation content matching module includes the following steps:

[0195] 1) Sentiment analysis: Convert facial expression labels into text (such as "happy") and use NLP tools to perform sentiment analysis to determine the customer's emotional state.

[0196] 2) Content generation: Based on the analyzed emotions, the robot generates corresponding conversation content. For example, when the customer is happy, it generates encouraging and positive conversations.

[0197] The control module 5 is used to receive and update the latest facial expression features and dialogue content in real time through connection with the cloud, and to control the robot's facial expression and voice output.

[0198] It is necessary to explain that cloud services are set up and configured to ensure secure data transmission and access permissions. This may involve the use of API interfaces, security tokens, and encrypted communication protocols (such as HTTPS) to implement a continuous network connection mechanism so that the robot can receive data updates and commands from the cloud in real time, and receive facial expression features and dialogue content updates from the cloud in real time. This includes new expression data, improved dialogue strategies, or other related information updates, processing and parsing the received data, converting it into a format that the robot system can recognize and execute, adjusting the expression animation on the robot's facial screen based on the latest facial expression data received to reflect the latest expression features, and adjusting the robot's voice output parameters such as intonation, speed, and voice content based on the updated dialogue content to more naturally synchronize with the dialogue content and expressions.

[0199] According to another embodiment of the present invention, Figure 2 As shown, a robot facial expression control method based on internal projection technology is also provided, and the robot facial expression control method comprises the following steps:

[0200] S1, using the camera pre-installed on the robot's eyes to collect the interlocutor's facial expression images, and extracting and classifying the facial expression images through a facial expression classification model;

[0201] S2, combining the facial expression features extracted and classified with natural language processing technology to generate dialogue content matching the facial expression features;

[0202] S3, converting the collected facial expression features into digital animations and presenting the interlocutor's facial expressions;

[0203] S4, using internal projection technology to project digital animation onto the robot's facial screen, synchronously displaying facial expressions that match those of the interlocutor;

[0204] S5. Through the connection with the cloud, the robot receives and updates the latest facial expression features and conversation content in real time, and regulates the robot's facial expressions and voice output.

[0205] In summary, with the help of the above technical solutions of the present invention, the present invention can capture and analyze the facial expressions of the interlocutor in real time, so that the robot can respond to human emotions and reactions more naturally and accurately, thereby enhancing the naturalness and effect of human-computer interaction. The high-resolution camera and advanced image processing technology (such as morphological filtering and median filtering) are used to ensure the accurate extraction of facial features. The optimized use of dynamic thresholds and structural elements can adapt to different environmental changes, thereby maintaining a high recognition accuracy rate. By using the average gray value of the previous frame as the dynamic threshold of the current frame, it can adapt to different lighting and background conditions and improve the performance stability in complex environments. The combination of the evolutionary algorithm and the LM algorithm optimizes the feature combination and classification model, improves the calculation efficiency, and ensures a faster processing speed. Through accurate and timely expression recognition and response, the robot can better understand and meet the user's emotions and needs, thereby improving the user's overall satisfaction and interactive experience. The present invention combines facial expression features with natural language processing technology to generate dialogue content that matches the current emotional state, which increases the personalization and realism of the interaction. The digital expression generation and synchronous display module further enhances this authenticity, allowing the robot to present the expressions of the interlocutor in an almost realistic way. The synchronous display of dialogue content and expressions can greatly improve the user's sense of participation and satisfaction. This tight feedback loop makes the user feel that the robot can truly understand and reflect their emotions and expressions, thereby establishing a stronger emotional connection. The present invention ensures smooth display of animation and efficient performance through advanced graphics rendering technology and real-time data processing, which is crucial for real-time interactive environments that require rapid response. It is not limited to face-to-face interaction scenarios, but can also be applied to virtual assistants, online education, telemedicine, and smart guides for exhibition halls and trade shows. The wide applicability of its technology provides huge market potential.

[0206] Although the present invention has been disclosed as above with preferred embodiments, the embodiments are merely examples for the convenience of description and are not intended to limit the present invention. Those skilled in the art may make several changes and modifications without departing from the spirit and scope of the present invention. The scope of protection claimed by the present invention shall be based on the claims.

Claims

1. A robot facial expression control system based on internal projection technology, characterized in that: The robot facial expression control system includes: a facial expression recognition module, a dialogue content matching module, a digital expression generation module, a facial expression synchronization display module and a control module; A facial expression recognition module is used to collect facial expression images of the interlocutor using a camera pre-installed on the robot's eyes, and to extract features and classify the facial expression images using a facial expression classification model; The dialogue content matching module is used to combine the facial expression features after feature extraction and classification with natural language processing technology to generate dialogue content that matches the facial expression features; A digital expression generation module is used to convert the collected facial expression features into digital animations and present the interlocutor's facial expressions; A facial expression synchronization display module is used to project digital animation onto the robot's facial screen using internal projection technology, and synchronously display facial expressions that match those of the interlocutor; The control module is used to receive and update the latest facial expression features and conversation content in real time through connection with the cloud, and control the robot's facial expressions and voice output; The facial expression recognition module collects the interlocutor's facial expression image by using the camera pre-installed on the robot's eye, and extracts and classifies the facial expression image by using the facial expression classification model, including: Using the camera pre-installed in the robot's eyes, the facial expression images of the interlocutor are captured in real time; The facial expression features of facial expressions are extracted by using the mean of consecutive image frames as a dynamic threshold and combining it with multi-structural element morphological filtering technology; The extracted facial expression features are optimized using differential evolution algorithm to determine the best feature combination and parameters; A facial expression classification model is constructed based on the optimal feature combination and parameters, and the facial expression classification model is trained using the LM algorithm; Evaluate the performance of the trained facial expression classification model and recognize and classify each facial expression; The facial expression features of facial expressions are extracted by using the mean of continuous image frames as a dynamic threshold and combining the multi-structural element morphological filtering technology, including: Obtaining the interlocutor's facial expression image and converting the facial expression image into a grayscale image; Use basic morphological filtering to remove noise from grayscale images; Calculate the global average grayscale value of each frame of grayscale image, and use the global average grayscale value of the previous frame as the adaptive threshold for processing the current frame; According to the preset goal of facial expression recognition, the structural elements of facial expression features are defined, and morphological filtering is performed on the structural elements; Based on the adaptive threshold, the structural elements after morphological filtering are binarized to identify and mark the facial feature areas; By comparing the changes in the position, size and shape of facial feature areas in consecutive frames, the changes in facial expressions are detected and the continuous tracking of facial expression changes is achieved; The method of constructing a facial expression classification model based on the optimal feature combination and parameters and training the facial expression classification model using the LM algorithm includes: Construct the architecture of the facial expression classification model and determine the type, number and parameters of each layer in the architecture; The best feature combination determined by the differential evolution algorithm is divided and standardized; Set the hyperparameters of the LM algorithm; The processed features are used to train the facial expression classification model, the prediction results of the model are calculated through forward propagation, and the hyperparameters of the LM algorithm are used to back propagate the errors and update the model parameters; Evaluate model performance on the validation set and adjust hyperparameters or model architecture based on the evaluation results; Test the performance of the facial expression classification model on the test set to evaluate its accuracy in real-world applications.

2. A robot facial expression control system based on internal projection technology according to claim 1, characterized in that: The method of removing noise in the grayscale image by using basic morphological filtering comprises: Perform visual or automated noise assessment on grayscale images to determine the type of noise; According to the type of noise, the window size of the median filter is determined, and the window of the median filter is slid pixel by pixel on the grayscale image; In each window, collect the gray values ​​of all pixels and calculate the median of the gray values; Replace the pixel value at the center of the window with the median of all pixel grayscale values ​​in the current window; The steps of calculating the median and replacing the median of the pixel grayscale value are repeated until each pixel of the grayscale image is processed, and whether the noise is effectively removed is evaluated.

3. A robot facial expression control system based on internal projection technology according to claim 2, characterized in that: The formula for calculating the global average grayscale value of each frame of grayscale image is: ; In the formula, Represents the grayscale image in i Line j The global average gray value of the column pixels; m Indicates the number of rows of the grayscale image; n Represents the number of columns of the grayscale image; Represents the grayscale image in i Line j Grayscale value of the column pixels.

4. A robot facial expression control system based on internal projection technology according to claim 3, characterized in that: The structural elements defining facial expression features and performing morphological filtering on the structural elements include: Defining structural elements a 1 and structural elements a 2; Using structural elements a 1 is used for opening operation, using structural elements a 2 is used for closing operation; Select structural elements of different sizes or shapes through opening and closing operations; Depending on the result, adjust the shape, size, or order of operations of the structure elements.

5. A robot facial expression control system based on internal projection technology according to claim 4, characterized in that: The definition of selecting structural elements of different sizes or shapes through opening and closing operations is: ; In the formula, , They represent different structural elements respectively; Indicates the opening operation; represents the closing operation; Represents a grayscale image; Represents the position of a pixel in a grayscale image; Indicates that the image after the morphological filtering operation is at position x The pixel value of .

6. A robot facial expression control system based on internal projection technology according to claim 2, characterized in that: The method of optimizing the extracted facial expression features using a differential evolution algorithm and determining the best feature combination and parameters includes: The goal is to improve the accuracy of facial expression classification, and the classification accuracy is used as the fitness function; Randomly generate an initial population containing a variety of feature combinations and parameter settings; In the process of differential evolution, individuals in the initial population are iteratively optimized through mutation, crossover, and selection operations; The differential evolution process is repeated until the maximum number of iterations is reached, and the individual with the highest fitness is selected from the optimized initial population to obtain the optimal feature combination and parameters.

7. The robot facial expression control system based on internal projection technology according to claim 1, characterized in that: The dialogue content matching module combines the facial expression features extracted and classified with the natural language processing technology to generate dialogue content matching the facial expression features, including: Convert the recognized facial expression features into a format for natural language processing; Use sentiment analysis in natural language processing technology to make sentiment judgments on the converted facial expression features and determine the emotional state of the interlocutor; Combining the analyzed emotion and facial expression features, the preset rules are used to generate dialogue content that matches the current emotional state.

8. A robot facial expression control method based on internal projection technology, used to implement the robot facial expression control system based on internal projection technology as described in any one of claims 1-7, characterized in that: The robot facial expression control method comprises the following steps: S1, using the camera pre-installed on the robot's eyes to collect the interlocutor's facial expression images, and extracting and classifying the facial expression images through a facial expression classification model; S2, combining the facial expression features extracted and classified with natural language processing technology to generate dialogue content matching the facial expression features; S3, converting the collected facial expression features into digital animations and presenting the interlocutor's facial expressions; S4, using internal projection technology to project digital animation onto the robot's facial screen, synchronously displaying facial expressions that match those of the interlocutor; S5. Through the connection with the cloud, the robot receives and updates the latest facial expression features and conversation content in real time, and regulates the robot's facial expressions and voice output.