Human-computer interaction method based on expression recognition, interactive intelligent toy and robot

Abstract

The invention discloses a human-computer interaction method based on expression recognition, an interactive intelligent toy and a robot. The man-machine interaction method comprises the following steps: preprocessing a user face image collected in real time, and determining a face effective area; preset key point information in the face effective area is extracted; determining facial features based on preset key point information, and analyzing the facial features based on the constructed classification model to obtain user expression information; determining an expression intensity level based on a preset expression library and the user expression information; if the expression intensity grade is smaller than a preset threshold value, triggering a first strategy to carry out multi-target grid deformation adjustment on the to-be-deformed target object; if the expression intensity grade is not smaller than the preset threshold value, a second strategy is triggered to conduct three-dimensional deformation adjustment on the target object to be deformed, and the appearance of the target object obtained after deformation adjustment is matched with the expression of the user. According to the method, the intelligent degree of the target body can be effectively improved, and intelligent, rich and personalized interaction experience is provided for a user.

Description

Technical Field

[0001] The present disclosure relates to the technical field of intelligent toys, and in particular to a human-computer interaction method based on expression recognition, an interactive intelligent toy, and a robot. Background Art

[0002] With the rapid development of artificial intelligence and computer vision technology, the smart toy market has ushered in new opportunities and challenges. Traditional toy interaction methods mainly rely on simple buttons, sensors or preset voice commands, lacking real intelligence and personalized experience. Although some high-end toys have begun to introduce expression recognition technology, there are still the following problems. First, expression recognition technology is not yet mature, with limited recognition accuracy. It can only recognize a few basic expressions, and it is difficult to accurately capture the user's subtle emotional changes, which limits the intelligence and personalized response capabilities of toys; second, the interactive method of toys is too single, lacking a diverse feedback mechanism, and usually can only respond to users through preset sounds or simple actions, unable to provide a rich and varied interactive experience.

[0003] At the same time, smart toys also have obvious deficiencies in morphological changes and computing resources. The deformation methods of deformable toys currently on the market are preset and limited, which cannot meet the needs of users for flexibility and creativity. In addition, high-end smart functions have high hardware requirements, and complex algorithms make it difficult for ordinary toys to carry them. In addition, the cumbersome calculation process leads to slow response and poor real-time performance of the toys, which seriously affects the user experience. Summary of the invention

[0004] In view of this, the embodiments of the present disclosure provide a human-computer interaction method, an interactive intelligent toy, and a robot based on expression recognition, which can solve the problems existing in the prior art such as low recognition accuracy, single interaction, limited morphological changes, and slow response.

[0005] In a first aspect, an embodiment of the present disclosure provides a human-computer interaction method based on expression recognition, the method specifically comprising the following steps:

[0006] Preprocess the user's facial image collected in real time to determine the effective facial area;

[0007] Extracting preset key point information in the effective facial area;

[0008] Based on the preset key point information, determine facial features, wherein the facial features include facial geometric features and facial texture features;

[0009] Analyzing the facial features based on the constructed classification model to obtain user expression information;

[0010] Determine the expression intensity level based on the preset expression library and the user expression information;

[0011] If the expression intensity level is less than a preset threshold, trigger the first strategy to perform multi-target grid deformation adjustment on the target object to be deformed;

[0012] If the expression intensity level is not less than the preset threshold, trigger the second strategy to perform three-dimensional deformation adjustment on the target object to be deformed, and the appearance of the deformed target object matches the user's expression.

[0013] Optionally, the preprocessing of the real-time acquired user facial image to determine the facial effective area includes:

[0014] Real-time capture of the user's facial image based on an image acquisition device with a preset resolution;

[0015] Perform enhancement processing on the user's facial image using a preset enhancement strategy;

[0016] Process the enhanced user facial image based on the Viola-Jones algorithm to determine the facial effective area.

[0017] Optionally, the preset key point information includes eye specific point information, nose specific point information, mouth specific point information, and eyebrow specific point information;

[0018] The facial geometric features include eye opening degree, mouth corner upward degree, and eyebrow bending degree;

[0019] The facial texture features are presented in the form of a texture feature histogram.

[0020] Optionally, the analysis of the facial features based on the constructed classification model to obtain user expression information includes:

[0021] Construct an initial learning model;

[0022] Train the initial learning model using a preset expression data set, and use the trained initial learning model as the classification model;

[0023] The preset expression data set includes a data set of facial images of different expressions;

[0024] Analyze the facial features based on the classification model to obtain the expression type and expression amplitude;

[0025] The expression type and the expression amplitude constitute the user expression information.

[0026] Optionally, determine the expression intensity level based on the preset expression library and the user expression information;

[0027] Add a regression head to the classification model to obtain an initial regression model;

[0028] Generate synthetic expression images based on a generative adversarial network;

[0029] Obtain a preset expression library based on the preset expression dataset and the synthetic expression images;

[0030] Train the initial regression model based on the preset expression library, and denote the trained initial regression model as the target regression model;

[0031] Analyze the user expression information based on the target regression model to obtain an expression intensity value, and obtain a corresponding expression intensity level based on a preset expression intensity information library.

[0032] Optionally, if the expression intensity level is less than a preset threshold, trigger a first strategy to perform multi-target grid deformation adjustment on the target object to be deformed, including:

[0033] Process the user expression information based on a constructed number of grid models to obtain geometric difference information, physical difference information, and aesthetic difference information respectively;

[0034] Obtain a target object deformation strategy based on the geometric difference information, the physical difference information, and the aesthetic difference information;

[0035] Perform multi-target grid deformation adjustment on the target object to be deformed based on the target object deformation strategy.

[0036] Optionally, the obtaining the target object deformation strategy based on the geometric difference information, the physical difference information, and the aesthetic difference information includes:

[0037] Extract geometric gradients, physical gradients, and aesthetic gradients based on the geometric difference information, the physical difference information, and the aesthetic difference information respectively;

[0038] Fuse the geometric gradient, the physical gradient, and the aesthetic gradient based on a self-adaptive weight strategy to obtain a fusion score function;

[0039] The fusion score function includes: F = w 1 * physical gradient + w 2 * geometric gradient + w 3 * aesthetic gradient; where, w 1 is the expression adaptive weight, w 2 is the geometric adaptive weight, w 3 is the material adaptive weight;

[0040] Calculate the grid vertex gradient of the target object to be deformed based on the fusion score function; the target object deformation strategy includes the grid vertex gradient.

[0041] Optionally, if the expression intensity level is not less than a preset threshold, trigger a second strategy to perform three-dimensional deformation adjustment on the target object to be deformed, including:

[0042] Create a target Octree structure and inject noise into the target Octree structure;

[0043] Extract the key expression features in the user expression information and convert the key expression features into a first vector;

[0044] Convert the preset style of the target object to be deformed into a target vector, where the target vector includes style features and style requirements;

[0045] Fuse the first vector with the target vector to obtain a conditional vector;

[0046] Construct an initial diffusion model; the initial diffusion model includes a neural network based on a 3D UNet structure;

[0047] Inject the conditional vector into different levels of the neural network to obtain a first model;

[0048] Train the first model based on a historical 3D expression dataset, and use the trained first model as the target diffusion model;

[0049] Gradually denoise the target Octree structure after injecting noise through the target diffusion model to obtain target three-dimensional data;

[0050] Perform three-dimensional deformation adjustment on the target object to be deformed based on the target three-dimensional data.

[0051] In a second aspect, an embodiment of the present disclosure further provides an interactive intelligent toy, including a toy body and a total control center disposed inside the toy body;

[0052] The total control center controls the intelligent interaction between the toy body and the user by using the above-mentioned human-computer interaction method based on expression recognition;

[0053] The toy body is the target object to be deformed.

[0054] In a third aspect, an embodiment of the present disclosure further provides a robot, including a robot body and a total control center disposed inside the robot body;

[0055] The total control center controls the intelligent interaction between the robot body and the user by using the above-mentioned human-computer interaction method based on expression recognition;

[0056] The robot body is the target object to be deformed.

[0057] Fourthly, embodiments of the present disclosure further provide a human-computer interaction system based on facial expression recognition, including:

[0058] A preprocessing module, configured to preprocess a user's facial image collected in real time and determine a facial valid region;

[0059] An extraction module, configured to extract preset key point information in the facial valid region;

[0060] A feature determination module, configured to determine facial features based on the preset key point information, where the facial features include facial geometric features and facial texture features;

[0061] An analysis module, configured to analyze the facial features based on a constructed classification model to obtain user expression information;

[0062] An intensity level acquisition module, configured to determine an expression intensity level based on a preset expression library and the user expression information;

[0063] A judgment module, configured to, if the expression intensity level is less than a preset threshold, trigger a first strategy to perform multi-target grid deformation adjustment on a target object to be deformed;

[0064] If the expression intensity level is not less than the preset threshold, trigger a second strategy to perform three-dimensional deformation adjustment on the target object to be deformed, and the appearance of the deformed target object matches the user's expression.

[0065] Fifthly, embodiments of the present disclosure further provide a computer device, adopting the following technical solution:

[0066] The computer device includes:

[0067] At least one processor; and,

[0068] A memory communicatively connected to the at least one processor; wherein,

[0069] The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the above-mentioned human-computer interaction method based on facial expression recognition.

[0070] Sixthly, embodiments of the present disclosure further provide a computer-readable storage medium, which stores computer instructions for causing a computer to execute the above-mentioned human-computer interaction method based on facial expression recognition.

[0071] Seventhly, embodiments of the present disclosure further provide a computer program product, including computer programs / instructions, and when the computer programs / instructions are executed by a processor, the steps of the above-mentioned method are implemented.

[0072] The human-computer interaction method based on facial expression recognition disclosed in this application makes human-computer interaction more natural and intuitive by capturing and analyzing users' facial expressions in real time; by preprocessing the collected facial images of users to determine the effective facial regions, background noise can be effectively removed, and the recognition accuracy of facial features can be improved; by extracting the preset key point information of the face, the solution can accurately capture the geometric features and texture features of the face; based on the constructed classification model, the extracted facial features are analyzed to obtain the specific facial expression information of users, which can effectively improve the recognition accuracy, thus overcoming the problem of limited facial expression recognition ability in traditional technologies; based on the facial expression information of users, the facial expression intensity level is determined. If the facial expression intensity level is low, the first strategy is adopted for multi-object grid deformation adjustment; if the facial expression intensity level is high, the second strategy is adopted for three-dimensional deformation adjustment. This method enables the target object (such as a toy, etc.) to provide different interaction feedbacks according to the emotional intensity of the user, thus providing a more diverse and rich interaction experience, that is, adjusting the deformation of the target object according to the intensity level of the facial expression, making the target object match the emotional state of the user, and enhancing the user's immersion and interaction experience; different adjustment strategies can be adopted according to different facial expression intensities, improving the flexibility and adaptability of the system. This facial expression recognition-based method can significantly improve the intelligence level of the human-computer interaction system, enabling the emotions and facial expressions of users to be more accurately reflected in the virtual environment, and providing a more intelligent, rich, and personalized interaction experience for users.

[0073] The above description is only an overview of the technical solution of the present disclosure. In order to understand the technical means of the present disclosure more clearly, it can be implemented according to the content of the specification. And in order to make the above and other purposes, features, and advantages of the present disclosure more obvious and understandable, the following specific preferred embodiments are given, and in conjunction with the accompanying drawings, the details are described as follows. Brief Description of the Drawings

[0074] In order to more clearly illustrate the technical solutions of the embodiments of the present disclosure, the accompanying drawings required to be used in the embodiments will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present disclosure. For those of ordinary skill in the art, without creative efforts, other accompanying drawings can be obtained based on these drawings.

[0075] Figure 1 It is a schematic flowchart of the human-computer interaction method based on facial expression recognition provided by the embodiments of the present disclosure.

[0076] Figure 2 It is a schematic flowchart of the method for determining the effective facial region provided by the embodiments of the present disclosure.

[0077] Figure 3Schematic flowchart of the method for obtaining user expression information provided by an embodiment of the present disclosure.

[0078] Figure 4 Schematic flowchart of the method for determining the expression intensity level provided by an embodiment of the present disclosure.

[0079] Figure 5 Schematic flowchart of the method for performing deformation adjustment based on the first strategy provided by an embodiment of the present disclosure.

[0080] Figure 6 Schematic flowchart of the method for obtaining the target body deformation strategy provided by an embodiment of the present disclosure.

[0081] Figure 7 Schematic flowchart of the method for performing deformation adjustment based on the second strategy provided by an embodiment of the present disclosure.

[0082] Figure 8 Schematic diagram of the structure of a computer device provided by an embodiment of the present disclosure. Detailed implementation manners

[0083] The embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings.

[0084] It should be clear that the following uses specific specific examples to illustrate the implementation manners of the present disclosure, and those skilled in the art can easily understand other advantages and effects of the present disclosure from the content disclosed in this specification. Obviously, the described embodiments are only a part of the embodiments of the present disclosure, rather than all the embodiments. The present disclosure can also be implemented or applied through other different specific implementation manners, and various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of the present disclosure. It should be noted that, without conflict, the following embodiments and the features in the embodiments can be combined with each other. All other embodiments obtained by those of ordinary skill in the art based on the embodiments in the present disclosure without making creative efforts belong to the scope of protection of the present disclosure.

[0085] It should be noted that the following describes various aspects of the embodiments within the scope of the appended claims. It should be obvious that the aspects described herein can be embodied in a wide variety of forms, and any specific structure and / or function described herein is illustrative only. Based on the present disclosure, those skilled in the art should understand that one aspect described herein can be implemented independently of any other aspect, and two or more of these aspects can be combined in various ways. For example, any number of aspects described herein can be used to implement the device and / or practice the method. In addition, this device can be implemented and this method can be practiced using other structures and / or functions in addition to one or more of the aspects described herein.

[0086] It should also be noted that the diagrams provided in the following embodiments only illustrate the basic concept of the present disclosure in a schematic manner. The diagrams only show the components related to the present disclosure, rather than being drawn according to the number, shape, and size of the components in actual implementation. The types, quantities, and proportions of the components in actual implementation can be arbitrarily changed, and the component layout type may also be more complex.

[0087] In addition, in the following description, specific details are provided to facilitate a thorough understanding of the examples. However, those skilled in the art will understand that the described aspects can be practiced without these specific details.

[0088] Referring to Figure 1 , a human-computer interaction method based on facial expression recognition is disclosed in the first aspect of the present application. The method specifically includes the following steps:

[0089] S100, preprocess the user's facial image collected in real time to determine the effective facial region.

[0090] In this step, the preprocessing and determination of the effective region can improve the accuracy of subsequent steps and reduce the interference of irrelevant regions on facial expression recognition.

[0091] S200, extract the preset key point information in the effective facial region.

[0092] Among them, the preset key point information includes eye specific point information, nose specific point information, mouth specific point information, and eyebrow specific point information.

[0093] Specifically, using a facial key point detection algorithm (such as the 68-key point detector of the Dlib library or other deep learning models), extract specific key points on the face (such as the corners of the eyes, the upper edges of the eyebrows, the corners of the mouth, etc.). Extracting accurate key point information is the basis for subsequent facial feature analysis and helps to accurately capture the subtle changes in facial expressions.

[0094] S300, based on the preset key point information, determine the facial features, and the facial features include facial geometric features and facial texture features.

[0095] Among them, the facial geometric features refer to the features obtained by calculating the relative positions, distances, angles, etc. between the key points. For example, the degree of eye opening can be obtained by calculating the position changes of the eye key points, the angle of the upward curvature of the mouth corners can also be calculated through the corresponding key points, and the changes in the eyebrows can be obtained by calculating the position changes of the eyebrow key points.

[0096] Specifically, the geometric features of the face (such as face proportions, distances between various parts) and texture features (such as skin texture, distribution of wrinkles) are calculated using the extracted key-point information. These features can be extracted and analyzed through a deep learning model. The facial geometric features and texture features can describe the user's expression state in detail, providing accurate information for subsequent classification and analysis.

[0097] Among them, texture is a certain change pattern of pixel gray levels or colors in a local area of an image. It reflects the attributes of the image itself. Texture feature extraction is the process of obtaining this texture information in the image.

[0098] Preferably, the facial texture features in this step are extracted using LBP (Local Binary Pattern) and preferably presented in the form of a texture feature histogram in this step.

[0099] Specifically, 1) Select the neighborhood and the central pixel; for each pixel point in the image, the LBP algorithm considers the neighborhood pixels around this pixel. A common choice is a 3x3 neighborhood, that is, each pixel point has 8 neighborhood pixels.

[0100] 2) Calculate the binary pattern; specifically, for the central pixel point and its 8 neighborhood pixels, where the central pixel is the current pixel to be processed, and the neighborhood pixels are the 8 pixels around the central pixel (the other 8 points in the 3x3 matrix); then compare the gray value of the neighborhood pixel with the gray value of the central pixel; if the gray value of the neighborhood pixel is greater than or equal to the gray value of the central pixel, the binary bit corresponding to this neighborhood pixel is 1; if the gray value of the neighborhood pixel is less than the gray value of the central pixel, the binary bit corresponding to this neighborhood pixel is 0; record the binary values of the neighborhood pixels from left to right and from top to bottom in sequence to form an 8-bit binary number.

[0101] 3) Convert the binary pattern to a decimal number; specifically, convert the obtained 8-bit binary number to a decimal number, and this decimal number is the LBP feature value of the current central pixel.

[0102] For example, assume that the binary sequence obtained by comparing the neighborhood pixels of a certain pixel is 10010011, and the converted decimal number is 147. Therefore, the LBP feature value of the central pixel is 147.

[0103] Repeat the above process for each pixel point in the image to obtain the LBP feature value of each pixel, and finally form an LBP feature map. To obtain the overall texture feature of the image, the LBP feature values can be statistically analyzed in a histogram, counting the frequency of each LBP feature value appearing in the image to obtain a feature histogram. This graph records the frequency of each LBP value appearing, and this histogram can be used to describe the texture feature of the entire image.

[0104] In order to better perform subsequent processing (such as classification), the feature histogram is usually normalized to ensure that the range of feature values ​​is consistent.

[0105] S400, analyzing facial features based on the constructed classification model to obtain user expression information.

[0106] The classification model can automatically recognize and interpret user expressions (such as happiness, sadness, anger, etc.), provide real-time feedback, and increase the naturalness and responsiveness of human-computer interaction.

[0107] S500, determining an expression intensity level based on a preset expression library and user expression information.

[0108] Specifically, the recognized user expression information can be compared with a preset expression library, and the expression intensity level (such as slight, obvious, strong, etc.) can be determined according to the matching degree and intensity of the expression.

[0109] S600, if the expression intensity level is less than a preset threshold, triggering a first strategy to perform multi-target mesh deformation adjustment on the target to be deformed;

[0110] If the expression intensity level is not less than a preset threshold, the second strategy is triggered to perform three-dimensional deformation adjustment on the target object to be deformed, and the appearance of the target object after the deformation adjustment matches the user's expression.

[0111] If the expression intensity level is lower than the threshold, a multi-target mesh deformation technique is applied to fine-tune the appearance of the target object, such as making local adjustments to the target object. If the expression intensity level is higher than the threshold, a more complex three-dimensional deformation technique is used, such as global shape adjustment based on the user's expression, so that the appearance of the target object is highly matched to the user's expression. This step enables the target object (such as a virtual character or image or smart toy) to be appropriately deformed under different expression intensities, thereby enhancing the authenticity and immersion of human-computer interaction.

[0112] This embodiment provides two deformation strategies: multi-target grid deformation adjustment and three-dimensional deformation adjustment. Multi-target grid deformation is suitable for situations with low expression intensity and can achieve a certain degree of morphological change; three-dimensional deformation adjustment is suitable for situations with high expression intensity and can perform more complex morphological changes according to the user's expression. This method improves the flexibility and creativity of the toy, enhances the toy's interactivity, expressiveness, and fun, and meets the user's demand for morphological changes.

[0113] By dividing the deformation strategies into two levels and selecting the appropriate deformation method according to the expression intensity, the scheme effectively optimizes the use of computing resources and reduces the impact of complex computing processes on real-time performance, thereby improving the toy's response speed and user experience.

[0114] The human-computer interaction method based on facial expression recognition disclosed in this application makes human-computer interaction more natural and intuitive by capturing and analyzing users' facial expressions in real time; by preprocessing the collected facial images of users to determine the effective facial area, background noise can be effectively removed, and the recognition accuracy of facial features can be improved; by extracting the preset key point information of the face, the solution can accurately capture the geometric features and texture features of the face; based on the constructed classification model, the extracted facial features are analyzed to obtain the specific facial expression information of the user, which can effectively improve the recognition accuracy, thus overcoming the problem of limited facial expression recognition ability in traditional technologies; based on the facial expression information of the user, the facial expression intensity level is determined. If the facial expression intensity level is low, the first strategy is adopted for multi-target grid deformation adjustment; if the facial expression intensity level is high, the second strategy is adopted for three-dimensional deformation adjustment. This method enables the target object (such as a toy) to provide different interactive feedback according to the emotional intensity of the user, thereby providing a more diverse interactive experience, that is, adjusting the deformation of the target object according to the intensity level of the facial expression to match the emotional state of the user and enhance the user's immersion and interactive experience; different adjustment strategies can be adopted according to different facial expression intensities, improving the flexibility and adaptability of the system. This facial expression recognition-based method can significantly improve the intelligence level of the human-computer interaction system, enabling the emotions and facial expressions of users to be more accurately reflected in the virtual environment and providing a more intelligent, rich, and personalized interactive experience for users.

[0115] Refer to Figure 2 , the method for determining the effective facial area specifically includes the following steps:

[0116] S110, use an image acquisition device with a preset resolution to capture the facial image of the user in real time.

[0117] Among them, the preset resolution refers to a high resolution not lower than 1080P, which ensures that clear facial images can be captured and provides more accurate data for subsequent processing.

[0118] In this embodiment, the image acquisition device is preferably a camera capable of capturing the facial image of the user in real time set in a toy or a custom-developed image acquisition software; through real-time capture, it is applicable to application scenarios that require immediate feedback, such as the intelligent interaction between a toy and a user (such as a child).

[0119] S120, perform enhancement processing on the facial image of the user using a preset enhancement strategy.

[0120] Among them, the preset enhancement strategy includes one or more of the histogram equalization strategy and the Gaussian filtering strategy.

[0121] If the histogram equalization strategy is adopted, that is, the histogram equalization algorithm (such as `cv2.equalizeHist` in OpenCV) is applied to enhance the contrast of the image, the visibility of facial features can be effectively improved. Specifically, the captured facial image is converted into a grayscale image, and histogram equalization is applied to enhance the contrast of the image.

[0122] If the Gaussian filtering strategy is adopted, the image can be smoothed, noise can be reduced, and the image quality can be improved. Specifically, an appropriate Gaussian kernel size (such as 5x5 or 7x7) is selected to filter the image to smooth the facial image and reduce the interference of noise on the recognition of the facial area.

[0123] In addition, a combination of the two strategies can be adopted, that is, the enhancement effect can be adjusted according to specific requirements.

[0124] Through the enhancement process, the contrast and clarity of the image can be effectively improved, making the facial features more obvious and reducing the impact of environmental factors (such as insufficient light) on the image quality.

[0125] S130, Process the enhanced user facial image based on the Viola-Jones algorithm to determine the effective facial area.

[0126] In this step, the Viola-Jones face detection algorithm (such as `cv2.CascadeClassifier` in OpenCV) is used to process the enhanced image to detect and locate the facial area; specifically, a pre-trained face detection classifier (such as the Haar cascade classifier) is loaded, and the classifier is applied to the enhanced image to detect the facial area and determine the bounding box of the effective facial area.

[0127] Furthermore, the detection parameters (such as the scaling ratio, the minimum facial size for detection) can be adjusted according to needs to improve the detection accuracy.

[0128] By adopting the Viola-Jones algorithm, the facial area can be quickly and accurately recognized, and the facial area can be precisely determined, providing a reliable basis for subsequent feature extraction and analysis.

[0129] The method for determining the effective facial region disclosed in this embodiment enhances the clarity and contrast of the image through enhancement processing to ensure the accuracy of subsequent face detection and feature extraction; uses the Viola-Jones algorithm for efficient and accurate face region detection, which can reduce the computational overhead; at the same time, it can provide accurate face region information, making subsequent expression analysis and human-computer interaction more natural and precise; by selecting appropriate enhancement strategies, it can adapt to different environmental conditions (such as light changes) and device characteristics, improving the robustness of the system. The overall solution realizes efficient and accurate face region detection through the precise cooperation of image acquisition, enhancement, and face region recognition, providing reliable data support for subsequent face feature analysis and expression recognition.

[0130] Referring to Figure 3 , the method for obtaining user expression information specifically includes the following steps:

[0131] S410, construct an initial learning model.

[0132] Specifically, 1) Determine that the basic network of the initial learning model is the lightweight EfficientNet B0, which can achieve a balance between computational efficiency and classification accuracy; 2) Initialize the model: Use the pre-trained weights of EfficientNet B0 as the initialization weights, which can accelerate the training process and improve the model performance; 3) Configure the model architecture: Adjust the last few layers of the network as needed, such as adding fully connected layers or custom output layers, to adapt to the expression classification task.

[0133] S420, use a preset expression dataset to train the initial learning model, and use the trained initial learning model as a classification model;

[0134] The preset expression dataset includes a dataset of facial images with different expressions.

[0135] Specifically, the preset expression dataset includes multiple expression categories such as smiling, laughing out loud, sad, surprised, etc.; use this preset expression dataset to train the initial learning model, and specifically, appropriate loss functions (such as cross-entropy loss) and optimizers (such as Adam) can be set, and hyperparameters such as the learning rate can be adjusted; the model obtained by training through this method can adapt to the expression characteristics in actual applications and can comprehensively understand and classify various expressions.

[0136] S430, analyze the facial features based on the classification model to obtain the expression type and expression amplitude;

[0137] The expression type and expression amplitude constitute the user expression information.

[0138] Specifically, a trained classification model is used to extract features and analyze the input facial images. The model predicts the expression type (such as happy, angry, etc.) based on the features in the input image. The amplitude of the expression can be estimated through the output probability value of the model or additional regression analysis. In this step, the model can provide accurate expression classification information to help understand the user's emotional state. The estimation of the expression amplitude can provide more details about the emotional intensity, contributing to a deeper emotional analysis.

[0139] Referring to Figure 4 , the method for determining the expression intensity level specifically includes the following steps:

[0140] S510, Add a regression head on the basis of the classification model to obtain an initial regression model.

[0141] The initial regression model obtained in this step is different from the classification model. The goal of the regression model is to predict a continuous value rather than a discrete category; for expression intensity estimation, the model needs to output an expression intensity value, which is usually a continuous value (for example, in the range from 0 to 1, indicating the intensity of the expression from none to strong).

[0142] Among them, the regression head is usually a fully connected layer or a group of fully connected layers, which is responsible for converting the features extracted by the classification model into an expression intensity value.

[0143] By adding a regression head on the basis of the classification model, the obtained model can not only classify expressions but also output a continuous value as the expression intensity.

[0144] S520, Generate synthetic expression images based on a generative adversarial network.

[0145] A generative adversarial network (GAN) is used to generate synthetic data similar to real data. By training a GAN, synthetic expression images with different intensities can be generated, thereby expanding the dataset. Specifically, a generative adversarial network (GAN) is used to generate synthetic images of various expressions. For example, the generator can be used to generate images with different expression intensities, and the discriminator is used to ensure that the generated images are visually realistic.

[0146] The expression images generated using a GAN can contain different intensities, which helps the model to see more diverse intensity samples during training, enhancing its recognition and estimation ability for expressions of different intensities, that is, increasing the model's recognition ability for different expression intensities and improving the model's generalization ability.

[0147] S530, Obtain a preset expression library based on the preset expression dataset and the synthetic expression images.

[0148] By combining real images and synthetic images, the preset expression library will contain a wider range of expression intensities, making the training data more comprehensive and helping to improve the accuracy and robustness of the model.

[0149] S540, train the initial regression model based on the preset expression library, and denote the trained initial regression model as the target regression model.

[0150] Use the images and label data in the preset expression library to train the initial regression model. During the training process, the model will learn the features of different expression intensities and optimize its regression output; through training, the model can more accurately predict the expression intensity value, and the target regression model can better adapt to real and synthetic data, thereby improving the ability to evaluate expression intensity.

[0151] S550, analyze the user's expression information based on the target regression model to obtain the expression intensity value, and obtain the corresponding expression intensity level based on the preset expression intensity information library.

[0152] Specifically, input the user's expression image into the target regression model, and the model outputs the expression intensity value. Then, compare this intensity value with the information in the preset expression intensity information library to determine the expression intensity level; this step can convert the continuous expression intensity value predicted by the model into discrete levels, which helps further application and interpretation, such as in sentiment analysis or human-computer interaction.

[0153] In this embodiment, by using the generative adversarial network for synthetic images, the diversity of the training data is enhanced, thereby improving the performance of the model in different scenarios; the synthetic expression images can make up for the shortage of samples in the actual dataset and improve the richness and coverage of the dataset; the model can not only perform expression classification but also evaluate the expression intensity, providing more comprehensive expression analysis results; by using the preset expression library for training, it can adapt to different application scenarios and requirements, improving the flexibility and application breadth of the model; through accurate expression intensity evaluation, it can provide more realistic sentiment analysis and help improve the interaction experience between users and the system.

[0154] Refer to Figure 5 , the method for deformation adjustment based on the first strategy specifically includes the following steps:

[0155] A100, process the user's expression information based on the constructed several grid models to obtain geometric difference information, physical difference information, and aesthetic difference information respectively.

[0156] Specifically, it includes the following steps:

[0157] A110. Geometric preservation models, expression recognition models, and material usage models are respectively constructed. That is, the several mesh models constructed in this step refer to geometric preservation models, expression recognition models, and material usage models.

[0158] A120. Geometric difference information can be extracted based on the geometric preservation model. Specifically, geometric difference information refers to calculating the distance and shape difference between the geometric preservation model and the user's expression. For example, when the user's expression is surprise, the Euclidean distance between the current cube shape and the target "surprise" shape can be measured. The extraction of geometric difference information can help evaluate the gap between the current shape and the target shape.

[0159] Physical difference information can be obtained based on the expression recognition model. Specifically, by simulating skin deformation and lighting effects, physical changes are detected to evaluate the energy change during the deformation process, ensuring that the deformation conforms to the physical properties of the material (e.g., avoiding unrealistic deformations).

[0160] Aesthetic difference information can be obtained based on the material usage model. Specifically, by comparing whether the expression conforms to aesthetic standards, such as natural facial expression curves, balance, etc. For example, calculating whether the deformed shape is symmetric and balanced to ensure good visual effects.

[0161] A200. Based on the geometric difference information, physical difference information, and aesthetic difference information, a target object deformation strategy is obtained.

[0162] For example, if the geometric difference information shows that facial features need to move in a specific area, the physical difference information indicates that the skin needs to be stretched or compressed, and the aesthetic difference information points out that the expression needs to be made more natural and conform to aesthetic standards, then a corresponding target object deformation strategy can be formulated, which may include adjusting the positions of mesh vertices, modifying surface curvatures, or applying physical simulation techniques.

[0163] The formulated target object deformation strategy will combine information from multiple aspects, thereby achieving more precise and comprehensive deformation adjustment to ensure that the adjusted result meets the requirements geometrically, physically, and aesthetically.

[0164] A300. Based on the target object deformation strategy, multi-objective mesh deformation adjustment is performed on the target object to be deformed.

[0165] Specifically, apply the deformation strategy formulated in step A200 to the target object to be deformed (such as the 3D facial model of a toy), and use multi-objective mesh deformation technology (for example, mesh adjustment based on optimization algorithms or physical simulation methods) to adjust and deform the mesh vertices of the target object, so as to achieve various target effects, such as correcting geometric deviations of facial expressions, optimizing physical performance, and improving aesthetic effects. By applying the target object deformation strategy, the target object can be effectively adjusted to better conform to the expected expression characteristics and aesthetic standards. Multi-objective mesh deformation adjustment can comprehensively consider the requirements of multiple aspects to ensure the high quality and high accuracy of the final effect.

[0166] In this embodiment, by processing the user's expression information in detail to obtain geometric, physical, and aesthetic difference information, a more accurate deformation strategy can be formulated to improve the deformation effect of the target object; comprehensively considering geometric differences, physical differences, and aesthetic differences to ensure that the deformation not only meets the expectations technically but also achieves good results visually and sensorially; through refined mesh deformation adjustment, the user's expression can be better simulated and adjusted, making the final result more in line with the user's expectations and enhancing the user experience; this method is applicable to a variety of target objects and expression deformation scenarios, with strong adaptability and flexibility, and can be widely applied in fields such as virtual reality, animation production, and game development; by considering aesthetic difference information, it can be ensured that the deformed target object is more natural visually and more in line with aesthetic standards, enhancing the overall visual effect.

[0167] Refer to Figure 6 , the method for obtaining the target object deformation strategy specifically includes the following steps:

[0168] A210, based on geometric difference information, physical difference information, and aesthetic difference information, extract geometric gradient, physical gradient, and aesthetic gradient respectively.

[0169] Specifically, for the geometric gradient: by comparing the geometric differences between the mesh model of the target object and the reference model, extract the changes in the positions and shapes of the mesh vertices. For example, use geometric analysis methods (such as mesh distance calculation) to obtain the displacement gradient of the mesh vertices.

[0170] For the physical gradient: by simulating physical deformations (such as skin stretching or contraction), obtain the physical gradient, which can be calculated through physical simulation tools, such as using finite element analysis (FEA) to estimate the impact of physical deformations.

[0171] For the aesthetic gradient: by evaluating the surface quality and visual effects of the target object, extract the aesthetic gradient, which can be calculated by comparing with aesthetic standards (such as the naturalness and symmetry of facial expressions, etc.) to calculate the adjustment amount related to aesthetics.

[0172] The geometric gradient can accurately capture the shape changes of the mesh, providing detailed geometric information for subsequent adjustments; the physical gradient takes into account the impact of physical deformation on the mesh, making the adjustments more in line with the actual physical situation; the aesthetic gradient ensures that the deformed result meets the aesthetic standards visually, improving the visual effect and user experience of the target object.

[0173] In this embodiment, the extracted geometric gradient is used as the shape fidelity score, the obtained physical gradient is used as the expression similarity score, and the obtained aesthetic gradient is used as the material naturalness score.

[0174] A220, based on the self-adaptive weight strategy, fuse the geometric gradient, physical gradient, and aesthetic gradient to obtain a fused score function;

[0175] The fused score function includes: F = w 1 * physical gradient + w 2 * geometric gradient + w 3 * aesthetic gradient; where w 1 is the expression adaptive weight, w 2 is the geometric adaptive weight, w 3 is the material adaptive weight.

[0176] By adjusting the weights, it can be optimized for different application requirements and target objects, providing a more flexible and adaptable deformation strategy; by fusing different gradient information, it can comprehensively consider the requirements in terms of geometry, physics, and aesthetics, ensuring that the deformation strategy is more comprehensive and accurate.

[0177] A230, calculate the mesh vertex gradient of the target object to be deformed based on the fused score function; the target body deformation strategy includes the mesh vertex gradient.

[0178] Use the fused score function FF to calculate the gradient of the mesh vertices of the target object. According to the weights and gradient values of each item in FF, calculate the adjustment direction and amplitude of each mesh vertex; apply the calculated mesh vertex gradient to the target object to be deformed, and complete the deformation of the target object by adjusting the positions of the mesh vertices.

[0179] Through the calculation of the mesh vertex gradient, precise mesh adjustment can be achieved, ensuring that the deformation effect of the target object meets the expectations; according to the comprehensive information provided by the fused score function, the adjusted target object can better meet the requirements in terms of geometry, physics, and aesthetics.

[0180] The method disclosed in this embodiment provides a comprehensive deformation strategy by combining geometric, physical, and aesthetic gradients, ensuring that all aspects during the deformation process are considered, and improving the accuracy and comprehensiveness of the deformation results. The self-adaptive weight strategy enables the method to adjust the deformation strategy according to different application scenarios and requirements, thereby enhancing the flexibility and adaptability of the solution. By calculating the gradient of the mesh vertices, efficient and precise deformation of the target object can be achieved to meet specific deformation requirements. By comprehensively considering the user's expression, physical characteristics, and aesthetic standards, the final target object can better meet the user's expectations and enhance the overall visual effect and user experience.

[0181] Furthermore, this application also includes: iteratively updating the positions of the mesh vertices based on the gradient descent method or a preset optimization algorithm; the gradient descent method includes: V new = V old - η·BSD Score , where η is the learning rate, which determines the step size of each update, V old is the old gradient, and V new is the new gradient. In each iteration, the application grid maintains constraints to ensure that the topological structure remains unchanged.

[0182] Furthermore, this application also includes: a user feedback mechanism that allows fine-tuning of the weights according to the toy interaction effect, and an evaluation mechanism; the evaluation mechanism can evaluate the results from multiple dimensions, including expression expressiveness, geometric fidelity, deformation smoothness, etc., and then based on the evaluation results, feedback and adjust the hyperparameters during the deformation adjustment process.

[0183] The following takes an interactive teddy bear that can change according to a child's expression as an example for illustration.

[0184] First, train an expression recognition model that can recognize basic emotions (such as happiness, sadness, surprise, etc.); train a geometry preservation model to ensure that the overall shape characteristics of the teddy bear are maintained during deformation; train a material adaptation model considering the deformation characteristics of the teddy bear's fur.

[0185] Second, extract the gradient of the expression recognition model as the expression similarity score; extract the gradient of the geometry preservation model as the shape fidelity score; extract the gradient of the material adaptation model as the material naturalness score; weighted function: F = w 1 * physical gradient + w 2 * geometric gradient + w 3 * aesthetic gradient.

[0186] Then, initialize the 3D mesh model of the teddy bear, including facial key points; when it is detected that the child shows a smile, the system starts the deformation process; calculate the gradient of F to guide the movement of the mesh vertices; the optimization algorithm may first raise the corners of the mouth and then adjust the eye area while keeping the bear's ears and overall contour unchanged.

[0187] Initial settings: w 1 (Expression) = 0.5, w 2 (Geometry) = 0.3, w 3 (Material) = 0.2.

[0188] If the expression after deformation is not obvious enough, the value of w can be automatically increased; if the shape of the bear becomes unnatural, the value of w can be increased. 1 2 value.

[0189] In another embodiment, we take a simple cube toy as an example to demonstrate how to achieve multi-objective deformation. Suppose we want to deform the cube into a spiky shape according to the user's "surprised" expression.

[0190] For the geometric gradient: The Euclidean distance between the current shape and the target "surprised" shape can be calculated to evaluate the change in surface curvature and encourage the formation of sharp features.

[0191] For the physical gradient: Calculate the deformation energy to ensure that the deformation process conforms to the physical properties of the material, evaluate the structural stability, and prevent collapse caused by excessive deformation.

[0192] For the aesthetic gradient: Evaluate the symmetry and balance of the shape; calculate the surface smoothness to avoid unnecessary wrinkles.

[0193] Then obtain the fusion score function. At the same time, the weights can be dynamically adjusted. For example, increase w at the beginning of deformation, 1 increase w later, 2 and w 3 .

[0194] Furthermore, iterative optimization can also be performed. Specifically, use the gradient descent method to update the mesh vertex positions. After each iteration, recalculate the scoring function and the gradient until the preset number of iterations is reached or the deformation effect meets the requirements.

[0195] Through this process, the cube toy will gradually form spiky protrusions while maintaining the stability and aesthetics of the overall structure. For example, a large spike may form on the top, and multiple small spikes may be generated on the sides, presenting a "surprised" visual effect as a whole.

[0196] Refer to Figure 7 , the method for deformation adjustment based on the second strategy specifically includes the following steps:

[0197] B100, create a target Octree structure and inject noise into the target Octree structure.

[0198] Specifically, create an Octree structure for the three-dimensional mesh data of the target object. Octree is a spatial partitioning method that recursively divides three-dimensional space into eight sub-regions and is a hierarchical structure suitable for processing large-scale three-dimensional data. Add random noise to each node in the Octree structure, which can be achieved by introducing small-scale random fluctuations in the nodes, thus providing the necessary conditions for the subsequent denoising process.

[0199] In this step, the Octree structure can efficiently manage and process complex three-dimensional data, making subsequent processing more efficient. The injection of noise makes the subsequent denoising process more challenging and improves the model's robustness to actual noise and irregular data.

[0200] B200, extract the key expression features from the user's expression information and convert the key expression features into a first vector.

[0201] Extract the key expression features from the user's expression image, such as facial key points, specific changes in expressions, etc.; encode these key features into a vector. For example, use a deep learning model to map facial expressions to feature vectors. Extracting key expression features can capture the core information of expressions and provide a specific basis for subsequent adjustments; through vectorization, feature information can be more conveniently used for calculation and fusion.

[0202] B300, convert the preset style of the target object to be deformed into a target vector, and the target vector includes style features and style requirements.

[0203] Specifically, define the style features and style requirements of the target object, such as style details, colors, and textures, etc., and convert these style features and requirements into a target vector. It may be possible to convert style information into a vector representation through a style encoder. In this step, clarifying the style features and requirements of the target object helps to maintain style consistency during the deformation process, and vectorizing the style information facilitates fusion and adjustment in the model.

[0204] B400, fuse the first vector and the target vector to obtain a conditional vector.

[0205] Specifically, fuse the extracted first vector (expression feature vector) and the target vector (style feature vector). Weighted summation or other fusion techniques can be used, and the resulting fused vector is used as the conditional vector, which can describe the requirements of the target object under specific expressions and styles.

[0206] In this step, by fusing expression features and style features, the resulting conditional vector can comprehensively consider expression and style information and provide comprehensive conditions for subsequent deformation.

[0207] B500, construct an initial diffusion model; the initial diffusion model includes a neural network based on the 3D UNet structure.

[0208] Specifically, construct a neural network model based on 3D UNet. UNet is a network architecture for image segmentation, and 3D UNet is extended to three-dimensional data processing; initialize the model to the starting state of the diffusion model.

[0209] In this step, 3D UNet can effectively process three-dimensional data, especially suitable for image segmentation and generation tasks; the initial diffusion model can provide an effective basis for subsequent denoising and deformation adjustment.

[0210] B600, inject conditional vectors at different levels of the neural network to obtain a first model.

[0211] Specifically, inject the conditional vectors at different levels of the 3D UNet network, which can be achieved through connection layers or additional feature fusion modules, to obtain a 3D UNet model combined with the conditional vectors. Injecting conditional vectors at different levels enables the model to consider the influence of expressions and styles at multiple levels, improving the accuracy of the generated results.

[0212] B700, train the first model based on the historical 3D expression dataset, and use the trained first model as the target diffusion model.

[0213] Specifically, use the historical 3D expression dataset to train the first model. The dataset includes three-dimensional data of various expressions and styles. The trained model can better adapt to the actual data and perform effective three-dimensional deformation adjustment.

[0214] In this step, training with real data can improve the adaptability and accuracy of the model to the actual application scenario. The trained model can better perform diffusion and denoising tasks.

[0215] B800, gradually denoise the target Octree structure after injecting noise through the target diffusion model to obtain the target three-dimensional data.

[0216] Specifically, input the Octree structure injected with noise into the trained target diffusion model. The model gradually restores the details of the target three-dimensional data through multiple iterations of denoising, and the finally obtained three-dimensional data contains the detailed structure of the target object after denoising.

[0217] In this step, gradual denoising can effectively restore the details of the target object and improve the quality of the three-dimensional data; the three-dimensional data processed by the diffusion model can better reflect the actual deformation result of the target object.

[0218] B900 performs three-dimensional deformation adjustment on the target object to be deformed based on the target three-dimensional data.

[0219] Specifically, the target three-dimensional data is used to adjust the target object to be deformed. The target three-dimensional data is applied to the mesh model of the target object for detailed three-dimensional deformation processing. For detail restoration, a deformation algorithm can be used to match and adjust the target three-dimensional data with the target object to ensure that the target object meets the expected deformation effect.

[0220] In this step, the deformation adjustment based on high-quality target three-dimensional data can ensure that the final target object meets the set requirements. The high precision of the three-dimensional data can restore the details of the target object, making the deformation effect more realistic and natural.

[0221] The solution disclosed in this embodiment can achieve high-precision three-dimensional deformation adjustment by combining expression features, style requirements, and diffusion models, ensuring that the deformation effect of the target object meets expectations. The diffusion model is used to gradually denoise the data injected with noise, improving the data quality and ensuring the accuracy of the deformation result. The solution integrates multiple aspects of expression, style, and three-dimensional data processing, making the deformation adjustment not only meet technical requirements but also visual and style requirements. This method is applicable to various three-dimensional deformation tasks, such as virtual character animation, character modeling in games, and model optimization in industrial design, with strong adaptability and flexibility. Through precise deformation adjustment, it can provide more natural and user-expected results, enhancing the overall user experience and satisfaction.

[0222] Among them, the method for creating the target Octree structure specifically includes the following steps:

[0223] B110, create a root node.

[0224] This step is the starting point of the Octree structure. First, a root node needs to be created. This node represents the outermost bounding volume of the entire three-dimensional space, and the root node contains the entire generated space (i.e., the spatial range of the entire model or scene).

[0225] Through this step, the boundary of the entire three-dimensional space can be determined, and this root node provides the basis for subsequent space division.

[0226] B120, space division.

[0227] After the root node is created, it needs to be recursively divided. Specifically, the Octree can divide the space into 8 sub-cubes (child nodes), and each sub-cube represents a part of the original space.

[0228] The partitioning process will be carried out recursively until the volume of each child node reaches the preset minimum voxel size, which means that the space will be subdivided into very small cubes called voxels.

[0229] Through recursive partitioning, the entire space can be subdivided into smaller voxels, which allows data to be processed at a higher resolution. For example, complex geometric shapes and details can be represented more precisely.

[0230] B130, Attribute Encoding.

[0231] After the partitioning is completed, each leaf node (i.e., the smallest voxel node) needs to encode its occupancy status and surface features; the occupancy status indicates whether the voxel is occupied by an object, and the surface features may include attributes such as the color and texture of the voxel.

[0232] In this step, it is determined whether each voxel is within the space of the target object, which helps to determine the boundaries and internal structure of the object; recording the surface features of the voxel helps to provide the necessary information during further processing (such as texture mapping, lighting calculation, etc.).

[0233] The method for creating the target Octree structure disclosed in this embodiment establishes the boundaries of the entire three-dimensional space as the starting point of the Octree; through recursive partitioning, the entire space is subdivided into smaller voxels to provide a high-resolution representation for handling details; recording the occupancy status and surface features of each voxel enables subsequent processing to utilize this information for further analysis and processing.

[0234] Next, taking the generation of a complex "surprised face" 3D model as an example, it demonstrates how to implement it using an Octree-based diffusion model.

[0235] 1) Octree initialization; create a root node representing the entire generation space (e.g., a cube of 1m 3 ) and recursively divide the space until the preset minimum voxel size is reached (e.g., 1cm 3 ); assign initial states (occupied / blank) and surface feature values to the leaf nodes.

[0236] 2) Conditional information encoding.

[0237] Convert the "surprised" expression features into vectors (e.g., eyebrow raising degree, mouth opening degree, etc.); encode the target style (such as cartoon style or realistic style); fuse the expression features and style information into a unified conditional vector.

[0238] 3) Diffusion model design and training.

[0239] Build a neural network based on 3D UNet and adapt the Octree structure; inject the conditional vectors generated in step B2 into multiple layers of the network; and train the model using a large-scale 3D expression dataset.

[0240] 4) Step-by-step denoising. Starting from a high noise level, predict and remove a certain amount of noise at each step; at each step, consider the influence of the conditional vector; dynamically adjust the Octree structure to subdivide areas where more details are needed; repeat the denoising process until a clear 3D shape is obtained.

[0241] Furthermore, post-processing strategies can also be included. Specifically, the mesh is reconstructed from the Octree using an improved Marching Cubes algorithm; mesh smoothing and optimization algorithms are applied to improve surface quality; and detailed features such as textures and microstructures are added.

[0242] Through this process, an expressive "surprised face" 3D model can be generated. For example, the eyebrow area may be prominent and present an upward shape; an exaggerated circular depression may be formed around the eyes; a large oval opening may be generated in the mouth area, and the entire face may present an expanded effect to emphasize the emotion of surprise.

[0243] This Octree-based method can not only efficiently generate complex 3D shapes, but also achieve precise control based on the input expression features and style information, thereby creating rich and diverse expression models.

[0244] Furthermore, the human-computer interaction method based on expression recognition disclosed in the present application also includes system integration and interaction control, specifically including asynchronous processing mechanism, caching mechanism, interaction mechanism, performance optimization mechanism, and user experience optimization mechanism.

[0245] Among them, the asynchronous processing mechanism includes: implementing asynchronous message queues to ensure the real-time performance of the system; asynchronous processing means that the message transmission between modules does not need to wait for the other party to respond immediately. Implementing asynchronous message queues allows the system to process tasks in the background without blocking the main thread or waiting for immediate feedback from other modules. In this way, even if a module is temporarily unable to process messages, other modules can continue to work. Asynchronous processing can significantly improve the real-time performance and response speed of the system. It allows the system to process multiple tasks in parallel, thereby making more efficient use of computing resources; in addition, by reducing waiting time, users can get feedback faster, thereby improving the overall user experience.

[0246] The caching mechanism includes: configuring intelligent caching to reduce duplicate calculations, which is used to store and quickly retrieve previously calculated data or intermediate results. In this way, when the same data request appears again, the system can directly obtain the result from the cache without having to recalculate. By reducing duplicate calculations, the computing efficiency and response speed of the system can be significantly improved, which helps save computing resources, especially when dealing with complex or time-consuming tasks; in addition, by reducing unnecessary calculations, the energy consumption and heat generation of the system can also be reduced.

[0247] The interaction mechanism includes an interaction state mechanism, a conversion logic mechanism, a feedback mapping mechanism, and a multimodal output mechanism; the interaction state mechanism is used to implement different interaction states of the toy, the conversion logic mechanism is used to implement different core logic controls of the toy, the feedback mapping mechanism is used to map the generated 3D shape to the physical feedback of the toy, and the multimodal output mechanism is used to coordinate various output methods such as sound and light.

[0248] The performance optimization mechanism includes a task scheduling strategy, a resource dynamic allocation strategy, and a predictive computing strategy. Among them, the task scheduling strategy is used to implement a priority-based task scheduling strategy, the resource dynamic allocation strategy is used to dynamically adjust the resource occupancy of each module according to the current load, and the predictive computing strategy is used to use historical data to predict possible interactions and perform calculations in advance.

[0249] The user experience optimization mechanism includes response time control, smooth transition control, and personalized adaptation control. Among them, the response time control is used to ensure that the response time of key interactions does not exceed 100 ms, the smooth transition control is used to control the smooth transition between different expressions and shapes, and the personalized adaptation control is used to record and analyze user habits and dynamically adjust interaction parameters.

[0250] Example 1: Smile recognition and toy deformation

[0251] When the user shows a smile to the toy, first, the degree of the smile is accurately captured. Suppose a medium-strength smile is recognized.

[0252] Next, it is processed according to the preset "smile" deformation target. For example, for a spherical toy, the specific deformation adjustments include: the upper half of the sphere will gradually expand to form a shape similar to the upward curvature of the smiling mouth; at the same time, depressions similar to eyes will be generated on the surface of the sphere to further enhance the visual effect of the "smiling face".

[0253] The entire deformation process will consider physical constraints to ensure that the deformed shape still maintains balance and structural stability.

[0254] Example 2: Surprised expression and 3D shape generation

[0255] When the user shows a surprised expression, that is, when the intensity level of the expression is not less than the preset threshold, three-dimensional deformation adjustment will be performed.

[0256] Specifically, the user's surprised expression is captured and its intensity is evaluated; then the features of the surprised expression are encoded as conditional information and input into the diffusion model; the diffusion model, based on the Octree structure, generates a series of 3D shapes related to "surprise", such as suddenly emerging spikes, rapidly expanding bubbles, etc.; the generated 3D shapes are reconstructed into a mesh through the Marching Cubes algorithm and fused with the deformation of the toy body; finally, the toy will present a form where the center suddenly expands and multiple small protrusions appear around it, vividly expressing the concept of "surprise".

[0257] The "target object" in this embodiment refers to those objects that can be deformed or adjusted according to the user's expression information. Specifically, the target object can be a toy, a robot, a smart device, a virtual character or an animated character, a user interface element, a digital product, an environmental element in virtual reality, etc.

[0258] For virtual characters or animated characters: In virtual reality (VR), augmented reality (AR), or games, the user's facial expressions can be used to drive the expression changes of virtual characters, making the performance of the characters more vivid and natural.

[0259] For user interface elements: For example, when the user interacts with a certain software or application, some elements in the interface can be adjusted according to the user's expression. For example, expression recognition can be used to adjust the animation effect of buttons or the way of information display in the interface.

[0260] For digital products or content: including digital avatars, emoji, etc., which can adjust their appearance according to the user's real-time expression, enhancing the user experience.

[0261] For environmental elements in virtual reality: For example, objects or landscapes in the virtual environment can be dynamically adjusted according to the user's mood or expression, making the virtual environment more in line with the user's emotional state.

[0262] The deformation adjustment of these target objects can select different strategies for adaptation according to the different expression intensities to achieve a more natural and personalized interaction experience.

[0263] The second aspect of this application discloses an interactive intelligent toy, including a toy body and a total control center arranged inside the toy body;

[0264] The total control center controls the intelligent interaction between the toy body and the user by using the human-computer interaction method based on expression recognition described in any one of the above;

[0265] The toy body is the target object to be deformed.

[0266] A third aspect of the present application discloses a robot, including a robot body and a general control center disposed inside the robot body;

[0267] The general control center controls the intelligent interaction between the robot body and the user by using the human-computer interaction method based on facial expression recognition described in any one of the above;

[0268] The robot body is an object to be deformed.

[0269] For robots or intelligent devices: In some high-end robots or intelligent devices, the user's facial expressions can be used to control the facial expressions or actions of the robot, making the interaction with the robot more natural and intuitive.

[0270] A fourth aspect of the present application discloses a human-computer interaction system based on facial expression recognition, including:

[0271] A preprocessing module, configured to preprocess the real-time collected user facial image to determine the facial effective area;

[0272] An extraction module, configured to extract the preset key point information in the facial effective area;

[0273] A feature determination module, configured to determine facial features based on the preset key point information, where the facial features include facial geometric features and facial texture features;

[0274] An analysis module, configured to analyze the facial features based on the constructed classification model to obtain user facial expression information;

[0275] An intensity level acquisition module, configured to determine the facial expression intensity level based on a preset facial expression library and the user facial expression information;

[0276] A judgment module, configured to, if the facial expression intensity level is less than a preset threshold, trigger a first strategy to perform multi-target grid deformation adjustment on the object to be deformed;

[0277] If the facial expression intensity level is not less than the preset threshold, trigger a second strategy to perform three-dimensional deformation adjustment on the object to be deformed, and the appearance of the deformed object matches the user's facial expression.

[0278] It should be noted that the solutions disclosed in the human-computer interaction method based on facial expression recognition disclosed in the first aspect of the present application are all applicable to the human-computer interaction system based on facial expression recognition disclosed in the fourth aspect of the present application, so they will not be elaborated here.

[0279] A computer device according to an embodiment of the present disclosure includes a memory and a processor. The memory is used to store non-transitory computer-readable instructions. Specifically, the memory may include one or more computer program products, and the computer program products may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include, for example, random access memory (RAM) and / or cache memory, etc. The non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc.

[0280] The processor may be a central processing unit (CPU) or other forms of processing units with data processing capabilities and / or instruction execution capabilities, and may control other components in the computer device to perform desired functions. In an embodiment of the present disclosure, the processor is used to run the computer-readable instructions stored in the memory, so that the computer device executes all or part of the steps of the human-computer interaction method based on facial expression recognition in the foregoing embodiments of the present disclosure.

[0281] Those skilled in the art should understand that, in order to solve the technical problem of how to obtain a good user experience effect, known structures such as communication buses and interfaces may also be included in this embodiment, and these known structures should also be included in the protection scope of the present disclosure.

[0282] As Figure 8 FIG. is a schematic structural diagram of a computer device provided by an embodiment of the present disclosure. It shows a schematic structural diagram suitable for implementing the computer device in the embodiments of the present disclosure. Figure 8 The shown computer device is only an example and should not impose any limitations on the functions and usage scope of the embodiments of the present disclosure.

[0283] As Figure 8 As shown, the computer device may include a processor (such as a central processor, a graphics processor, etc.), which may perform various appropriate actions and processes according to a program stored in read-only memory (ROM) or a program loaded from a storage device into random access memory (RAM). In the RAM, various programs and data required for the operation of the computer device are also stored. The processor, ROM, and RAM are connected to each other through a bus. The input / output (I / O) interface is also connected to the bus.

[0284] Generally, the following devices may be connected to the I / O interface: input devices including, for example, sensors or visual information acquisition devices; output devices including, for example, display screens; storage devices including, for example, magnetic tapes, hard disks, etc.; and communication devices. The communication device may allow the computer device to communicate with other devices (such as edge computing devices) wirelessly or wiredly to exchange data. Although Figure 8A computer device having various devices is shown, but it should be understood that it is not required to implement or have all the shown devices. Instead, more or fewer devices may be implemented or had.

[0285] In particular, according to an embodiment of the present disclosure, the processes described above with reference to the flowcharts may be implemented as computer software programs. For example, an embodiment of the present disclosure includes a computer program product that includes a computer program carried on a non-transitory computer-readable medium, and the computer program includes program codes for executing the methods shown in the flowcharts. In such an embodiment, the computer program may be downloaded and installed from a network through a communication device, or installed from a storage device, or installed from a ROM. When the computer program is executed by a processor, all or part of the steps of the human-computer interaction method based on facial expression recognition according to the embodiments of the present disclosure are executed.

[0286] For a detailed description of this embodiment, reference may be made to the corresponding descriptions in the foregoing embodiments, and details will not be repeated here.

[0287] A computer-readable storage medium according to an embodiment of the present disclosure stores non-temporary computer-readable instructions. When the non-temporary computer-readable instructions are run by a processor, all or part of the steps of the human-computer interaction method based on facial expression recognition according to the foregoing embodiments of the present disclosure are executed.

[0288] The above-mentioned computer-readable storage medium includes but is not limited to: optical storage media (such as CD-ROMs and DVDs), magneto-optical storage media (such as MOs), magnetic storage media (such as magnetic tapes or external hard drives), media having built-in rewritable non-volatile memories (such as memory cards), and media having built-in ROMs (such as ROM cartridges).

[0289] For a detailed description of this embodiment, reference may be made to the corresponding descriptions in the foregoing embodiments, and details will not be repeated here.

[0290] The basic principles of the present disclosure have been described above in conjunction with specific embodiments. However, it should be noted that the advantages, benefits, effects, etc. mentioned in the present disclosure are only examples and not limitations, and it cannot be considered that these advantages, benefits, effects, etc. are essential for each embodiment of the present disclosure. In addition, the above-mentioned specific details are only for the purpose of illustration and easy understanding, rather than limitations, and the above details do not limit the present disclosure to necessarily adopt the above specific details for implementation.

[0291] In this disclosure, relational terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. The block diagrams of devices, apparatuses, equipment, and systems involved in this disclosure are only illustrative examples and do not intend to require or imply that they must be connected, arranged, and configured in the manner shown in the block diagrams. As those skilled in the art will recognize, these devices, apparatuses, equipment, and systems can be connected, arranged, and configured in any manner. Words such as "including", "comprising", "having", etc. are open-ended terms, meaning "including but not limited to", and can be used interchangeably with each other. The words "or" and "and" used herein refer to the word "and / or", and can be used interchangeably with it, unless the context clearly indicates otherwise. The word "such as" used herein refers to the phrase "such as but not limited to", and can be used interchangeably with it.

[0292] In addition, as used herein, "or" in the listing of items starting with "at least one" indicates a disjunctive listing, so that for example, the listing of "at least one of A, B, or C" means A or B or C, or AB or AC or BC, or ABC (i.e., A and B and C). Furthermore, the term "exemplary" does not mean that the examples described are preferred or better than other examples.

[0293] It should also be noted that in the systems and methods of this disclosure, each component or each step can be decomposed and / or recombined. These decompositions and / or recombinations should be regarded as equivalent solutions of this disclosure.

[0294] Various changes, substitutions, and alterations to the technologies described herein can be made without departing from the teachings defined by the appended claims. In addition, the scope of the claims of this disclosure is not limited to the specific aspects of the processes, machines, manufactures, compositions of events, means, methods, and acts described above. Current or later-developed processes, machines, manufactures, compositions of events, means, methods, or acts that perform substantially the same function or achieve substantially the same result as the corresponding aspects described herein can be utilized. Thus, the appended claims include such processes, machines, manufactures, compositions of events, means, methods, or acts within their scope.

[0295] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use this disclosure. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein can be applied to other aspects without departing from the scope of this disclosure. Therefore, this disclosure is not intended to be limited to the aspects shown herein, but rather to the broadest scope consistent with the principles and novel features disclosed herein.

[0296] The foregoing description has been presented for purposes of illustration and description. Furthermore, this description is not intended to limit embodiments of the present disclosure to the form disclosed herein. Although several example aspects and embodiments have been discussed above, those of skill in the art will recognize some of their variations, modifications, alterations, additions, and subcombinations.

Claims

1. A human-computer interaction method based on expression recognition, characterized in that: include: Preprocess the user's facial image collected in real time to determine the effective facial area; Extracting preset key point information in the effective facial area; Based on the preset key point information, determine facial features, wherein the facial features include facial geometric features and facial texture features; Analyzing the facial features based on the constructed classification model to obtain user expression information; Determine the expression intensity level based on the preset expression library and the user expression information; If the expression intensity level is less than a preset threshold, triggering a first strategy to perform multi-target mesh deformation adjustment on the target to be deformed; If the expression intensity level is not less than a preset threshold, a second strategy is triggered to perform three-dimensional deformation adjustment on the target object to be deformed.

2. The human-computer interaction method based on facial expression recognition according to claim 1, characterized in that: The preprocessing of the user's facial image collected in real time to determine the effective facial area includes: An image acquisition device based on a preset resolution captures a user's facial image in real time; Performing enhancement processing on the user's facial image using a preset enhancement strategy; The enhanced user facial image is processed based on the Viola-Jones algorithm to determine the effective facial area.

3. The human-computer interaction method based on facial expression recognition according to claim 2, characterized in that: The preset key point information includes eye specific point information, nose specific point information, mouth specific point information, and eyebrow specific point information; The facial geometric features include the degree of eye opening, the degree of upward movement of the mouth corners and the curvature of the eyebrows.

4. The human-computer interaction method based on facial expression recognition according to claim 1, characterized in that: The facial features are analyzed based on the constructed classification model to obtain user expression information, including: Build an initial learning model; Using a preset expression data set to train the initial learning model, and using the trained initial learning model as the classification model; The preset expression data set includes a data set of facial images with different expressions; Analyze the facial features based on the classification model to obtain expression type and expression amplitude; The expression type and the expression amplitude constitute the user expression information.

5. The human-computer interaction method based on facial expression recognition according to claim 4, characterized in that: Determining the expression intensity level based on the preset expression library and the user expression information; Adding a regression head on the basis of the classification model to obtain an initial regression model; Generate synthetic expression images based on generative adversarial networks; Obtaining a preset expression library based on the preset expression data set and the synthetic expression image; Training the initial regression model based on the preset expression library, and recording the trained initial regression model as a target regression model; The user expression information is analyzed based on the target regression model to obtain an expression intensity value, and a corresponding expression intensity level is obtained based on a preset expression intensity information library.

6. The human-computer interaction method based on facial expression recognition according to claim 1, characterized in that: If the expression intensity level is less than a preset threshold, triggering a first strategy to perform multi-target mesh deformation adjustment on the target to be deformed includes: Processing the user expression information based on the constructed several grid models to obtain geometric difference information, physical difference information, and aesthetic difference information respectively; Based on the geometric difference information, the physical difference information, and the aesthetic difference information, obtaining a target body deformation strategy; Based on the target body deformation strategy, multi-target mesh deformation adjustment is performed on the target object to be deformed.

7. The human-computer interaction method based on facial expression recognition according to claim 6, characterized in that: The step of obtaining a target body deformation strategy based on the geometric difference information, the physical difference information, and the aesthetic difference information includes: Based on the geometric difference information, the physical difference information, and the aesthetic difference information, respectively extracting a geometric gradient, a physical gradient, and an aesthetic gradient; The geometric gradient, the physical gradient, and the aesthetic gradient are fused based on an adaptive weight strategy to obtain a fusion score function; The fusion score function includes: F=w1*physical gradient+w2*geometric gradient+w3*aesthetic gradient; wherein w1 is the expression adaptive weight, w2 is the geometric adaptive weight, and w3 is the material adaptive weight; The mesh vertex gradient of the target object to be deformed is calculated based on the fusion score function; and the target body deformation strategy includes the mesh vertex gradient.

8. The human-computer interaction method based on facial expression recognition according to claim 1, characterized in that: If the expression intensity level is not less than a preset threshold, triggering a second strategy to perform three-dimensional deformation adjustment on the target object to be deformed includes: Creating a target Octree structure, and injecting noise into the target Octree structure; Extracting key expression features from the user expression information, and converting the key expression features into a first vector; Converting a preset style of the target object to be deformed into a target vector, wherein the target vector includes style features and style requirements; Merging the first vector with the target vector to obtain a conditional vector; Constructing an initial diffusion model; the initial diffusion model includes a neural network based on a 3D UNet structure; Injecting the conditional vectors at different levels of the neural network to obtain a first model; Training the first model based on the historical 3D expression data set, and using the trained first model as a target diffusion model; The target diffusion model is used to gradually remove noise in the target Octree structure after the noise is injected, so as to obtain target three-dimensional data; The target object to be deformed is subjected to three-dimensional deformation adjustment based on the target three-dimensional data.

9. An interactive intelligent toy, characterized in that: It includes a toy body and a master control center arranged in the toy body; The master control center uses the human-computer interaction method based on expression recognition described in any one of claims 1 to 8 to control the intelligent interaction between the toy body and the user; The toy body is a target object to be deformed.

10. A robot, characterized in that: It includes a robot body and a master control center arranged in the robot body; The master control center uses the human-computer interaction method based on expression recognition described in any one of claims 1 to 8 to control the intelligent interaction between the robot body and the user; The robot body is a target object to be deformed.

Images