Human-computer interaction method based on facial expression recognition, interactive intelligent toys, robots

Abstract

The present application discloses a human-computer interaction method, interactive intelligent toy, and robot based on expression recognition. The human-computer interaction method includes: pre-processing a real-time user facial image to determine the effective facial area; extracting preset key point information from the effective facial area; determining facial features based on the preset key point information, analyzing the facial features based on a constructed classification model, and obtaining user expression information; determining the expression intensity level based on a preset expression library and user expression information; if the expression intensity level is less than a preset threshold, triggering a first strategy to perform multi-target mesh deformation adjustment on the target to be deformed; if the expression intensity level is not less than the preset threshold, triggering a second strategy to perform three-dimensional deformation adjustment on the target to be deformed, so that the appearance of the deformed target matches the user's expression. This method can effectively improve the intelligence level of the target object and provide users with an intelligent, rich, and personalized interactive experience.

Description

Technical Field

[0001] The present disclosure relates to the technical field of intelligent toys, and in particular to a human-computer interaction method based on expression recognition, an interactive intelligent toy, and a robot. Background Art

[0002] With the rapid development of artificial intelligence and computer vision technologies, the smart toy market is facing new opportunities and challenges. Traditional toy interactions rely primarily on simple buttons, sensors, or preset voice commands, lacking true intelligence and personalized experiences. Although some high-end toys have begun to incorporate expression recognition technology, the following problems still exist: First, expression recognition technology is still immature and has limited accuracy. It can only recognize a few basic expressions and cannot accurately capture the user's subtle emotional changes, which limits the toy's intelligent and personalized response capabilities. Second, the toy's interactive methods are too simple and lack diverse feedback mechanisms. Usually, it can only respond to users through preset sounds or simple movements, failing to provide a rich and varied interactive experience.

[0003] At the same time, smart toys also face significant deficiencies in terms of form transformation and computing resources. Transformable toys currently on the market offer limited and predefined transformation methods, failing to meet users' demands for flexibility and creativity. Furthermore, high-end intelligent functions require high hardware requirements, while complex algorithms make them difficult for ordinary toys to implement. Furthermore, the cumbersome computational process results in slow toy response and poor real-time performance, all of which severely impact the user experience. Summary of the Invention

[0004] In view of this, the embodiments of the present disclosure provide a human-computer interaction method, an interactive intelligent toy, and a robot based on expression recognition, which can solve the problems existing in the prior art such as low recognition accuracy, single interaction, limited morphological changes, and slow response.

[0005] In a first aspect, an embodiment of the present disclosure provides a human-computer interaction method based on expression recognition, the method specifically comprising the following steps:

[0006] Pre-process the user's facial image collected in real time to determine the effective facial area;

[0007] Extracting preset key point information in the effective facial area;

[0008] Determining facial features based on the preset key point information, wherein the facial features include facial geometric features and facial texture features;

[0009] Analyzing the facial features based on the constructed classification model to obtain user expression information;

[0010] Determining an expression intensity level based on a preset expression library and the user's expression information;

[0011] If the expression intensity level is less than a preset threshold, triggering a first strategy to perform multi-target grid deformation adjustment on the target to be deformed;

[0012] If the expression intensity level is not less than a preset threshold, a second strategy is triggered to perform three-dimensional deformation adjustment on the target object to be deformed, and the appearance of the target object after the deformation adjustment matches the user's expression.

[0013] Optionally, preprocessing the user's facial image collected in real time to determine the effective facial area includes:

[0014] An image acquisition device based on a preset resolution captures a user's facial image in real time;

[0015] Performing enhancement processing on the user's facial image using a preset enhancement strategy;

[0016] The enhanced user facial image is processed based on the Viola-Jones algorithm to determine a valid facial area.

[0017] Optionally, the preset key point information includes specific point information of eyes, specific point information of nose, specific point information of mouth, and specific point information of eyebrows;

[0018] The facial geometric features include the degree of eye opening, the degree of upturned mouth corners and the curvature of eyebrows;

[0019] The facial texture features are presented in the form of a texture feature histogram.

[0020] Optionally, analyzing the facial features based on the constructed classification model to obtain user expression information includes:

[0021] Build an initial learning model;

[0022] Using a preset expression data set to train the initial learning model, and using the trained initial learning model as the classification model;

[0023] The preset expression data set includes a data set of facial images of different expressions;

[0024] Analyzing the facial features based on the classification model to obtain expression type and expression amplitude;

[0025] The expression type and the expression amplitude constitute the user expression information.

[0026] Optionally, the expression intensity level is determined based on a preset expression library and the user expression information;

[0027] Adding a regression head to the classification model to obtain an initial regression model;

[0028] Generate synthetic expression images based on generative adversarial networks;

[0029] Obtaining a preset expression library based on the preset expression data set and the synthesized expression image;

[0030] Training the initial regression model based on the preset expression library, and recording the trained initial regression model as a target regression model;

[0031] The user expression information is analyzed based on the target regression model to obtain an expression intensity value, and a corresponding expression intensity level is obtained based on a preset expression intensity information library.

[0032] Optionally, if the expression intensity level is less than a preset threshold, triggering a first strategy to perform multi-target mesh deformation adjustment on the target object to be deformed includes:

[0033] Processing the user expression information based on the constructed grid models to obtain geometric difference information, physical difference information, and aesthetic difference information respectively;

[0034] Obtaining a target body deformation strategy based on the geometric difference information, the physical difference information, and the aesthetic difference information;

[0035] The target object to be deformed is subjected to multi-target mesh deformation adjustment based on the target body deformation strategy.

[0036] Optionally, obtaining a target body deformation strategy based on the geometric difference information, the physical difference information, and the aesthetic difference information includes:

[0037] extracting a geometric gradient, a physical gradient, and an aesthetic gradient based on the geometric difference information, the physical difference information, and the aesthetic difference information, respectively;

[0038] fusing the geometric gradient, the physical gradient, and the aesthetic gradient based on an adaptive weight strategy to obtain a fusion score function;

[0039] The fusion score function includes: F = w1*physical gradient + w2*geometric gradient + w3*aesthetic gradient; wherein w1 is the expression adaptation weight, w2 is the geometric adaptation weight, and w3 is the material adaptation weight;

[0040] The mesh vertex gradients of the target object to be deformed are calculated based on the fusion score function; and the target body deformation strategy includes the mesh vertex gradients.

[0041] Optionally, if the expression intensity level is not less than a preset threshold, triggering a second strategy to perform three-dimensional deformation adjustment on the target object to be deformed includes:

[0042] Creating a target Octree structure and injecting noise into the target Octree structure;

[0043] Extracting key expression features from the user expression information, and converting the key expression features into a first vector;

[0044] Converting a preset style of the target object to be deformed into a target vector, wherein the target vector includes style features and style requirements;

[0045] Fusing the first vector with the target vector to obtain a conditional vector;

[0046] Constructing an initial diffusion model; the initial diffusion model includes a neural network based on a 3D UNet structure;

[0047] Injecting the conditional vectors at different levels of the neural network to obtain a first model;

[0048] Training the first model based on a historical 3D expression dataset, and using the trained first model as a target diffusion model;

[0049] gradually denoising the target Octree structure after noise injection using the target diffusion model to obtain target three-dimensional data;

[0050] The target object to be deformed is subjected to three-dimensional deformation adjustment based on the target three-dimensional data.

[0051] In a second aspect, the embodiments of the present disclosure further provide an interactive intelligent toy, comprising a toy body and a master control center disposed in the toy body;

[0052] The master control center uses the human-computer interaction method based on expression recognition to control the intelligent interaction between the toy body and the user;

[0053] The toy body is a target object to be transformed.

[0054] In a third aspect, an embodiment of the present disclosure further provides a robot, comprising a robot body and a master control center disposed in the robot body;

[0055] The master control center uses the human-computer interaction method based on facial expression recognition to control the intelligent interaction between the robot body and the user;

[0056] The robot body is a target object to be deformed.

[0057] In a fourth aspect, the present disclosure also provides a human-computer interaction system based on expression recognition, including:

[0058] A preprocessing module is used to preprocess the user's facial image collected in real time and determine the effective facial area;

[0059] An extraction module, configured to extract information of preset key points in the effective facial area;

[0060] A feature determination module, configured to determine facial features based on the preset key point information, wherein the facial features include facial geometric features and facial texture features;

[0061] An analysis module, configured to analyze the facial features based on the constructed classification model to obtain user expression information;

[0062] An intensity level acquisition module, configured to determine an expression intensity level based on a preset expression library and the user's expression information;

[0063] A judgment module, configured to trigger a first strategy to perform multi-target mesh deformation adjustment on the target object to be deformed if the expression intensity level is less than a preset threshold;

[0064] If the expression intensity level is not less than a preset threshold, a second strategy is triggered to perform three-dimensional deformation adjustment on the target object to be deformed, and the appearance of the target object after the deformation adjustment matches the user's expression.

[0065] In a fifth aspect, the embodiments of the present disclosure further provide a computer device, which adopts the following technical solution:

[0066] The computer device comprises:

[0067] at least one processor; and,

[0068] a memory communicatively connected to the at least one processor; wherein,

[0069] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute any of the above-mentioned human-computer interaction methods based on expression recognition.

[0070] In a sixth aspect, an embodiment of the present disclosure further provides a computer-readable storage medium, which stores computer instructions, and the computer instructions are used to enable a computer to execute any of the above-mentioned human-computer interaction methods based on expression recognition.

[0071] In a seventh aspect, an embodiment of the present disclosure further provides a computer program product, comprising a computer program / instruction, which implements the steps of any of the above methods when executed by a processor.

[0072] The human-computer interaction method based on expression recognition disclosed in this application makes human-computer interaction more natural and intuitive by capturing and analyzing user expressions in real time. By preprocessing the collected user facial images and determining the effective facial area, background noise can be effectively removed, improving the recognition accuracy of facial features. By extracting preset key point information of the face, the scheme can accurately capture the geometric and texture features of the face. Based on the constructed classification model, the extracted facial features are analyzed to obtain the user's specific expression information, which can effectively improve recognition accuracy and overcome the limited expression recognition ability of traditional technologies. The expression intensity level is determined based on the user's expression information. If the expression intensity level is low, a first strategy is adopted to perform multi-target mesh deformation adjustment; if the expression intensity level is high, a second strategy is adopted to perform three-dimensional deformation adjustment. This method enables target objects (such as toys) to provide different interactive feedback based on the user's emotional intensity, thereby providing a richer and more diverse interactive experience. That is, the deformation of the target object is adjusted according to the expression intensity level, so that the target object matches the user's emotional state, enhancing the user's immersion and interactive experience. Different adjustment strategies can be adopted according to different expression intensities, improving the flexibility and adaptability of the system. This expression recognition-based method can significantly improve the intelligence level of the human-computer interaction system, allowing users' emotions and expressions to be more accurately reflected in the virtual environment, and providing users with a more intelligent, rich, and personalized interactive experience.

[0073] The above description is only an overview of the technical solution of the present disclosure. In order to more clearly understand the technical means of the present disclosure, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the present disclosure more obvious and easy to understand, the following specifically cites preferred embodiments and describes them in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0074] In order to more clearly illustrate the technical solutions of the embodiments of the present disclosure, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0075] Figure 1 A flowchart of a human-computer interaction method based on expression recognition provided in an embodiment of the present disclosure.

[0076] Figure 2 A flowchart of a method for determining a valid facial area according to an embodiment of the present disclosure.

[0077] Figure 3 A flowchart of a method for obtaining user expression information provided in an embodiment of the present disclosure.

[0078] Figure 4 A flowchart of a method for determining an expression intensity level provided in an embodiment of the present disclosure.

[0079] Figure 5 A flowchart of a method for performing deformation adjustment based on a first strategy provided in an embodiment of the present disclosure.

[0080] Figure 6 A flowchart of a method for acquiring a target body deformation strategy provided in an embodiment of the present disclosure.

[0081] Figure 7 A flowchart of a method for performing deformation adjustment based on a second strategy provided in an embodiment of the present disclosure is provided.

[0082] Figure 8 A schematic diagram of the structure of a computer device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0083] The embodiments of the present disclosure are described in detail below with reference to the accompanying drawings.

[0084] It should be clear that the following embodiments of the present disclosure are described through specific concrete examples, and those skilled in the art can easily understand other advantages and effects of the present disclosure from the contents disclosed in this specification. Obviously, the described embodiments are only a part of the embodiments of the present disclosure, rather than all the embodiments. The present disclosure can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present disclosure. It should be noted that the following embodiments and features in the embodiments can be combined with each other in the absence of conflict. Based on the embodiments in the present disclosure, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present disclosure.

[0085] It should be noted that various aspects of the embodiments within the scope of the appended claims are described below. It should be apparent that the aspects described herein can be embodied in a wide variety of forms, and any specific structure and / or function described herein is merely illustrative. Based on this disclosure, it should be understood by those skilled in the art that an aspect described herein can be implemented independently of any other aspect, and two or more of these aspects can be combined in various ways. For example, any number of aspects described herein can be used to implement the device and / or practice the method. In addition, other structures and / or functionalities other than one or more of the aspects described herein can be used to implement this device and / or practice this method.

[0086] It should also be noted that the illustrations provided in the following embodiments are only schematic illustrations of the basic concept of the present disclosure. The illustrations only show components related to the present disclosure and are not drawn according to the number, shape and size of components in actual implementation. In actual implementation, the type, quantity and proportion of each component can be changed at will, and the component layout type may also be more complicated.

[0087] Additionally, in the following description, specific details are provided to provide a thorough understanding of the examples. However, one skilled in the art will appreciate that the aspects described can be practiced without these specific details.

[0088] Reference Figure 1 The first aspect of the present application discloses a human-computer interaction method based on expression recognition, which specifically includes the following steps:

[0089] S100, pre-processing the user's facial image collected in real time to determine the effective facial area.

[0090] In this step, preprocessing and determination of effective areas can improve the accuracy of subsequent steps and reduce the interference of irrelevant areas on expression recognition.

[0091] S200: extracting preset key point information in the effective facial area.

[0092] The preset key point information includes eye specific point information, nose specific point information, mouth specific point information, and eyebrow specific point information.

[0093] Specifically, facial key point detection algorithms (such as the 68 key point detectors in the Dlib library or other deep learning models) are used to extract specific key points on the face (such as the corners of the eyes, the upper edge of the eyebrows, the corners of the mouth, etc.). Extracting accurate key point information is the basis for subsequent facial feature analysis and helps to accurately capture subtle changes in facial expressions.

[0094] S300: Determine facial features based on preset key point information, where the facial features include facial geometric features and facial texture features.

[0095] Facial geometric features are those obtained by calculating the relative positions, distances, and angles between key points. For example, the degree of eye opening can be determined by calculating the positional changes of the eye key points, the angle of the mouth corners can be calculated using the corresponding key points, and the changes in the eyebrows can be determined by calculating the positional changes of the eyebrow key points.

[0096] Specifically, the extracted key point information is used to calculate the facial geometric features (such as facial proportions and the distance between various parts) and texture features (such as skin texture and the distribution of wrinkles). These features can be extracted and analyzed through deep learning models; facial geometric features and texture features can describe the user's expression state in detail, providing accurate information for subsequent classification and analysis.

[0097] Among them, texture is a certain change pattern of pixel grayscale or color in a local area of ​​an image, which reflects the properties of the image itself. Texture feature extraction is the process of obtaining this texture information in the image.

[0098] The facial texture features in this step are preferably extracted using LBP (Local Binary Pattern) and are preferably presented in the form of a texture feature histogram.

[0099] Specifically, 1) select neighborhood and center pixels; for each pixel in the image, the LBP algorithm considers the neighborhood pixels around the pixel. A common choice is a 3x3 neighborhood, that is, each pixel has 8 neighborhood pixels.

[0100] 2) Calculate the binary pattern; specifically, for the central pixel and its 8 neighboring pixels, where the central pixel is the current pixel to be processed and the neighboring pixels are the 8 pixels around the central pixel (the other 8 points in the 3x3 matrix); then compare the grayscale value of the neighboring pixels with the grayscale value of the central pixel; if the grayscale value of the neighboring pixel is greater than or equal to the grayscale value of the central pixel, the binary bit corresponding to the neighboring pixel is 1; if the grayscale value of the neighboring pixel is less than the grayscale value of the central pixel, the binary bit corresponding to the neighboring pixel is 0; record the binary values ​​of the neighboring pixels from left to right and from top to bottom to form an 8-bit binary number.

[0101] 3) Convert the binary pattern into a decimal number; specifically, convert the obtained 8-bit binary number into a decimal number, which is the LBP feature value of the current center pixel.

[0102] For example, suppose the binary sequence obtained by comparing the neighboring pixels of a certain pixel is 10010011, which is converted to a decimal number of 147. Therefore, the LBP eigenvalue of the central pixel is 147.

[0103] Repeat the above process for each pixel in the image to obtain the LBP eigenvalue of each pixel, and finally form an LBP feature map. In order to obtain the overall texture characteristics of the image, the LBP eigenvalues ​​can be histogram-based. The frequency of each LBP eigenvalue in the image is counted to obtain a feature histogram. This graph records the frequency of each LBP value. This histogram can be used to describe the texture characteristics of the entire image.

[0104] In order to better perform subsequent processing (such as classification), the feature histogram is usually normalized to ensure that the range of feature values ​​is consistent.

[0105] S400: Analyze facial features based on the constructed classification model to obtain user expression information.

[0106] The classification model can automatically recognize and interpret user expressions (such as happiness, sadness, anger, etc.), provide real-time feedback, and increase the naturalness and responsiveness of human-computer interaction.

[0107] S500: Determine an expression intensity level based on a preset expression library and user expression information.

[0108] Specifically, the recognized user expression information can be compared with a preset expression library, and the expression intensity level (such as mild, obvious, strong, etc.) can be determined based on the matching degree and intensity of the expression.

[0109] S600, if the expression intensity level is less than a preset threshold, triggering a first strategy to perform multi-target mesh deformation adjustment on the target to be deformed;

[0110] If the expression intensity level is not less than a preset threshold, a second strategy is triggered to perform three-dimensional deformation adjustment on the target object to be deformed, and the appearance of the target object after deformation adjustment matches the user's expression.

[0111] If the expression intensity level is below a threshold, a multi-target mesh deformation technique is applied to subtly adjust the object's appearance, such as making local adjustments to the object. If the expression intensity level is above the threshold, more complex three-dimensional deformation techniques are used, such as global shape adjustment based on the user's expression, to closely match the object's appearance to the user's expression. This step enables objects (such as virtual characters, images, or smart toys) to deform appropriately at different expression intensities, enhancing the realism and immersion of human-computer interaction.

[0112] This embodiment provides two deformation strategies: multi-target grid deformation adjustment and three-dimensional deformation adjustment. Multi-target grid deformation is suitable for situations with low expression intensity and can achieve a certain degree of morphological change; three-dimensional deformation adjustment is suitable for situations with high expression intensity and can perform more complex morphological changes based on the user's expression. This method improves the flexibility and creativity of the toy, enhances the toy's interactivity, expressiveness, and fun, and meets the user's demand for morphological changes.

[0113] By dividing the deformation strategies into two levels and selecting the appropriate deformation method according to the intensity of the expression, the scheme effectively optimizes the use of computing resources and reduces the impact of complex calculation processes on real-time performance, thereby improving the toy's response speed and user experience.

[0114] The human-computer interaction method based on expression recognition disclosed in this application makes human-computer interaction more natural and intuitive by capturing and analyzing user expressions in real time. By preprocessing the collected user facial images and determining the effective facial area, background noise can be effectively removed, improving the recognition accuracy of facial features. By extracting preset key point information of the face, the scheme can accurately capture the geometric and texture features of the face. Based on the constructed classification model, the extracted facial features are analyzed to obtain the user's specific expression information, which can effectively improve recognition accuracy and overcome the limited expression recognition ability of traditional technologies. The expression intensity level is determined based on the user's expression information. If the expression intensity level is low, a first strategy is adopted to perform multi-target mesh deformation adjustment; if the expression intensity level is high, a second strategy is adopted to perform three-dimensional deformation adjustment. This method enables target objects (such as toys) to provide different interactive feedback based on the user's emotional intensity, thereby providing a richer and more diverse interactive experience. That is, the deformation of the target object is adjusted according to the expression intensity level, so that the target object matches the user's emotional state, enhancing the user's immersion and interactive experience. Different adjustment strategies can be adopted according to different expression intensities, improving the flexibility and adaptability of the system. This expression recognition-based method can significantly improve the intelligence level of the human-computer interaction system, allowing users' emotions and expressions to be more accurately reflected in the virtual environment, and providing users with a more intelligent, rich, and personalized interactive experience.

[0115] Reference Figure 2 , the method for determining the effective facial area specifically includes the following steps:

[0116] S110, an image acquisition device based on a preset resolution captures a user's facial image in real time.

[0117] Among them, the preset resolution refers to a high resolution of not less than 1080P, which ensures that clear facial images can be captured and provides more accurate data for subsequent processing.

[0118] In this embodiment, the image acquisition device is preferably a camera installed in the toy that can capture the user's facial image in real time or custom-developed image acquisition software; through real-time capture, it is suitable for application scenarios that require instant feedback, such as intelligent interaction between the toy and the user (such as a child).

[0119] S120: Enhance the user's facial image using a preset enhancement strategy.

[0120] The preset enhancement strategy includes one or more of a histogram equalization strategy and a Gaussian filtering strategy.

[0121] If a histogram equalization strategy is adopted, that is, applying a histogram equalization algorithm (such as `cv2.equalizeHist` in OpenCV) to enhance the contrast of the image, the visibility of facial features can be effectively improved. Specifically, the collected facial image is converted to a grayscale image and histogram equalization is applied to improve the contrast of the image.

[0122] If a Gaussian filtering strategy is adopted, the image can be smoothed, noise can be reduced, and image quality can be improved. Specifically, an appropriate Gaussian kernel size (such as 5x5 or 7x7) is selected to filter the image to smooth the facial image and reduce the interference of noise on facial area recognition.

[0123] Furthermore, a combination of both strategies can be employed, meaning that the enhancement effect can be adjusted to specific needs.

[0124] Through enhancement processing, the contrast and clarity of the image can be effectively improved, facial features can be made more obvious, and the impact of environmental factors (such as insufficient light) on image quality can be reduced.

[0125] S130 , processing the enhanced user facial image based on the Viola-Jones algorithm to determine a valid facial area.

[0126] In this step, the enhanced image is processed using the Viola-Jones face detection algorithm (such as `cv2.CascadeClassifier` in OpenCV) to detect and locate the facial area. Specifically, a pre-trained face detection classifier (such as a Haar cascade classifier) ​​is loaded and applied to the enhanced image to detect the facial area and determine the bounding box of the valid facial area.

[0127] Furthermore, the detection parameters (such as scaling ratio, minimum face size for detection) can be adjusted as needed to improve the detection accuracy.

[0128] By adopting the Viola-Jones algorithm, facial areas can be identified quickly and accurately, and facial areas can be precisely determined, providing a reliable basis for subsequent feature extraction and analysis.

[0129] The method for determining the effective facial area disclosed in this embodiment improves the clarity and contrast of the image through enhancement processing, ensuring the accuracy of subsequent facial detection and feature extraction. It uses the Viola-Jones algorithm for efficient and accurate facial area detection, which can reduce computational overhead. At the same time, it can provide accurate facial area information, making subsequent expression analysis and human-computer interaction more natural and accurate. By selecting appropriate enhancement strategies, it can adapt to different environmental conditions (such as lighting changes) and device characteristics, improving the robustness of the system. The overall solution achieves efficient and accurate facial area detection through the precise coordination of image acquisition, enhancement, and facial area recognition, providing reliable data support for subsequent facial feature analysis and expression recognition.

[0130] Reference Figure 3 , the method for obtaining user expression information specifically includes the following steps:

[0131] S410, constructing an initial learning model.

[0132] Specifically, 1) the base network of the initial learning model is determined to be the lightweight EfficientNet B0, which can strike a balance between computational efficiency and classification accuracy; 2) the model is initialized: the pre-trained weights of EfficientNet B0 are used as the initial weights, which can accelerate the training process and improve model performance; 3) the model architecture is configured: the last few layers of the network are adjusted as needed, such as adding a fully connected layer or a customized output layer, to adapt to the expression classification task.

[0133] S420, using a preset expression data set to train an initial learning model, and using the trained initial learning model as a classification model;

[0134] The preset expression dataset includes a dataset of facial images with different expressions.

[0135] Specifically, the preset expression dataset includes multiple expression categories such as smile, laughter, sadness, surprise, etc.; the preset expression dataset is used to train the initial learning model. Specifically, the appropriate loss function (such as cross entropy loss) and optimizer (such as Adam) can be set, and hyperparameters such as learning rate can be adjusted; the model trained by this method can adapt to the expression characteristics in practical applications, and can fully understand and classify various expressions.

[0136] S430, analyzing facial features based on the classification model to obtain expression type and expression amplitude;

[0137] Expression type and expression amplitude constitute user expression information.

[0138] Specifically, a trained classification model is used to extract and analyze features from the input facial image. The model predicts the type of expression (e.g., happiness, anger, etc.) based on the features in the input image. The model's output probability values ​​or additional regression analysis can be used to estimate the magnitude of the expression. This step provides accurate expression classification information, helping to understand the user's emotional state. The estimation of expression magnitude provides more detailed information about the intensity of the emotion, facilitating deeper emotional analysis.

[0139] Reference Figure 4 The method for determining the expression intensity level specifically includes the following steps:

[0140] S510: Add a regression head based on the classification model to obtain an initial regression model.

[0141] The initial regression model obtained in this step is different from the classification model. The goal of the regression model is to predict a continuous value rather than a discrete category. For expression intensity estimation, the model needs to output an expression intensity value, which is usually a continuous value (for example, ranging from 0 to 1, indicating that the intensity of the expression ranges from none to strong).

[0142] Among them, the regression head is usually a fully connected layer or a group of fully connected layers, which is responsible for converting the features extracted by the classification model into expression intensity values.

[0143] By adding a regression head on the basis of the classification model, the obtained model can not only classify expressions but also output a continuous value as the expression intensity.

[0144] S520, generating a synthetic expression image based on a generative adversarial network.

[0145] Generative Adversarial Networks (GANs) are used to generate synthetic data similar to real data. By training a GAN, synthetic expression images of varying intensities can be generated, thereby expanding the dataset. Specifically, GANs are used to generate synthetic images of various expressions. For example, a generator can generate images of varying intensities of expression, while a discriminator ensures that the generated images are visually realistic.

[0146] The facial expression images generated using GAN can contain different intensities, which helps the model see more diverse intensity samples during training and improves its ability to recognize and estimate expressions of different intensities. In other words, it increases the model's ability to recognize different facial expression intensities and improves the model's generalization ability.

[0147] S530, obtaining a preset expression library based on the preset expression data set and the synthesized expression image.

[0148] By combining real and synthetic images, the preset expression library will contain a wider range of expression intensities, making the training data more comprehensive and helping to improve the accuracy and robustness of the model.

[0149] S540: Train the initial regression model based on the preset expression library, and record the trained initial regression model as the target regression model.

[0150] An initial regression model is trained using images and labeled data from a pre-set expression library. During training, the model learns the characteristics of different expression intensities and optimizes its regression output. Through training, the model can more accurately predict expression intensity values, and the target regression model can better adapt to real and synthetic data, thereby improving its ability to assess expression intensity.

[0151] S550: Analyze the user's facial expression information based on the target regression model to obtain an expression intensity value, and obtain a corresponding expression intensity level based on a preset expression intensity information library.

[0152] Specifically, the user's facial expression image is input into the target regression model, and the model outputs an expression intensity value. This intensity value is then compared with the information in a preset expression intensity information library to determine the expression intensity level. This step can convert the continuous expression intensity value predicted by the model into a discrete level, which is helpful for further application and interpretation, such as in sentiment analysis or human-computer interaction.

[0153] In this embodiment, by using a generative adversarial network for synthesized images, the diversity of training data is enhanced, thereby improving the performance of the model in different scenarios; synthesized expression images can make up for the problem of insufficient samples in the actual data set, thereby improving the richness and coverage of the data set; the model can not only classify expressions, but also evaluate expression intensity, providing more comprehensive expression analysis results; by using a preset expression library for training, it can adapt to different application scenarios and needs, thereby improving the flexibility and application breadth of the model; through accurate expression intensity evaluation, it can provide more realistic sentiment analysis, which helps to improve the user's interactive experience with the system.

[0154] Reference Figure 5 The method for deformation adjustment based on the first strategy specifically includes the following steps:

[0155] A100 processes user expression information based on several constructed grid models to obtain geometric difference information, physical difference information, and aesthetic difference information respectively.

[0156] The specific steps include:

[0157] A110, constructing a geometry preservation model, an expression recognition model, and a material usage model respectively. That is, the several mesh models constructed in this step refer to the geometry preservation model, the expression recognition model, and the material usage model.

[0158] A120, based on the geometry-preserving model, geometric difference information can be extracted. Specifically, geometric difference information refers to the distance and shape difference calculated between the geometry-preserving model and the user's expression. For example, when the user's expression is surprised, the Euclidean distance between the current cube shape and the target "surprised" shape can be measured. Extracting geometric difference information can help assess the difference between the current shape and the target shape.

[0159] Physical difference information can be obtained based on the expression recognition model. Specifically, by simulating skin deformation and lighting effects, physical changes are detected to evaluate the energy changes during deformation and ensure that the deformation conforms to the physical properties of the material (for example, avoiding unrealistic deformation).

[0160] Aesthetic difference information can be obtained based on the material usage model. Specifically, by comparing whether the expression meets aesthetic standards, such as natural facial expression curves and sense of balance, for example, calculating whether the deformed shape is symmetrical and balanced to ensure its visual effect is good.

[0161] A200 obtains the target body deformation strategy based on geometric difference information, physical difference information, and aesthetic difference information.

[0162] For example, if geometric difference information shows that facial features need to be moved in specific areas, physical difference information indicates that the skin needs to be stretched or compressed, and aesthetic difference information indicates that the expression needs to be made more natural and in line with aesthetic standards, then corresponding target body deformation strategies can be formulated, which may include adjusting the position of mesh vertices, modifying surface curvature, or applying physical simulation technology.

[0163] The developed target body deformation strategy will combine multiple aspects of information to achieve more accurate and comprehensive deformation adjustment, ensuring that the adjusted results meet the requirements in terms of geometry, physics and aesthetics.

[0164] A300 performs multi-target mesh deformation adjustment on deformable targets based on target body deformation strategy.

[0165] Specifically, the deformation strategy formulated in step A200 is applied to the target object to be deformed (such as a three-dimensional facial model of a toy), and a multi-target mesh deformation technology (for example, mesh adjustment based on an optimization algorithm or a physical simulation method) is used to adjust and deform the mesh vertices of the target object. This can achieve a variety of target effects, such as correcting geometric deviations of facial expressions, optimizing physical performance, and improving aesthetic effects. By applying the target body deformation strategy, the target object can be effectively adjusted to better meet the expected expression characteristics and aesthetic standards. Multi-target mesh deformation adjustment can comprehensively consider the needs of multiple aspects to ensure the high quality and high accuracy of the final effect.

[0166] In this embodiment, by processing the user's expression information in detail and obtaining geometric, physical and aesthetic difference information, a more accurate deformation strategy can be formulated to improve the deformation effect of the target object; by comprehensively considering the geometric differences, physical differences and aesthetic differences, it is ensured that the deformation not only achieves the expected technical results, but also achieves good visual and sensory effects; through refined grid deformation adjustment, the user's expression can be better simulated and adjusted, so that the final result is more in line with user expectations, thereby improving the user experience; this method is applicable to a variety of target objects and expression deformation scenarios, has strong adaptability and flexibility, and can be widely used in virtual reality, animation production, game development and other fields; by considering the aesthetic difference information, it can ensure that the deformed target object is more visually natural and more in line with aesthetic standards, thereby enhancing the overall visual effect.

[0167] Reference Figure 6 , the method for obtaining the target body deformation strategy specifically includes the following steps:

[0168] A210 extracts geometric gradient, physical gradient and aesthetic gradient based on geometric difference information, physical difference information and aesthetic difference information respectively.

[0169] Specifically, for geometric gradients, by comparing the geometric differences between the target object's mesh model and the reference model, the changes in the position and shape of the mesh vertices are extracted. For example, geometric analysis methods (such as mesh distance calculation) are used to obtain the displacement gradients of the mesh vertices.

[0170] For physical gradients: Physical gradients are obtained by simulating physical deformations (such as skin stretching or contraction), which can be calculated by physical simulation tools, such as using finite element analysis (FEA) to estimate the effects of physical deformations.

[0171] For aesthetic gradient: Aesthetic gradient is extracted by evaluating the surface quality and visual effects of the target object, which can be compared with aesthetic standards (such as naturalness of facial expression, symmetry, etc.) to calculate the aesthetic-related adjustment amount.

[0172] The geometric gradient can accurately capture the shape changes of the mesh and provide detailed geometric information for subsequent adjustments; the physical gradient takes into account the impact of physical deformation on the mesh, making the adjustment more consistent with real-world physical conditions; the aesthetic gradient ensures that the deformed result visually meets aesthetic standards, improving the visual effect and user experience of the target object.

[0173] In this embodiment, the extracted geometric gradient is used as the shape fidelity score, the obtained physical gradient is used as the expression similarity score, and the obtained aesthetic gradient is used as the material naturalness score.

[0174] A220, based on the adaptive weight strategy, fuses the geometric gradient, physical gradient, and aesthetic gradient to obtain the fusion score function;

[0175] The fusion score function includes: F = w1*physical gradient + w2*geometric gradient + w3*aesthetic gradient; among them, w1 is the expression adaptation weight, w2 is the geometric adaptation weight, and w3 is the material adaptation weight.

[0176] By adjusting the weights, it is possible to optimize for different application requirements and target objects, providing a more flexible and adaptive deformation strategy; by fusing different gradient information, it is possible to comprehensively consider geometric, physical and aesthetic requirements, ensuring that the deformation strategy is more comprehensive and accurate.

[0177] A230, calculating mesh vertex gradients of the target object to be deformed based on the fusion score function; the target body deformation strategy includes mesh vertex gradients.

[0178] The fusion score function FF is used to calculate the gradient of the mesh vertices of the target object. According to the weights and gradient values ​​of each item in FF, the adjustment direction and amplitude of each mesh vertex are calculated. The calculated mesh vertex gradient is applied to the target object to be deformed, and the deformation of the target object is completed by adjusting the position of the mesh vertices.

[0179] By calculating the gradients of mesh vertices, precise mesh adjustment can be achieved to ensure that the deformation effect of the target object meets the expectations; based on the comprehensive information provided by the fusion score function, the adjusted target object can better meet the geometric, physical and aesthetic requirements.

[0180] The method disclosed in this embodiment provides a comprehensive deformation strategy by combining geometric, physical, and aesthetic gradients, ensuring that all aspects of the deformation process are taken into account, thereby improving the accuracy and comprehensiveness of the deformation results; the adaptive weight strategy enables the method to adjust the deformation strategy according to different application scenarios and needs, thereby enhancing the flexibility and adaptability of the solution; by calculating the mesh vertex gradients, efficient and accurate deformation of the target object can be achieved to meet specific deformation requirements; by comprehensively considering the user's expression, physical characteristics, and aesthetic standards, the final target object can better meet the user's expectations, improving the overall visual effect and user experience.

[0181] Furthermore, the present application also includes: iteratively updating the position of the mesh vertex based on the gradient descent method or the preset optimization algorithm; the gradient descent method includes: V new =V old -η·BSD Score , where η is the learning rate, which determines the step size of each update, V old For the old gradient, V new is the new gradient. In each iteration, a mesh preserving constraint is applied to ensure that the topology remains unchanged.

[0182] Furthermore, the present application also includes: a user feedback mechanism that allows fine-tuning of weights based on the toy interaction effect, and an evaluation mechanism; the evaluation mechanism can evaluate the results from multiple dimensions, including expressiveness, geometric fidelity, deformation smoothness, etc., and then based on the evaluation results, feedback can be used to adjust the hyperparameters in the deformation adjustment process.

[0183] The following is an example of an interactive teddy bear that changes according to a child's facial expressions.

[0184] First, an expression recognition model is trained to identify basic emotions (such as happiness, sadness, surprise, etc.); a geometry preservation model is trained to ensure that the overall shape characteristics of the teddy bear are maintained when it is deformed; and a material adaptation model is trained to take into account the deformation characteristics of the teddy bear's fur.

[0185] Secondly, the gradient of the expression recognition model is extracted as the expression similarity score; the gradient of the geometry preservation model is extracted as the shape fidelity score; the gradient of the material adaptation model is extracted as the material naturalness score; the weighting function is: F = w1*physical gradient + w2*geometric gradient + w3*aesthetic gradient.

[0186] Then, the 3D mesh model of the teddy bear is initialized, including the facial key points; the system detects that the child is smiling and starts the deformation process; the gradient of F is calculated to guide the movement of the mesh vertices; the optimization algorithm may first lift the corners of the mouth and then adjust the eye area, while keeping the bear's ears and overall outline unchanged.

[0187] Initial settings: w1(expression)=0.5, w2(geometry)=0.3, w3(texture)=0.2.

[0188] If the expression is not obvious enough after deformation, the value of w1 can be automatically increased; if the shape of the bear becomes unnatural, the value of w2 can be increased.

[0189] In another example, we use a simple cube toy as an example to demonstrate how to implement multi-target deformation. Suppose we want to deform the cube into a spiked shape based on the user's "surprise" expression.

[0190] For geometric gradients: the Euclidean distance between the current shape and the target "surprise" shape can be calculated to evaluate the change in surface curvature and encourage the formation of sharp features.

[0191] For physical gradients: calculate deformation energy, ensure that the deformation process conforms to the physical properties of the material, evaluate structural stability, and prevent collapse caused by excessive deformation.

[0192] For aesthetic gradients: assess the symmetry and balance of shapes; calculate surface smoothness to avoid unnecessary wrinkles.

[0193] Then the fusion score function is obtained. At the same time, the weights can be adjusted dynamically, for example, increasing w1 in the early stage of deformation and increasing w2 and w3 in the later stage.

[0194] Furthermore, iterative optimization can be performed. Specifically, the mesh vertex positions are updated using the gradient descent method. After each iteration, the scoring function and gradient are recalculated until a preset number of iterations is reached or the deformation effect meets the requirements.

[0195] Through this process, the cube toy will gradually form spike-like protrusions while maintaining the stability and aesthetics of the overall structure. For example, a large spike may form on the top, while multiple smaller spikes may form on the sides, creating a "surprising" visual effect.

[0196] Reference Figure 7 The method for deformation adjustment based on the second strategy specifically includes the following steps:

[0197] B100, create a target Octree structure and inject noise into the target Octree structure.

[0198] Specifically, an Octree structure is created for the three-dimensional mesh data of the target object. Octree is a spatial partitioning method that recursively divides the three-dimensional space into eight sub-regions and is suitable for processing the hierarchical structure of large-scale three-dimensional data. Random noise is added to each node of the Octree structure. This can be achieved by introducing small-scale random changes in the nodes, thereby providing the necessary conditions for the subsequent denoising process.

[0199] In this step, the Octree structure can efficiently manage and process complex three-dimensional data, making subsequent processing more efficient. The injection of noise makes the subsequent denoising process more challenging, improving the robustness of the model to actual noise and irregular data.

[0200] B200, extract key expression features from the user's expression information and convert the key expression features into a first vector.

[0201] Extract key facial features from the user's facial expression image, such as key points and specific changes in expression. Encode these key features into a vector. For example, use a deep learning model to map facial expressions into feature vectors. Extracting key facial features captures the core information of the expression, providing a concrete basis for subsequent adjustments. By vectorizing these features, they can be more easily used for calculations and fusion.

[0202] B300 converts the preset style of the target object to be deformed into a target vector, which includes style features and style requirements.

[0203] Specifically, define the style features and style requirements of the target object, such as style details, color, and texture, and convert these style features and requirements into a target vector. The style information may be converted into a vector representation through a style encoder. In this step, clarifying the style features and requirements of the target object helps to maintain style consistency during the deformation process, vectorize the style information, and facilitate integration and adjustment in the model.

[0204] B400, fuse the first vector with the target vector to obtain the conditional vector.

[0205] Specifically, the extracted first vector (expression feature vector) and the target vector (style feature vector) are fused, and weighted summation or other fusion techniques can be used. The obtained fusion vector is used as a conditional vector to describe the requirements of the target object under specific expressions and styles.

[0206] In this step, by fusing expression features and style features, the obtained conditional vector can comprehensively consider expression and style information, providing comprehensive conditions for subsequent deformation.

[0207] B500, builds an initial diffusion model; the initial diffusion model includes a neural network based on a 3D UNet structure.

[0208] Specifically, a neural network model based on 3D UNet is constructed. UNet is a network architecture used for image segmentation, and 3D UNet is extended to three-dimensional data processing; the model is initialized to the starting state of the diffusion model.

[0209] In this step, 3D UNet can effectively process three-dimensional data and is particularly suitable for image segmentation and generation tasks; the initial diffusion model can provide an effective basis for subsequent denoising and deformation adjustment.

[0210] B600, injects conditional vectors at different layers of the neural network to obtain the first model.

[0211] Specifically, the conditional vectors are injected into different layers of the 3D UNet network, either through concatenated layers or additional feature fusion modules, resulting in a 3D UNet model that incorporates the conditional vectors. Injecting conditional vectors at different levels allows the model to consider the influence of expression and style at multiple levels, improving the accuracy of the generated results.

[0212] B700, trains the first model based on the historical 3D expression dataset and uses the trained first model as the target diffusion model.

[0213] Specifically, the first model is trained using a historical 3D expression dataset, which includes 3D data of various expressions and styles. The trained model can better adapt to actual data and perform effective 3D deformation adjustment.

[0214] In this step, using real data for training can improve the model's adaptability and accuracy to actual application scenarios. The trained model can better perform diffusion and denoising tasks.

[0215] B800, through the target diffusion model, gradually denoises the target Octree structure after the noise is injected to obtain the target three-dimensional data.

[0216] Specifically, the noise-injected Octree structure is input into the trained target diffusion model. The model performs denoising multiple times and gradually restores the details of the target three-dimensional data. The final three-dimensional data contains the detailed structure of the target object after denoising.

[0217] In this step, progressive denoising can effectively restore the details of the target object and improve the quality of the three-dimensional data; the three-dimensional data processed by the diffusion model can better reflect the actual deformation results of the target object.

[0218] B900 performs 3D deformation adjustment on the target object to be deformed based on the target 3D data.

[0219] Specifically, the target three-dimensional data is used to adjust the target object to be deformed, and the target three-dimensional data is applied to the mesh model of the target object to perform detailed three-dimensional deformation processing; for detail recovery, the deformation algorithm can be used to match and adjust the target three-dimensional data with the target object to ensure that the target object meets the expected deformation effect.

[0220] In this step, deformation adjustments based on high-quality target 3D data can ensure that the final target object meets the set requirements. The high precision of the 3D data can restore the details of the target object, making the deformation effect more realistic and natural.

[0221] The solution disclosed in this embodiment can achieve high-precision three-dimensional deformation adjustment by combining expression features, style requirements and diffusion models, ensuring that the deformation effect of the target object meets the expectations; the diffusion model is used to gradually denoise the data injected with noise, improving the data quality and ensuring the accuracy of the deformation results; the solution integrates multiple aspects of expression, style and three-dimensional data processing, so that the deformation adjustment not only meets technical requirements, but also meets visual and style requirements; the method is applicable to a variety of three-dimensional deformation tasks, such as virtual character animation, character modeling in games, model optimization in industrial design, etc., and has strong adaptability and flexibility; through precise deformation adjustment, it can provide more natural and user-expected results, thereby improving the overall user experience and satisfaction.

[0222] The method for creating the target Octree structure specifically includes the following steps:

[0223] B110, create the root node.

[0224] This step is the starting point of the Octree structure. First, you need to create a root node, which represents the outermost bounding volume of the entire three-dimensional space. The root node contains the entire generated space (that is, the spatial range of the entire model or scene).

[0225] Through this step, the boundary of the entire three-dimensional space can be determined, and this root node provides the basis for subsequent space division.

[0226] B120, spatial division.

[0227] After the root node is created, it needs to be recursively partitioned. Specifically, Octree can divide the space into 8 sub-cubes (sub-nodes), each of which represents a part of the original space.

[0228] The partitioning process is performed recursively until the volume of each child node reaches the preset minimum voxel size, which means that the space will be subdivided into very small cubes, which are called voxels.

[0229] By recursively partitioning, the entire space can be subdivided into smaller voxels, which allows the data to be processed at a higher resolution. For example, complex geometries and details can be represented more accurately.

[0230] B130, attribute code.

[0231] After the division is completed, each leaf node (i.e., the smallest voxel node) needs to encode its occupancy state and surface features; the occupancy state indicates whether the voxel is occupied by an object, and the surface features may include the voxel's color, texture and other attributes.

[0232] In this step, it is determined whether each voxel is in the space of the target object, which helps to judge the boundary and internal structure of the object; recording the surface features of the voxel helps to provide necessary information for further processing (such as texture mapping, lighting calculation, etc.).

[0233] The method for creating a target Octree structure disclosed in this embodiment establishes the boundary of the entire three-dimensional space as the starting point of the Octree; through recursive partitioning, the entire space is subdivided into smaller voxels to provide a high-resolution representation for processing details; and the occupancy state and surface characteristics of each voxel are recorded so that subsequent processing can use this information for further analysis and processing.

[0234] The following example uses the generation of a complex "surprised face" 3D model to demonstrate how to use the Octree-based diffusion model to achieve this.

[0235] 1) Octree initialization: Create a root node to represent the entire generation space (e.g. 1m 3 cube); recursively divide the space until a preset minimum voxel size is reached (e.g. 1cm 3 ); assign initial states (occupied / empty) and surface feature values ​​to leaf nodes.

[0236] 2) Conditional information encoding.

[0237] The "surprise" facial expression features are converted into vectors (for example, eyebrow raising, mouth opening, etc.); the target style is encoded (such as cartoon style or realistic style); and the facial expression features and style information are fused into a unified conditional vector.

[0238] 3) Diffusion model design and training.

[0239] Build a neural network based on 3D UNet and adapt it to the Octree structure; inject the conditional vectors generated in step B2 into multiple layers of the network; and train the model using a large-scale 3D expression dataset.

[0240] 4) Stepwise denoising. Starting from a high noise level, a certain amount of noise is predicted and removed at each step. At each step, the influence of the conditional vector is considered. The Octree structure is dynamically adjusted to subdivide areas where more detail is needed. The denoising process is repeated until a clear 3D shape is obtained.

[0241] Furthermore, post-processing strategies can be included. Specifically, a modified Marching Cubes algorithm is used to reconstruct the mesh from the Octree; mesh smoothing and optimization algorithms are applied to improve surface quality; and detailed features such as texture and microstructure are added.

[0242] This process generates an expressive 3D model of a "surprised face." For example, the eyebrows might be accentuated, creating an upward curve; exaggerated circular depressions might form around the eyes; and a large oval opening might be created in the mouth, creating a puffed-up effect across the entire face, emphasizing the emotion of surprise.

[0243] This Octree-based method can not only efficiently generate complex 3D shapes, but also achieve precise control based on the input expression features and style information, thereby creating rich and diverse expression models.

[0244] Furthermore, the human-computer interaction method based on expression recognition disclosed in this application also includes system integration and interaction control, specifically including asynchronous processing mechanism, caching mechanism, interaction mechanism, performance optimization mechanism, and user experience optimization mechanism.

[0245] The asynchronous processing mechanism includes implementing asynchronous message queues to ensure system real-time performance. Asynchronous processing means that messages between modules do not need to wait for immediate responses from each other. Implementing asynchronous message queues allows the system to process tasks in the background without blocking the main thread or waiting for immediate feedback from other modules. This allows other modules to continue working even if a module is temporarily unable to process a message. Asynchronous processing can significantly improve the system's real-time performance and responsiveness. It allows the system to process multiple tasks in parallel, making more efficient use of computing resources. In addition, by reducing waiting time, users can receive feedback more quickly, thereby improving the overall user experience.

[0246] Caching mechanisms include configuring intelligent caching to reduce duplicate computations. This allows for the storage and rapid retrieval of previously calculated data or intermediate results. This allows the system to retrieve the results directly from the cache when the same data is requested again, eliminating the need to recalculate. This significantly improves system computing efficiency and responsiveness, helping conserve computing resources, particularly when processing complex or time-consuming tasks. Furthermore, reducing unnecessary computations reduces system energy consumption and heat generation.

[0247] The interaction mechanism includes the interaction state mechanism, conversion logic mechanism, feedback mapping mechanism, and multimodal output mechanism; the interaction state mechanism is used to realize different interaction states of the toy, the conversion logic mechanism is used to realize different core logic controls of the toy, the feedback mapping mechanism is used to map the generated 3D shape to the physical feedback of the toy, and the multimodal output mechanism is used to coordinate various output methods such as sound and light.

[0248] The performance optimization mechanism includes task scheduling strategy, resource dynamic allocation strategy, and predictive computing strategy. Among them, the task scheduling strategy is used to implement a priority-based task scheduling strategy, the resource dynamic allocation strategy is used to dynamically adjust the resource occupancy of each module according to the current load, and the predictive computing strategy is used to use historical data to predict possible interactions and perform calculations in advance.

[0249] The user experience optimization mechanism includes response time control, smooth transition control, and personalized adaptation control. Among them, response time control is used to ensure that the response time of key interactions does not exceed 100ms, smooth transition control is used to control the smooth transition between different expressions and shapes, and personalized adaptation control is used to record and analyze user habits and dynamically adjust interaction parameters.

[0250] Example 1: Smile Recognition and Toy Transformation

[0251] When the user smiles at the toy, the first step is to accurately capture the degree of the smile. Suppose the smile recognized is of medium intensity.

[0252] Next, the image is processed according to the preset "smile" deformation target. For example, for a spherical toy, the deformation adjustment includes: the upper half of the sphere will gradually expand to form a shape similar to the upward corners of a smile; at the same time, the surface of the sphere will generate depressions similar to eyes, further enhancing the visual effect of the "smiley face".

[0253] The entire deformation process takes physical constraints into account to ensure that the deformed shape still maintains balance and structural stability.

[0254] Example 2: Surprised Expression and 3D Shape Generation

[0255] When the user shows an expression of surprise, that is, when the intensity level of the expression is not less than a preset threshold, three-dimensional deformation adjustment will be performed.

[0256] Specifically, the user's surprised expression is captured and its intensity is evaluated. The features of the surprised expression are then encoded as conditional information and input into the diffusion model. The diffusion model, based on the Octree structure, generates a series of 3D shapes related to "surprise", such as suddenly popping spikes and rapidly expanding bubbles. The generated 3D shapes are reconstructed into meshes using the Marching Cubes algorithm and merged with the deformation of the toy itself. Ultimately, the toy will present a shape with a sudden expansion in the center and multiple small protrusions around it, vividly expressing the concept of "surprise".

[0257] The "target object" in this embodiment refers to an object that can be transformed or adjusted based on the user's facial expression information. Specifically, the target object can be a toy, a robot, a smart device, a virtual or animated character, a user interface element, a digital product, an environmental element in a virtual reality, etc.

[0258] For virtual or animated characters: In virtual reality (VR), augmented reality (AR), or games, the user's facial expressions can be used to drive the virtual character's facial expressions, making the character's performance more vivid and natural.

[0259] For user interface elements: For example, when a user interacts with a piece of software or an application, certain elements of the interface can be adjusted based on the user's facial expressions. For example, facial expression recognition can be used to adjust the animation effects of buttons or the way information is displayed in the interface.

[0260] For digital products or content: including digital avatars, emoticons, etc., their appearance can be adjusted according to the user's real-time expression to enhance the user experience.

[0261] For environmental elements in virtual reality: for example, objects or landscapes in the virtual environment can be dynamically adjusted based on the user's emotions or expressions, making the virtual environment more consistent with the user's emotional state.

[0262] The deformation adjustment of these targets can be adapted by selecting different strategies according to the intensity of the expression to achieve a more natural and personalized interactive experience.

[0263] A second aspect of the present application discloses an interactive intelligent toy, comprising a toy body and a master control center disposed in the toy body;

[0264] The main control center uses any of the human-computer interaction methods based on expression recognition to control the intelligent interaction between the toy body and the user;

[0265] The toy body is the target object to be transformed.

[0266] A third aspect of the present application discloses a robot, comprising a robot body and a master control center disposed in the robot body;

[0267] The main control center uses any of the human-computer interaction methods based on facial expression recognition to control the intelligent interaction between the robot body and the user;

[0268] The robot body is the target object to be deformed.

[0269] For robots or smart devices: In some high-end robots or smart devices, the user's facial expressions can be used to control the robot's expressions or movements, making interaction with the robot more natural and intuitive.

[0270] The fourth aspect of the present application discloses a human-computer interaction system based on expression recognition, comprising:

[0271] A preprocessing module is used to preprocess the user's facial image collected in real time and determine the effective facial area;

[0272] An extraction module is used to extract preset key point information in the effective facial area;

[0273] A feature determination module is used to determine facial features based on preset key point information, where facial features include facial geometric features and facial texture features;

[0274] The analysis module is used to analyze facial features based on the constructed classification model to obtain user expression information;

[0275] An intensity level acquisition module is used to determine the intensity level of an expression based on a preset expression library and user expression information;

[0276] A judgment module, configured to trigger a first strategy to perform multi-target mesh deformation adjustment on the target object to be deformed if the expression intensity level is less than a preset threshold;

[0277] If the expression intensity level is not less than a preset threshold, a second strategy is triggered to perform three-dimensional deformation adjustment on the target object to be deformed, and the appearance of the target object after deformation adjustment matches the user's expression.

[0278] It should be noted that the solutions disclosed in the human-computer interaction method based on expression recognition disclosed in the first aspect of this application are all applicable to the human-computer interaction system based on expression recognition disclosed in the fourth aspect of this application, so they will not be repeated here.

[0279] The computer device according to an embodiment of the present disclosure includes a memory and a processor. The memory is used to store non-transitory computer-readable instructions. Specifically, the memory may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include, for example, random access memory (RAM) and / or cache memory (cache). The non-volatile memory may include, for example, read-only memory (ROM), a hard disk, a flash memory, etc.

[0280] The processor may be a central processing unit (CPU) or other form of processing unit with data processing capabilities and / or instruction execution capabilities, and may control other components in the computer device to perform desired functions. In one embodiment of the present disclosure, the processor is used to execute the computer-readable instructions stored in the memory, so that the computer device executes all or part of the steps of the human-computer interaction method based on expression recognition in each embodiment of the present disclosure.

[0281] Those skilled in the art should understand that in order to solve the technical problem of how to obtain a good user experience, this embodiment may also include well-known structures such as a communication bus and an interface, and these well-known structures should also be included in the scope of protection of this disclosure.

[0282] like Figure 8 The present invention provides a schematic diagram of the structure of a computer device according to an embodiment of the present invention. Figure 8 The computer device shown is only an example and should not limit the functions and scope of use of the embodiments of the present disclosure.

[0283] like Figure 8 As shown, the computer device may include a processor (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) or a program loaded from a storage device into a random access memory (RAM). Various programs and data required for the operation of the computer device are also stored in the RAM. The processor, ROM, and RAM are connected to each other via a bus. An input / output (I / O) interface is also connected to the bus.

[0284] Typically, the following devices can be connected to the I / O interface: input devices such as sensors or visual information acquisition devices; output devices such as display screens; storage devices such as tapes and hard disks; and communication devices. The communication device can allow the computer device to communicate with other devices (such as edge computing devices) wirelessly or by wire to exchange data. Figure 8A computer device having various devices is shown, but it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed instead.

[0285] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a non-transitory computer-readable medium, and the computer program contains program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through a communication device, or installed from a storage device, or installed from a ROM. When the computer program is executed by the processor, all or part of the steps of the human-computer interaction method based on expression recognition of the embodiment of the present disclosure are executed.

[0286] For detailed description of this embodiment, please refer to the corresponding description in the aforementioned embodiments, which will not be repeated here.

[0287] According to an embodiment of the present disclosure, a computer-readable storage medium stores non-transitory computer-readable instructions. When the non-transitory computer-readable instructions are executed by a processor, all or part of the steps of the human-computer interaction method based on expression recognition described in the aforementioned embodiments of the present disclosure are executed.

[0288] The above-mentioned computer-readable storage media include, but are not limited to, optical storage media (e.g., CD-ROMs and DVDs), magneto-optical storage media (e.g., MOs), magnetic storage media (e.g., magnetic tapes or mobile hard disks), media with built-in rewritable non-volatile memory (e.g., memory cards), and media with built-in ROM (e.g., ROM cartridges).

[0289] For detailed description of this embodiment, please refer to the corresponding description in the aforementioned embodiments, which will not be repeated here.

[0290] The basic principles of the present disclosure have been described above in conjunction with specific embodiments. However, it should be noted that the advantages, strengths, and effects mentioned in this disclosure are merely illustrative and not restrictive, and should not be construed as necessarily possessed by each embodiment of the present disclosure. Furthermore, the specific details disclosed above are provided for illustrative purposes and to facilitate understanding, rather than as limitations. These details do not limit the present disclosure to necessarily being implemented using these specific details.

[0291] In the present disclosure, relational terms such as first and second, etc. are merely used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply that there is any such actual relationship or order between these entities or operations. The block diagrams of the devices, devices, equipment, and systems involved in the present disclosure are merely illustrative examples and are not intended to require or imply that they must be connected, arranged, or configured in the manner shown in the block diagrams. As will be appreciated by those skilled in the art, these devices, devices, equipment, and systems can be connected, arranged, or configured in any manner. Words such as "including," "comprising," "having," and the like are open-ended words, meaning "including but not limited to," and can be used interchangeably therewith. The words "or" and "and" used herein refer to the words "and / or" and can be used interchangeably therewith, unless the context clearly indicates otherwise. The word "such as" used herein refers to the phrase "such as but not limited to," and can be used interchangeably therewith.

[0292] Additionally, as used herein, "or" used in a list of items beginning with "at least one" indicates a separate list, so that, for example, a list of "at least one of A, B, or C" means A or B or C, or AB or AC or BC, or ABC (i.e., A and B and C). Furthermore, the word "exemplary" does not mean that the example described is preferred or better than other examples.

[0293] It should also be noted that in the system and method of the present disclosure, each component or each step can be decomposed and / or recombined. Such decomposition and / or recombination should be regarded as equivalent solutions of the present disclosure.

[0294] Various changes, substitutions, and modifications may be made to the technology described herein without departing from the teachings defined by the appended claims. Moreover, the scope of the claims of this disclosure is not limited to the specific aspects of the processes, machines, manufactures, compositions of things, means, methods, and actions described above. Currently existing or later developed processes, machines, manufactures, compositions of things, means, methods, or actions that perform substantially the same function or achieve substantially the same results as the corresponding aspects described herein may be utilized. Accordingly, the appended claims include within their scope such processes, machines, manufactures, compositions of things, means, methods, or actions.

[0295] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use the present disclosure. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects without departing from the scope of the present disclosure. Therefore, the present disclosure is not intended to be limited to the aspects shown herein, but rather to be accorded the widest scope consistent with the principles and novel features disclosed herein.

[0296] The above description has been provided for the purpose of illustration and description. In addition, this description is not intended to limit the embodiments of the present disclosure to the forms disclosed herein. Although a number of example aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, alterations, additions, and sub-combinations thereof.

Claims

1. A human-computer interaction method based on facial expression recognition, characterized in that: include: Pre-process the user's facial image collected in real time to determine the effective facial area; Extracting preset key point information in the effective facial area; Determining facial features based on the preset key point information, wherein the facial features include facial geometric features and facial texture features; Analyzing the facial features based on the constructed classification model to obtain user expression information; Determining an expression intensity level based on a preset expression library and the user's expression information; If the expression intensity level is less than a preset threshold, triggering a first strategy to perform multi-target mesh deformation adjustment on the target to be deformed; If the expression intensity level is not less than a preset threshold, triggering a second strategy to perform three-dimensional deformation adjustment on the target object to be deformed; If the expression intensity level is not less than a preset threshold, triggering a second strategy to perform three-dimensional deformation adjustment on the target object to be deformed includes: Creating a target Octree structure and injecting noise into the target Octree structure; Extracting key expression features from the user expression information, and converting the key expression features into a first vector; Converting a preset style of the target object to be deformed into a target vector, wherein the target vector includes style features and style requirements; Fusing the first vector with the target vector to obtain a conditional vector; Constructing an initial diffusion model; the initial diffusion model includes a neural network based on a 3D UNet structure; Injecting the conditional vectors at different levels of the neural network to obtain a first model; Training the first model based on a historical 3D expression dataset, and using the trained first model as a target diffusion model; gradually denoising the target Octree structure after noise injection using the target diffusion model to obtain target three-dimensional data; The target object to be deformed is subjected to three-dimensional deformation adjustment based on the target three-dimensional data.

2. The human-computer interaction method based on facial expression recognition according to claim 1, characterized in that: The preprocessing of the user's facial image collected in real time to determine the effective facial area includes: An image acquisition device based on a preset resolution captures a user's facial image in real time; Performing enhancement processing on the user's facial image using a preset enhancement strategy; The enhanced user facial image is processed based on the Viola-Jones algorithm to determine a valid facial area.

3. The human-computer interaction method based on facial expression recognition according to claim 2, characterized in that: The preset key point information includes eye specific point information, nose specific point information, mouth specific point information, and eyebrow specific point information; The facial geometric features include the degree of eye opening, the degree of upward movement of the mouth corners and the curvature of the eyebrows.

4. The human-computer interaction method based on facial expression recognition according to claim 1, characterized in that: The facial features are analyzed based on the constructed classification model to obtain user expression information, including: Build an initial learning model; Using a preset expression data set to train the initial learning model, and using the trained initial learning model as the classification model; The preset expression data set includes a data set of facial images of different expressions; Analyzing the facial features based on the classification model to obtain expression type and expression amplitude; The expression type and the expression amplitude constitute the user expression information.

5. The human-computer interaction method based on facial expression recognition according to claim 4, characterized in that: Determining the expression intensity level based on the preset expression library and the user expression information; Adding a regression head to the classification model to obtain an initial regression model; Generate synthetic expression images based on generative adversarial networks; Obtaining a preset expression library based on the preset expression data set and the synthesized expression image; Training the initial regression model based on the preset expression library, and recording the trained initial regression model as a target regression model; The user expression information is analyzed based on the target regression model to obtain an expression intensity value, and a corresponding expression intensity level is obtained based on a preset expression intensity information library.

6. The human-computer interaction method based on facial expression recognition according to claim 1, characterized in that: If the expression intensity level is less than a preset threshold, triggering a first strategy to perform multi-target mesh deformation adjustment on the target to be deformed includes: Processing the user expression information based on the constructed grid models to obtain geometric difference information, physical difference information, and aesthetic difference information respectively; Obtaining a target body deformation strategy based on the geometric difference information, the physical difference information, and the aesthetic difference information; The target object to be deformed is subjected to multi-target mesh deformation adjustment based on the target body deformation strategy.

7. The human-computer interaction method based on facial expression recognition according to claim 6, characterized in that: The obtaining of a target body deformation strategy based on the geometric difference information, the physical difference information, and the aesthetic difference information includes: extracting a geometric gradient, a physical gradient, and an aesthetic gradient based on the geometric difference information, the physical difference information, and the aesthetic difference information, respectively; fusing the geometric gradient, the physical gradient, and the aesthetic gradient based on an adaptive weight strategy to obtain a fusion score function; The fusion score function includes: F=w1*physical gradient+w2*geometric gradient+w3*aesthetic gradient; wherein w1 is the expression adaptation weight, w2 is the geometric adaptation weight, and w3 is the material adaptation weight; The mesh vertex gradients of the target object to be deformed are calculated based on the fusion score function; and the target body deformation strategy includes the mesh vertex gradients.

8. An interactive intelligent toy, characterized in that: It includes a toy body and a master control center arranged in the toy body; The master control center uses the human-computer interaction method based on expression recognition according to any one of claims 1 to 7 to control the intelligent interaction between the toy body and the user; The toy body is a target object to be transformed.

9. A robot, characterized in that: It includes a robot body and a master control center arranged in the robot body; The master control center uses the human-computer interaction method based on expression recognition according to any one of claims 1 to 7 to control the intelligent interaction between the robot body and the user; The robot body is a target object to be deformed.

Images