Facial expression recognition method and medical robot

Abstract

The present disclosure provides a facial expression recognition method and a medical care robot, belonging to the field of image recognition technology. The method includes: extracting features from a first image based on an active appearance model to obtain a first expression feature; wherein the first image is a facial image of a target person; determining a target expression feature based on the perspective of the first image and the first expression feature, and inputting the target expression feature into a target random forest model to obtain an expression recognition result of the target person; wherein the target random forest model is a model trained with a first data set and a second data set, the data in the first data set being real expression feature data and corresponding expression recognition results, and the data in the second data set being expression feature data generated based on a generative adversarial network model and corresponding expression recognition results; and determining the working mode of the medical care robot based on the expression recognition result of the target person. The present disclosure can improve the accuracy and reliability of facial expression recognition.

Description

Technical Field

[0001] The present disclosure belongs to the field of image recognition technology, and more specifically, relates to a facial expression recognition method and a medical robot. Background Art

[0002] As the aging population accelerates, the integrated medical and nursing model is becoming increasingly important. Medical and nursing robots, as key auxiliary tools in this model, can provide comprehensive care and services for the elderly.

[0003] Traditional facial expression recognition relies on traditional machine learning algorithms. However, due to the lack of side datasets and the diversity of perspectives used when recognizing facial expressions, the accuracy and reliability of facial expression recognition are low.

[0004] Therefore, an accurate and reliable facial expression recognition method is needed. Summary of the Invention

[0005] The purpose of the present disclosure is to provide a facial expression recognition method and a medical robot to improve the accuracy and reliability of facial expression recognition.

[0006] A first aspect of the present disclosure provides a facial expression recognition method applied to a medical robot, comprising:

[0007] Extracting features from the first image based on the active appearance model to obtain a first expression feature; wherein the first image is a facial image of a target person;

[0008] Determining a target expression feature based on the perspective of the first image and the first expression feature, and inputting the target expression feature into a target random forest model to obtain an expression recognition result of the target person; wherein the target random forest model is a model trained with a first data set and a second data set, the data in the first data set are real expression feature data and corresponding expression recognition results, and the data in the second data set are expression feature data generated based on a generative adversarial network model and corresponding expression recognition results;

[0009] The working mode of the medical robot is determined based on the facial expression recognition results of the target person.

[0010] A second aspect of the present disclosure provides a facial expression recognition device, which is applied to a medical robot, including:

[0011] a feature extraction module, configured to extract features from a first image based on an active appearance model to obtain a first expression feature; wherein the first image is a facial image of a target person;

[0012] An expression recognition module is configured to determine a target expression feature based on the viewing angle of the first image and the first expression feature, and input the target expression feature into a target random forest model to obtain an expression recognition result of the target person; wherein the target random forest model is a model trained with a first data set and a second data set, the data in the first data set are real expression feature data and corresponding expression recognition results, and the data in the second data set are expression feature data and corresponding expression recognition results generated based on a generative adversarial network model;

[0013] The control module is used to determine the working mode of the medical robot based on the expression recognition results of the target person.

[0014] According to a third aspect of the embodiments of the present disclosure, a medical robot is provided, comprising a memory, a processor, and a computer program stored in the memory and running on the processor. When the processor executes the computer program, the steps of the above-mentioned facial expression recognition method are implemented.

[0015] According to a fourth aspect of the embodiments of the present disclosure, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the above-mentioned facial expression recognition method are implemented.

[0016] The beneficial effects of the facial expression recognition method and medical robot provided by the embodiments of the present disclosure are:

[0017] By considering the perspective of the first image and adjusting the facial expression features accordingly, the present disclosure can reduce recognition errors caused by perspective and enhance the robustness of the present disclosure. The target random forest model is dual-trained with real data and data generated by a generative adversarial network, which enhances its generalization ability and improves the accuracy and reliability of facial expression recognition. When the accuracy and reliability of facial expression recognition are high, it can enable the medical robot to more accurately judge the emotional state of the target person, thereby adjusting its working mode and improving the user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions in the embodiments of the present disclosure, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0019] Figure 1 A flowchart of a facial expression recognition method provided by one embodiment of the present disclosure;

[0020] Figure 2 A structural block diagram of a facial expression recognition device provided in one embodiment of the present disclosure;

[0021] Figure 3 This is a schematic block diagram of a medical care robot provided in one embodiment of the present disclosure. DETAILED DESCRIPTION

[0022] In the following description, specific details such as specific system structures and techniques are provided for purposes of illustration rather than limitation to facilitate a thorough understanding of the embodiments of the present disclosure. However, it will be apparent to those skilled in the art that the present disclosure may be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to avoid obscuring the description of the present disclosure with unnecessary detail.

[0023] In order to make the purpose, technical solutions and advantages of the present disclosure more clear, specific embodiments will be described below with reference to the accompanying drawings.

[0024] Please refer to Figure 1 , Figure 1 This is a flow chart of a facial expression recognition method provided in one embodiment of the present disclosure, which is applied to a medical robot, including:

[0025] S101: Extracting features from a first image based on an active appearance model to obtain a first expression feature; wherein the first image is a facial image of a target person.

[0026] In this embodiment, the active appearance model establishes a facial shape model and texture model through statistical analysis of a large number of facial images. The shape model describes the distribution and variation patterns of key facial features (such as the corners of the eyes, mouth, and nose tip); the texture model describes the grayscale or color information of the facial surface. When applied, the active appearance model searches for the shape and texture that best matches the model within the input facial image, thereby extracting features that reflect facial expression, posture, and other information. The first image refers to the facial image of the target person, which can be acquired using the camera equipped with the medical robot. The target person refers to the person to be identified, such as an elderly person in a nursing home or a patient undergoing rehabilitation in a hospital. The first expression feature refers to feature data that reflects the target person's current facial expression, such as the four corners of the eyes, the endpoints of the eyebrows, and the corners of the mouth.

[0027] S102: Determine the target expression feature based on the viewing angle of the first image and the first expression feature, and input the target expression feature into the target random forest model to obtain the expression recognition result of the target person; wherein, the target random forest model is a model obtained by training the first data set and the second data set, the data in the first data set is the real expression feature data and the corresponding expression recognition results, and the data in the second data set is the expression feature data and the corresponding expression recognition results generated based on the generative adversarial network model.

[0028] In this embodiment, the viewing angle of the first image can be determined based on certain features within the image, such as simple features like the number of eyes or ears. The viewing angle of the first image (e.g., frontal, side, etc.) can affect the completeness and accuracy of the facial expression features. Under different viewing angles, certain facial expression features may be obscured or distorted, resulting in missing information. Therefore, it is necessary to process the first expression features in conjunction with the viewing angle of the first image to obtain a more comprehensive and accurate representation of the facial expression features.

[0029] Specifically, determining the target expression feature based on the viewing angle of the first image and the first expression feature includes:

[0030] In response to the perspective of the first image being a frontal perspective, determining the first facial expression feature as a target facial expression feature;

[0031] In response to the perspective of the first image being a side perspective, a second expression feature is generated based on the first expression feature, and the first expression feature and the second expression feature are determined as target expression features; wherein the second expression feature corresponds to the frontal perspective of the first image.

[0032] In this embodiment, considering that if the first image is from a frontal perspective, the first expression feature generally better reflects the target person's expression, as the frontal perspective can more completely display facial expression information. In this case, the first expression feature can be directly determined as the target expression feature. However, in non-frontal perspectives, the first expression feature is incomplete and requires processing. For example, a second expression feature can be generated based on the first expression feature through a mapping or generation method (e.g., based on a pre-trained mapping relationship or generation model). The first and second expression features are then combined to form the target expression feature to supplement the missing expression information. The second expression feature is the frontal perspective expression feature corresponding to the first expression feature.

[0033] The target facial features can be determined through the preceding description. These features are then input into the target random forest model to obtain the target person's facial expression recognition results. Random forest is an ensemble learning method consisting of multiple decision trees. During training, each decision tree learns based on a different subset of features and samples. The final classification decision is ultimately made by combining the outputs of multiple decision trees (e.g., voting). In this disclosure, the target random forest model determines the target person's facial expression category based on the input target facial features.

[0034] The training data of the target random forest model are the first data set and the second data set. The first data set contains real expression feature data and corresponding expression recognition results. The data in the first data set is collected in actual scenes and has high authenticity and reliability. The second data set is expression feature data and corresponding expression recognition results generated based on the generative adversarial network model.

[0035] It's important to note that, because data on side-view expressions is relatively scarce in reality, the proportion of frontal-view facial expression feature data in the first dataset is greater than that of side-view facial expression feature data. The purpose of the generative adversarial network is to generate and expand side-view facial expression feature data. Therefore, the proportion of side-view facial expression feature data in the second dataset is greater than that of frontal-view facial expression feature data. That is, the proportion of side-view facial expression feature data in the first dataset is less than that of side-view facial expression feature data in the second dataset. The data in the second dataset may or may not include frontal-view facial expression feature data. That is, whether the generative adversarial network generates frontal-view facial expression feature data can be determined based on the amount of frontal-view facial expression feature data in the first dataset. If the amount is greater than a preset number, the generative adversarial network may not generate frontal-view facial expression feature data. If the amount is less than the preset number, the generative adversarial network may generate frontal-view facial expression feature data until the preset number or a certain value above the preset number is reached. However, excessive generation is not recommended, as it may lead to excessive learning during random forest model training, resulting in reduced training effectiveness.

[0036] S103: Determine the working mode of the medical robot based on the facial expression recognition result of the target person.

[0037] In this embodiment, considering that different expressions often reflect the different physical and mental states and needs of the target person, the medical robot needs to adjust its working mode according to these different situations. For example:

[0038] In response to the target person's facial expression recognition result being a positive emotion, the medical care robot's working mode is set to a companionship and entertainment mode;

[0039] In response to the target person's facial expression recognition result being negative, the medical care robot's working mode is determined to be a comforting and caring mode;

[0040] In response to the target person's facial expression recognition result being pain, the working mode of the medical robot is determined to be an emergency treatment mode.

[0041] In this embodiment, the companionship and entertainment mode can include playing lighthearted music, engaging in pleasant conversations with the target person, telling jokes, or recommending interesting videos. The comfort and care mode can include proactively communicating with the target person, offering comfort and encouragement, asking if they need help, or providing relaxation guidance, such as deep breathing exercises or meditation instruction, to help the target person alleviate negative emotions. The emergency treatment mode can include promptly notifying medical staff or relevant caregivers of any possible physical discomfort.

[0042] From the above, it can be concluded that the present disclosure can reduce recognition errors caused by perspective by considering the perspective of the first image and adjusting the expression features accordingly, thereby enhancing the robustness of the present disclosure. The target random forest model is dual-trained with real data and data generated by the generative adversarial network, which enhances its generalization ability and improves the accuracy and reliability of facial expression recognition. When the accuracy and reliability of facial expression recognition are high, the medical robot can more accurately judge the emotional state of the target person, thereby adjusting its working mode and improving the user experience.

[0043] As can be seen from the foregoing, the second facial expression feature is generated based on the first facial expression feature. Specifically, in one embodiment of the present disclosure, generating the second facial expression feature based on the first facial expression feature includes:

[0044] A second expression feature is generated based on the first expression feature and a first mapping relationship, where the first mapping relationship is determined based on a third data set, and data in the third data set includes multiple groups of expression feature data with the same expression but different perspectives.

[0045] In this embodiment, the data in the third data set are multiple groups of expression feature data with the same expression but different perspectives. By analyzing a large amount of side and front facial image data, the geometric transformation parameters between different feature points can be statistically calculated. Specifically, for the perspective conversion from side to front, it is necessary to determine the parameters such as the translation amount, rotation angle and scaling ratio of the feature points in the x, y, and z axis directions, and fit and optimize them through the least squares method, so that the feature point mapping based on these parameters can be consistent with the actual facial morphology to the greatest extent, and the first mapping relationship is obtained.

[0046] Alternatively, the second expression feature may be generated based on the neural network model. Specifically, generating the second expression feature based on the first expression feature includes:

[0047] A second facial expression feature is generated based on a first neural network model and a first facial expression feature, where the first neural network model is trained with facial expression feature data from a first number of side and front facial images.

[0048] From the above, it can be concluded that the present disclosure can generate a second expression feature based on the first expression feature by introducing a first mapping relationship. The first mapping relationship is obtained by analyzing multiple groups of expression feature data with the same expression but different perspectives in the third data set, which ensures the accuracy and reliability of the expression feature data of the front perspective generated by the present disclosure based on the side perspective, thereby improving the accuracy and reliability of the target random forest model, thereby achieving the improvement of the accuracy and reliability of facial expression recognition.

[0049] In one embodiment of the present disclosure, inputting the target expression feature into the target random forest model to obtain the expression recognition result of the target person includes:

[0050] In response to the perspective of the first image being a side perspective, inputting the first expression feature into the target random forest model to obtain a first voting result, and inputting the second expression feature into the target random forest model to obtain a second voting result;

[0051] The first voting result and the second voting result are weighted to determine the expression recognition result of the target person.

[0052] In this embodiment, considering that the perspective of the first image is a side perspective, and the expression feature data about the side perspective in the training data of the target random forest is generated based on the generative adversarial network, not only the first expression feature can be input into the target random forest model, but also the second expression feature can be input into the target random forest model to achieve better recognition effect.

[0053] The principle is: as mentioned above, the target random forest model is obtained by training the first data set and the second data set. The first data set contains more expression feature data from the frontal perspective, and the second data set contains more expression feature data from the side perspective. Therefore, the target random forest model can recognize both the expression feature data from the frontal perspective and the expression feature data from the side perspective.

[0054] The random forest model consists of multiple decision trees. Each decision tree classifies and judges the first facial expression based on its own learned rules and gives a voting result (i.e., a "vote" to determine whether the expression belongs to a certain expression category). The voting results of all decision trees are combined to obtain the first voting result, which represents the model's judgment of the target person's expression based on the side facial expression features.

[0055] Similarly, the second expression feature is input into the target random forest model, and the various decision trees in the model analyze and vote on it, and finally summarize the second voting result, which reflects the model's judgment of the target person's expression based on the frontal perspective expression feature.

[0056] The first voting result is based on facial features from the side view, which may be partially obscured or distorted, limiting its accuracy and reliability. The second voting result is based on generated facial features from the front view, which may contain errors. Therefore, we cannot simply use one voting result as the final expression recognition result; we need to consider both voting results comprehensively.

[0057] More deeply, considering that the perspective of the first image is a side perspective, and most of the training data under the side perspective is generated data, and the expression feature data of the front perspective is also generated in essence, the distribution of weights during weighted calculation needs to be set more accurately.

[0058] Therefore, in one embodiment of the present disclosure, before performing weighted calculation on the first voting result and the second voting result, the method further includes:

[0059] Determine a first expression recognition result based on the first voting result;

[0060] A generation weight is determined based on the first expression recognition result. The sum of the generation weight and the mapping weight is one. The generation weight is the weight corresponding to the first voting result, and the mapping weight is the weight corresponding to the second voting result.

[0061] In this embodiment, the weights of the first voting result and the second voting result should be related to their reliability. On this basis, considering that the second voting result is obtained based on the mapping relationship or the aforementioned first neural network model, the second voting result is more stable for different expressions, that is, the generation effect difference for different expressions is smaller.

[0062] For generative adversarial network models, the difficulty and effectiveness of generating different expressions vary. Simple expressions, such as smiling and frowning, have relatively simple and clear facial muscle movement patterns and intuitive features. After learning a large amount of relevant data, the generator can easily capture the key features of these expressions, thereby generating more realistic expression images, and the generation effect is usually good. However, complex expressions, such as contempt and disdain, involve subtle coordinated changes in multiple facial muscle groups, and the expressions of different individuals may vary significantly. The generator needs to learn more complex feature combinations and change patterns, which increases the difficulty of generation and may result in the generated expression images being deficient in details, naturalness, etc., resulting in relatively poor generation effect.

[0063] Therefore, the corresponding expression can be determined according to the first expression recognition result, and then the weight in the weighted calculation can be determined based on the generation effect of the expression.

[0064] From the above, it can be concluded that the present disclosure analyzes the generation effects of different expressions and determines the generation weights and mapping weights accordingly, so that the present disclosure can dynamically adjust according to the complexity and generation difficulty of different expressions, thereby improving the accuracy and reliability of the weighted calculation results, and thus improving the accuracy and reliability of facial expression recognition.

[0065] As can be seen from the foregoing, the weights corresponding to the first voting result and the second voting result should be determined based on the expression generation effect corresponding to the first expression recognition result. Specifically, determining the generation weight based on the first expression recognition result includes:

[0066] In response to the generation effect evaluation value being greater than the first evaluation value, increasing the reference value of the generation weight according to the first step to obtain the generation weight;

[0067] In response to the generation effect evaluation value being less than the second evaluation value, increasing the reference value of the mapping weight according to a second step length to obtain the mapping weight;

[0068] The generation effect evaluation value is an evaluation value of the generation effect of the expression corresponding to the first expression recognition result, and the first evaluation value is greater than the second evaluation value.

[0069] The first evaluation value and the second evaluation value may be set based on experience.

[0070] In this embodiment, the evaluation value of the generative adversarial network model for the generation effect of different expressions is calculated, and then the weight distribution is determined based on the evaluation value. That is, after the generative adversarial network model is generated, the following steps should be performed:

[0071] Calculate the mutual information between various types of expression feature data generated by the generative adversarial network model and their corresponding real expression feature data to obtain a first mutual information;

[0072] The first mutual information is used as the evaluation value of the generation effect of the expression.

[0073] Mutual information measures the correlation between two variables. The mutual information between the generated image and the real image is calculated. A higher mutual information indicates a stronger correlation between the generated image and the real image, and the generated expression is closer to the real one.

[0074] If the generation effect evaluation value is greater than the first evaluation value, it indicates that the profile expression features generated by the generative adversarial network model are effective and more reliable in expression recognition. At this point, the reference value of the generation weight is increased according to the first step length to obtain the final generation weight. At the same time, to make the sum of the weights equal to 1, the reference value of the mapping weight can be lowered according to the same first step length to obtain the final mapping weight. This gives greater weight to the first voting result, making it have a greater impact on the final expression recognition results.

[0075] If the generation effect evaluation value is less than the second evaluation value, the profile expression features generated by the generative adversarial network model are poorly effective. In this case, the reference value of the mapping weights is increased according to the second step size to obtain the final mapping weights. To achieve a sum of the weights of 1, the reference value of the generation weights can be decreased according to the same second step size to obtain the final generation weights. This ensures that the frontal perspective expression features generated based on the mapping relationship (the second voting result) have a greater impact on the final expression recognition. The first step size, second step size, reference values ​​for the mapping weights, and reference values ​​for the generation weights can all be determined based on parameter settings for solving similar problems or experimental data.

[0076] It should be noted that if the generation effect evaluation value is between the first evaluation value and the second evaluation value, no adjustment is required, that is, in response to the generation effect evaluation value being greater than or equal to the second evaluation value, and the generation effect evaluation value being less than or equal to the first evaluation value, the reference value of the generation weight is determined as the generation weight, and the reference value of the mapping weight is determined as the mapping weight.

[0077] The first and second voting results are weighted together to determine which expression has the highest number of votes. The expression with the highest number of votes is then output as the target expression recognition result. The process for determining the first expression recognition result based on the first voting result is the same as the first voting result, with the expression with the highest number of votes being output as the result.

[0078] In one embodiment of the present disclosure, in addition to the above-mentioned step size adjustment method, the generated weight can also be determined based on the first formula, which is: ,in, To generate weights, is the minimum value of the generated weight, is the maximum value of the generated weight, To generate effect evaluation values, To adjust the exponential decay parameters, The bigger, When younger The faster the decline, the Larger The faster it rises, is the natural base. 、 and Can be set based on experience.

[0079] The logic of the first formula is, Indicates the adjustable range of generated weights, When the exponential function When smaller, Approaching 1, Approaching 0, the value of the adjustment part is small, and the generated weight is close to , which means that the generation weight is low when the generation effect is poor; when When it increases, Approaching 0, Approaches 1, and the value of the adjustment part approaches , generating weights close to , which means that the generation weight is higher when the generation effect is good. Normalize the entire adjustment part to ensure that When the adjustment part can reach , so that the generated weight Can reach maximum value .

[0080] From the above, it can be concluded that the present disclosure can objectively measure the quality of generated expressions by calculating the mutual information between the expression feature data generated by the generative adversarial network model and the real expression feature data as the evaluation value of the generation effect. The generation weight and mapping weight are dynamically adjusted according to the generation effect evaluation value, so that when the quality of the generated expression is high, the first voting result is given a greater weight, thereby improving the accuracy and credibility of expression recognition. Conversely, when the quality of the generated expression is low, the weight of the mapping weight is increased to reduce the recognition error caused by poor generation effect, thereby improving the accuracy and reliability of facial expression recognition.

[0081] A facial expression recognition method corresponding to the above embodiment, Figure 2 This is a structural block diagram of a facial expression recognition device provided by an embodiment of the present disclosure. For ease of explanation, only the parts related to the embodiment of the present disclosure are shown. Figure 2 The facial expression recognition device 20 is applied to a medical robot and includes: a feature extraction module 21, an expression recognition module 22 and a control module 23.

[0082] The feature extraction module 21 is configured to extract features from the first image based on the active appearance model to obtain a first expression feature; wherein the first image is a facial image of a target person;

[0083] An expression recognition module 22 is configured to determine a target expression feature based on the viewing angle of the first image and the first expression feature, and input the target expression feature into a target random forest model to obtain an expression recognition result of the target person; wherein the target random forest model is a model trained with a first data set and a second data set, the data in the first data set are real expression feature data and corresponding expression recognition results, and the data in the second data set are expression feature data and corresponding expression recognition results generated based on a generative adversarial network model;

[0084] The control module 23 is used to determine the working mode of the medical robot based on the expression recognition result of the target person.

[0085] In one embodiment of the present disclosure, the expression recognition module 22 is specifically configured to determine the first expression feature as the target expression feature in response to the perspective of the first image being a frontal perspective;

[0086] In response to the perspective of the first image being a side perspective, a second expression feature is generated based on the first expression feature, and the first expression feature and the second expression feature are determined as target expression features; wherein the second expression feature corresponds to the frontal perspective of the first image.

[0087] In one embodiment of the present disclosure, the expression recognition module 22 is further configured to, in response to the perspective of the first image being a side perspective, input the first expression feature into the target random forest model to obtain a first voting result, and input the second expression feature into the target random forest model to obtain a second voting result;

[0088] The first voting result and the second voting result are weighted to determine the expression recognition result of the target person.

[0089] In one embodiment of the present disclosure, a facial expression recognition apparatus 20 further includes: a weight determination module for determining a first expression recognition result based on the first voting result;

[0090] A generation weight is determined based on the first expression recognition result. The sum of the generation weight and the mapping weight is one. The generation weight is the weight corresponding to the first voting result, and the mapping weight is the weight corresponding to the second voting result.

[0091] In one embodiment of the present disclosure, the weight determination module is specifically configured to, in response to the generation effect evaluation value being greater than the first evaluation value, increase the reference value of the generation weight according to the first step to obtain the generation weight;

[0092] In response to the generation effect evaluation value being less than the second evaluation value, increasing the reference value of the mapping weight according to a second step length to obtain the mapping weight;

[0093] The generation effect evaluation value is an evaluation value of the generation effect of the expression corresponding to the first expression recognition result, and the first evaluation value is greater than the second evaluation value.

[0094] In one embodiment of the present disclosure, the expression recognition module 22 is further used to generate a second expression feature based on the first expression feature and the first mapping relationship, the first mapping relationship is determined based on a third data set, and the data in the third data set includes multiple groups of expression feature data with the same expression but different perspectives.

[0095] In one embodiment of the present disclosure, the control module 23 is specifically configured to set the working mode of the medical care robot to the companionship and entertainment mode in response to the expression recognition result of the target person being a positive emotion;

[0096] In response to the target person's facial expression recognition result being negative, the medical care robot's working mode is determined to be a comforting and caring mode;

[0097] In response to the target person's facial expression recognition result being pain, the working mode of the medical robot is determined to be an emergency treatment mode.

[0098] See also Figure 3 , Figure 3 This is a schematic block diagram of a medical robot provided by an embodiment of the present disclosure. Figure 3 The medical robot 300 in the embodiment shown may include: one or more processors 301, one or more input devices 302, one or more output devices 303 and one or more memories 304. The processors 301, input devices 302, output devices 303 and memories 304 communicate with each other via a communication bus 305. The memory 304 is used to store computer programs, which include program instructions. The processor 301 is used to execute the program instructions stored in the memory 304. The processor 301 is configured to call the program instructions to execute the functions of the modules / units in the above-mentioned device embodiments, such as Figure 2 The functions of modules 21 to 23 are shown.

[0099] It should be understood that in the embodiments of the present disclosure, the processor 301 may be a central processing unit (CPU), or may be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0100] The input device 302 may include a touchpad, a fingerprint collection sensor (for collecting user fingerprint information and fingerprint direction information), a microphone, etc. The output device 303 may include a display (LCD, etc.), a speaker, etc.

[0101] The memory 304 may include a read-only memory and a random access memory, and provides instructions and data to the processor 301. A portion of the memory 304 may also include a non-volatile random access memory. For example, the memory 304 may also store device type information.

[0102] In a specific implementation, the processor 301, input device 302, and output device 303 described in the embodiments of the present disclosure can execute the implementation methods described in the first and second embodiments of a facial expression recognition method provided in the embodiments of the present disclosure, and can also execute the implementation methods of the medical care robot described in the embodiments of the present disclosure, which will not be repeated here.

[0103] In another embodiment of the present disclosure, a computer-readable storage medium is provided. The computer-readable storage medium stores a computer program. The computer program includes program instructions. When the program instructions are executed by a processor, all or part of the process of the method in the above embodiment is implemented. The computer program can also be used to instruct related hardware to complete the process. The computer program can be stored in a computer-readable storage medium. When the computer program is executed by the processor, the steps of each of the above method embodiments are implemented. The computer program includes computer program code, which can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium can include: any entity or device capable of carrying computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signal, telecommunication signal and software distribution medium.

[0104] The computer-readable storage medium can be the internal storage unit of the medical care robot described in any of the aforementioned embodiments, such as the robot's hard drive or memory. The computer-readable storage medium can also be an external storage device of the medical care robot, such as a plug-in hard drive, a Smart Media Card (SMC), a Secure Digital (SD) card, a flash memory card, etc. Furthermore, the computer-readable storage medium can include both the internal storage unit of the medical care robot and an external storage device. The computer-readable storage medium is used to store computer programs and other programs and data required by the medical care robot. The computer-readable storage medium can also be used to temporarily store data that has been output or is about to be output.

[0105] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the above description has generally described the composition and steps of each example according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this disclosure.

[0106] Those skilled in the art will clearly understand that, for the sake of convenience and brevity of description, the specific working processes of the medical robots and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0107] In the several embodiments provided in this application, it should be understood that the disclosed medical care robots and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of units is only a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces or units, or it can be an electrical, mechanical or other form of connection.

[0108] Units described as separate components may or may not be physically separate, and components shown as units may or may not be physical units, i.e., they may be located in one place or distributed across multiple network units. Some or all of these units may be selected based on actual needs to achieve the objectives of the embodiments of the present disclosure.

[0109] In addition, the functional units in the various embodiments of the present disclosure may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0110] The above are only specific embodiments of the present disclosure, but the scope of protection of the present disclosure is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or replacements within the technical scope disclosed in this disclosure, and such modifications or replacements should be included in the scope of protection of the present disclosure. Therefore, the scope of protection of the present disclosure should be based on the scope of protection of the claims.

Claims

1. A facial expression recognition method, characterized in that: Applied to medical robots, including: Extracting features from a first image based on an active appearance model to obtain a first expression feature; wherein the first image is a facial image of a target person; Determining a target expression feature based on the viewing angle of the first image and the first expression feature, and inputting the target expression feature into a target random forest model to obtain an expression recognition result of the target person; The target random forest model is a model trained with a first data set and a second data set, wherein the data in the first data set are real facial expression feature data and corresponding facial expression recognition results, and the data in the second data set are facial expression feature data and corresponding facial expression recognition results generated based on a generative adversarial network model; Determining the working mode of the medical robot based on the expression recognition result of the target person; The determining of a target expression feature based on the viewing angle of the first image and the first expression feature includes: In response to the perspective of the first image being a frontal perspective, determining the first facial expression feature as a target facial expression feature; In response to the perspective of the first image being a side perspective, generating a second expression feature based on the first expression feature, and determining the first expression feature and the second expression feature as target expression features; wherein the second expression feature corresponds to a frontal perspective of the first image; Inputting the target facial expression features into a target random forest model to obtain the facial expression recognition result of the target person includes: In response to the perspective of the first image being a side perspective, inputting the first expression feature into a target random forest model to obtain a first voting result, and inputting the second expression feature into the target random forest model to obtain a second voting result; Performing a weighted calculation on the first voting result and the second voting result to determine an expression recognition result of the target person; Before performing weighted calculation on the first voting result and the second voting result, the method further includes: Determine a first expression recognition result based on the first voting result; Determine a generation weight based on the first expression recognition result, where the sum of the generation weight and the mapping weight is one, the generation weight being the weight corresponding to the first voting result, and the mapping weight being the weight corresponding to the second voting result; Calculate the mutual information between various types of expression feature data generated by the generative adversarial network model and their corresponding real expression feature data to obtain a first mutual information; The first mutual information is used as the evaluation value of the generation effect of the expression; The determining a generation weight based on the first expression recognition result includes: The generation weight is determined based on a first formula, wherein the first formula is: ,in, To generate weights, is the minimum value of the generated weight, is the maximum value of the generated weight, To generate effect evaluation values, To adjust the exponential decay parameters, is the natural base.

2. A facial expression recognition method as claimed in claim 1, characterized in that, Generating a second facial expression feature based on the first facial expression feature includes: A second expression feature is generated based on the first expression feature and a first mapping relationship, where the first mapping relationship is determined based on a third data set, and the data in the third data set are multiple groups of expression feature data with the same expression but different perspectives.

3. A facial expression recognition method as claimed in claim 1, characterized in that, The determining of the working mode of the medical robot based on the expression recognition result of the target person includes: In response to the target person's facial expression recognition result being a positive emotion, setting the working mode of the medical care robot to a companionship and entertainment mode; In response to the target person's facial expression recognition result being a negative emotion, determining the working mode of the medical care robot to be a comforting and caring mode; In response to the expression recognition result of the target person being in pain, the working mode of the medical robot is determined to be an emergency treatment mode.

4. A facial expression recognition device, characterized in that: Applied to medical robots, including: a feature extraction module, configured to extract features from a first image based on an active appearance model to obtain a first expression feature; wherein the first image is a facial image of a target person; an expression recognition module, configured to determine a target expression feature based on the viewing angle of the first image and the first expression feature, and input the target expression feature into a target random forest model to obtain an expression recognition result of the target person; wherein the target random forest model is a model trained with a first data set and a second data set, the data in the first data set is real expression feature data and corresponding expression recognition results, and the data in the second data set is expression feature data and corresponding expression recognition results generated based on a generative adversarial network model; A control module, configured to determine a working mode of the medical robot based on the expression recognition result of the target person; an expression recognition module, specifically configured to determine the first expression feature as a target expression feature in response to the perspective of the first image being a frontal perspective; In response to the perspective of the first image being a side perspective, generating a second expression feature based on the first expression feature, and determining the first expression feature and the second expression feature as target expression features; wherein the second expression feature corresponds to the frontal perspective of the first image; The expression recognition module is further configured to input the first expression feature into the target random forest model to obtain a first voting result in response to the perspective of the first image being a side perspective, and to input the second expression feature into the target random forest model to obtain a second voting result; Perform weighted calculation on the first voting result and the second voting result to determine the expression recognition result of the target person; A weight determination module, configured to determine a first expression recognition result based on the first voting result; Determine a generation weight based on the first expression recognition result, where the sum of the generation weight and the mapping weight is one, the generation weight is the weight corresponding to the first voting result, and the mapping weight is the weight corresponding to the second voting result; Calculate the mutual information between various types of expression feature data generated by the generative adversarial network model and their corresponding real expression feature data to obtain a first mutual information; The first mutual information is used as the evaluation value of the generation effect of the expression; The determining a generation weight based on the first expression recognition result includes: The generation weight is determined based on a first formula, wherein the first formula is: ,in, To generate weights, is the minimum value of the generated weight, is the maximum value of the generated weight, To generate effect evaluation values, To adjust the exponential decay parameters, is the natural base.

5. A medical robot comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 3 are implemented.

6. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 3 are implemented.

Images