Multi-modal voice interaction method based on face recognition

Abstract

The invention relates to the technical field of face recognition, and discloses a multi-modal voice interaction method based on face recognition, which comprises the following steps of: 1, acquiring face features, and recording and numbering a plurality of face features; step 2, face analysis is carried out in combination with the face features, and a face score is calculated; 3, judging the face recognition accuracy according to the face scoring result; 4, extracting voice features of the crowd with low face recognition accuracy; step 5, combining the voice features to form a final voice feature, and matching the final voice feature with a pre-stored voice instruction; and step 6, sending a final instruction. According to the face recognition method, the face score is calculated through the face features, the face score is compared with the standard threshold value, the face information can be captured more comprehensively through the face recognition technology, and therefore the recognition accuracy is improved, and under the condition that the face recognition accuracy is low, judgment is carried out through the voice features, an instruction is sent out, and the recognition accuracy is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of face recognition, and in particular to a multimodal voice interaction method based on face recognition. Background Art

[0002] Face recognition technology is a technology that identifies people based on their facial features. It covers multiple fields such as image processing, pattern recognition and computer vision. Face recognition technology automatically detects and tracks faces by collecting images or video streams containing faces, and then performs facial recognition on the detected faces. This technology mainly relies on human facial feature information, including the shape, size and position relationship of organs such as eyes, nose and mouth, as well as the characteristics of facial contours. The research on face recognition technology began in the 1960s, when it was mainly based on matching methods based on geometric features. After entering the 1970s, with the development of computer technology, researchers began to use more measurement points to improve recognition accuracy. In the 1980s, the linear subspace discrimination model was introduced, which further improved face recognition technology. In the 1990s, researchers began to try to shift from two-dimensional images to three-dimensional data collection and recognition. In the 21st century, with the improvement of computer computing power and the advancement of camera technology, deep learning algorithms have achieved remarkable results in face recognition, especially the application of convolutional neural networks (CNN), which makes face recognition technology more accurate and efficient.

[0003] During the face recognition process, if the facial features are blocked or the light is dim, the recognition rate will drop significantly. At the same time, face recognition has certain requirements for facial posture. When the head posture of the recognizer changes, the angle and position of the facial features will change, thus affecting the recognition accuracy. Most current algorithms are mainly aimed at frontal and quasi-frontal facial images. The recognition rate will drop significantly for pitch or left and right face rotations. In order to solve the above problems, a multimodal voice interaction method is needed to improve the accuracy of face recognition. Summary of the invention

[0004] 1. Technical issues to be resolved

[0005] In view of the deficiencies in the prior art, the present invention provides a multimodal voice interaction method based on face recognition, which has the advantages of first calculating a face score by using face features and comparing the face score with a standard threshold, so that face recognition technology can capture facial information more comprehensively, thereby improving recognition accuracy. In the case of low face recognition accuracy, voice features are used to make judgments and issue instructions, thereby comprehensively improving recognition accuracy and security, etc.

[0006] (II) Technical solution

[0007] To achieve the above object, the present invention provides the following technical solution: a multimodal voice interaction method based on face recognition, comprising the following steps:

[0008] Step 1: Obtain facial features and record and number multiple facial features;

[0009] Step 2: Perform facial analysis based on facial features to calculate facial scores;

[0010] Step 3: Determine the accuracy of face recognition based on the face scoring result;

[0011] Step 4: Extracting speech features of people with low face recognition accuracy;

[0012] Step 5: Combine the voice features to form a final voice feature, and match the final voice feature with the pre-stored voice command;

[0013] Step 6: Issue the final instruction.

[0014] Preferably, the facial features include: geometric features, texture features, skin color features, expression features, and physiological features, wherein the geometric features represent the geometric shape and structure of the face, including the positions and distances of key points such as eyes, nose, mouth, and chin;

[0015] Texture features represent detailed information on the surface of a face, such as skin texture, spots, wrinkles, etc.

[0016] The skin color feature represents the facial skin color as well as lip and eye color of a person’s face;

[0017] Expression features represent the expression changes of human face, including danger, seriousness, frowning, etc.

[0018] Physiological features represent the physiological structural characteristics of the human face, including eye size, nose bridge height, etc.

[0019] Preferably, the record number of the facial feature is:

[0020] RLtz=JHtz, WLtz, FStz, BQtz, SLtz

[0021] Among them, RLtz represents individual facial features, JHtz represents geometric features, WLtz represents texture features, FStz represents skin color features, BQtz represents expression features, and SLtz represents physiological features.

[0022] Preferably, the calculation formula for the face score is:

[0023]

[0024] MBpf represents the facial score of a person, i is the feature index, which is a specific feature of a person's face, and ω i Represents the weight of the i-th specific feature in the individual face feature, RLtz iA Represents the feature vector of the current face to be recognized on the i-th specific feature, RLtz iB Represents the feature vector of the face stored in the system on the i-th specific feature, d is a function representing the distance metric, d(RLtz iA , RLtz iB ) represents the distance measure between the current face to be recognized and the face stored in the system in the i-th specific feature, G max，i is the maximum distance of the i-th specific feature, which is used to standardize the distance metric so that it falls within the range of [0, 1].

[0025] Preferably, when the face score MBpf is less than the face score standard threshold, it means that the current face fails and the face recognition accuracy is low.

[0026] Preferably, the voice features include timbre features, normal volume, and speaking speed features, wherein the timbre features are the timbre features of the individual, the normal volume is the volume of the individual's voice under normal circumstances, and the speaking speed features are the sound speed frequency of the individual's voice under normal circumstances.

[0027] Preferably, the record number of the voice feature is:

[0028] YYtz=YStz、ZCyl、YStz

[0029] Among them, YYtz represents the voice feature, YStz represents the timbre feature, ZCyl represents the normal volume, and YStz represents the speaking speed feature.

[0030] Preferably, the calculation formula of the final speech feature is:

[0031] ZZyy=YStz*ZCyl*YStz

[0032] Among them, ZZyy is the final speech feature.

[0033] Preferably, the pre-stored voice commands include: confirm, open, close, pause, continue, etc. The user can set the voice commands according to actual application needs.

[0034] Preferably, the final voice feature is matched and compared with the pre-stored voice instructions, and the corresponding matching voice instructions are extracted as the final instructions.

[0035] Compared with the prior art, the present invention provides a multimodal voice interaction method based on face recognition, which has the following beneficial effects:

[0036] 1. The present invention comprehensively determines facial features by combining geometric features, texture features, skin color features, expression features, and physiological features, calculates the facial score of the face based on the facial score, and compares the facial score of the face with the standard threshold, so that the face recognition technology can capture facial information more comprehensively, thereby improving the accuracy of recognition. For example, geometric features focus on the shape changes of the face, texture features analyze the texture changes on the surface of the facial skin, skin color features help to distinguish people of different races, and expression features and physiological features provide more information about individual emotions and health conditions. This multi-feature fusion method can improve the adaptability of the system to different environmental conditions and enhance recognition accuracy. At the same time, a single feature may perform poorly in the face of complex environments (such as lighting changes, facial occlusion, etc.), and multi-feature fusion can improve the robustness of the system under these conditions. Even when some features are interfered with, other features can still assist in completing the recognition task.

[0037] 2. The present invention integrates timbre, volume and speech speed features so that the system can more accurately identify voice commands. Different timbre, volume and speech speed help to distinguish the voices of different people or the voices of the same person in different situations. When users use voice commands, the volume and speech speed may change due to emotions, environment and other factors. After combining these features, the system can better adapt to the user's natural voice habits and improve user experience. In a noisy environment or when the user's emotions fluctuate, the simple timbre or volume features may not be sufficient to accurately identify voice commands. Combining multiple features can improve the robustness of the system, so that it can work stably in various complex environments. Combining timbre, volume and speech speed features can improve the accuracy and safety of recognition. Only when multiple features match, the system will perform the corresponding operation, thereby reducing the risk of misoperation. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 It is a step diagram of the method of the present invention. DETAILED DESCRIPTION

[0039] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0040] See also Figure 1 , a multimodal voice interaction method based on face recognition, comprising the following steps:

[0041] Step 1: Obtain facial features and record and number multiple facial features;

[0042] Facial features include: geometric features, texture features, skin color features, expression features, and physiological features. Geometric features represent the geometric shape and structure of the face, including key facial feature points, such as the corners of the eyes, the tip of the nose, the corners of the mouth, etc., the distances between feature points, such as the distance between the two eyes, the distance from the eyes to the nose, etc., the angle relationship between feature points, and the proportional relationship between the various parts of the face, such as the relative size of the eyes and the mouth, the local shape features of the facial features such as the eyes, nose, and mouth, and the face shape features, and the distribution of the facial features on the face, including their position, size, and relationship to each other;

[0043] Texture features represent detailed information on the surface of the face, including skin details, pores, wrinkles, spots and other microscopic features, skin microstructure, such as pore distribution, skin roughness, etc., texture patterns in specific facial areas, such as fine lines around the eyes, forehead wrinkles, etc., facial pigmentation, such as freckles, moles, etc., the distribution and density of eyebrows, beards and other hair, facial scars, tattoos or other permanent marks;

[0044] The skin color feature represents the facial skin color as well as lip and eye color of a person’s face;

[0045] Expression features represent the expression changes of human face, including danger, seriousness, frowning, etc.

[0046] Physiological features represent the physiological structural features of the face, including eye size, nose bridge height, etc.

[0047] The record number of the facial features is:

[0048] RLtz=JHtz, WLtz, FStz, BQtz, SLtz

[0049] Among them, RLtz represents individual face features, JHtz represents geometric features, WLtz represents texture features, FStz represents skin color features, BQtz represents expression features, and SLtz represents physiological features;

[0050] By combining geometric features, texture features, skin color features, expression features, and physiological features to comprehensively determine facial features, face recognition technology can capture facial information more comprehensively, thereby improving recognition accuracy. For example, geometric features focus on changes in the shape of the face, texture features analyze changes in the texture of the facial skin surface, skin color features help distinguish people of different races, and expression features and physiological features provide more information about individual emotions and health conditions. This multi-feature fusion method can improve the system's adaptability to different environmental conditions and improve recognition accuracy. At the same time, a single feature may perform poorly in the face of complex environments (such as lighting changes, facial occlusion, etc.), and multi-feature fusion can improve the system's robustness under these conditions. Even when some features are interfered with, other features can still assist in completing the recognition task.

[0051] Step 2: Perform facial analysis based on facial features to calculate facial scores;

[0052] The calculation formula for face score is:

[0053]

[0054] MBpf represents the facial score of a person, i is the feature index, which is a specific feature of a person's face, and ω i Represents the weight of the i-th specific feature in the individual face feature, RLtz iA Represents the feature vector of the current face to be recognized on the i-th specific feature, RLtz iB Represents the feature vector of the face stored in the system on the i-th specific feature, d is a function representing the distance metric, d(RLtz iA , RLtz iB ) represents the distance measure between the current face to be recognized and the face stored in the system in the i-th specific feature, G max，i is the maximum distance of the i-th specific feature, which is used to standardize the distance metric so that it falls within the range of [0, 1];

[0055] Step 3: Determine the accuracy of face recognition based on the face scoring result;

[0056] When the face score MBpf is less than the face score standard threshold, it means that the current face fails and the face recognition accuracy is low;

[0057] Face scoring can evaluate key factors such as the clarity, posture, and lighting of face images. High-quality face images usually have higher contrast, clearer features (such as eyes, nose, and mouth), and more uniform lighting conditions. These factors directly affect the performance of face recognition. Comparing the scoring results with the standard threshold can significantly improve the accuracy of recognition.

[0058] Step 4: Extracting speech features of people with low face recognition accuracy;

[0059] The voice features include timbre features, normal volume, and speech speed features. The timbre features are the timbre features of the individual, the normal volume is the volume of the individual's voice under normal circumstances, and the speech speed features are the sound speed frequency of the individual's voice under normal circumstances.

[0060] The record number of the voice feature is:

[0061] YYtz=YStz、ZCyl、YStz

[0062] Among them, YYtz represents the voice feature, YStz represents the timbre feature, ZCyl represents the normal volume, and YStz represents the speaking speed feature;

[0063] Step 5: Combine the voice features to form a final voice feature, and match the final voice feature with the pre-stored voice command;

[0064] The calculation formula of the final speech feature is:

[0065] ZZyy=YStz*ZCyl*YStz

[0066] Among them, ZZyy is the final speech feature;

[0067] Pre-stored voice commands include: confirm, open, close, pause, continue, etc. Users can set voice commands according to actual application needs;

[0068] The final voice features are matched and compared with the pre-stored voice commands, and the corresponding voice commands are extracted as the final commands;

[0069] By integrating the features of timbre, volume and speech speed, the system can more accurately identify voice commands. Different timbre, volume and speech speed help distinguish the voices of different people or the voices of the same person in different situations. When users use voice commands, the volume and speech speed may change due to factors such as emotions and environment. After combining these features, the system can better adapt to the user's natural voice habits and improve the user experience. In a noisy environment or when the user's emotions fluctuate, the simple timbre or volume features may not be enough to accurately identify voice commands. Combining multiple features can improve the robustness of the system, so that it can work stably in various complex environments. Combining timbre, volume and speech speed features can improve the accuracy and security of recognition. Only when multiple features match, the system will perform the corresponding operation, thereby reducing the risk of misoperation.

[0070] Step 6: Issue the final instruction.

[0071] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A multimodal voice interaction method based on face recognition, characterized in that: The following steps are involved: Step 1: Obtain facial features and record and number multiple facial features; Step 2: Perform facial analysis based on facial features to calculate facial scores; Step 3: Determine the accuracy of face recognition based on the face scoring result; Step 4: Extracting speech features of people with low face recognition accuracy; Step 5: Combine the voice features to form a final voice feature, and match the final voice feature with the pre-stored voice command; Step 6: Issue the final instruction.

2. The multimodal voice interaction method based on face recognition according to claim 1, characterized in that: The facial features include: geometric features, texture features, skin color features, expression features, and physiological features, wherein the geometric features represent the geometric shape and structure of the face, including the positions and distances of key points such as eyes, nose, mouth, and chin; Texture features represent detailed information on the surface of a face, such as skin texture, spots, wrinkles, etc. The skin color feature represents the facial skin color as well as lip and eye color of a person’s face; Expression features represent the expression changes of human face, including danger, seriousness, frowning, etc. Physiological features represent the physiological structural characteristics of the human face, including eye size, nose bridge height, etc.

3. The multimodal voice interaction method based on face recognition according to claim 2, characterized in that: The record number of the facial feature is: RLtz=JHtz, WLtz, FStz, BQtz, SLtz Among them, RLtz represents individual facial features, JHtz represents geometric features, WLtz represents texture features, FStz represents skin color features, BQtz represents expression features, and SLtz represents physiological features.

4. The multimodal voice interaction method based on face recognition according to claim 3, characterized in that: The calculation formula of the face score is: MBpf represents the facial score of a person, i is the feature index, which is a specific feature of a person's face, and ω i Represents the weight of the i-th specific feature in the individual face feature, RLtz iA Represents the feature vector of the current face to be recognized on the i-th specific feature, RLtz iB Represents the feature vector of the face stored in the system on the i-th specific feature, d is a function representing the distance metric, d(RLtz iA , RLtz iB ) represents the distance measure between the current face to be recognized and the face stored in the system in the i-th specific feature, G max，i is the maximum distance of the i-th specific feature, which is used to standardize the distance metric so that it falls in the range of [0,1].

5. The multimodal voice interaction method based on face recognition according to claim 4, characterized in that: When the face score MBpf is less than the face score standard threshold, it means that the current face fails and the face recognition accuracy is low.

6. The multimodal voice interaction method based on face recognition according to claim 1, characterized in that: The voice features include timbre features, normal volume, and speech speed features. The timbre features are the timbre features of the individual, the normal volume is the volume of the individual's voice under normal circumstances, and the speech speed features are the sound speed frequency of the individual's voice under normal circumstances.

7. The multimodal voice interaction method based on face recognition according to claim 6, characterized in that: The record number of the voice feature is: YYtz=YStz、ZCyl、YStz Among them, YYtz represents the voice feature, YStz represents the timbre feature, ZCyl represents the normal volume, and YStz represents the speaking speed feature.

8. The multimodal voice interaction method based on face recognition according to claim 7, characterized in that: The calculation formula of the final speech feature is: ZZYy=YStz*ZCyl*YStz Among them, ZZyy is the final speech feature.

9. The multimodal voice interaction method based on face recognition according to claim 8, characterized in that: The pre-stored voice commands include: confirm, open, close, pause, continue, etc. The user can set the voice commands according to actual application needs.

10. The multimodal voice interaction method based on face recognition according to claim 9, characterized in that: The final voice feature is matched and compared with the pre-stored voice instructions, and the corresponding matching voice instructions are extracted as the final instructions.

Images