A user emotion recognition method, system and device based on multi-sensor technology

Abstract

The present invention discloses a user emotion recognition method, system and equipment based on multi-sensor technology, which relate to the technical field of emotion recognition; it comprises a data acquisition module, a data analysis module, a feature screening module, an emotion recognition module, a comprehensive analysis module and a feedback processing module: the data acquisition module is used to collect user emotion data authenticated by the user and collect user emotion data in real time, and set it as standard emotion data, the user emotion data comprises facial picture data and voice data of the user in different emotions, and the technical key points are: analyzing the emotion data authenticated by the user, and the subsequent comparison is also based on the user's data, therefore, the system screens emotion characteristics based on the user's personal data, so that the accuracy of emotion recognition is very high, the accuracy of emotion recognition is relatively high, the use effect is good, and it has good application prospects.

Description

Technical Field

[0001] The present invention relates to the technical field of emotion recognition, and in particular to a method, system and device for user emotion recognition based on multi-sensor technology. Background Art

[0002] Electronic pets are electronic toys made of electronic components (hardware and software). Now they also refer to specific programs running on computers or Internet servers. Electronic pets have the same life characteristics as real pets. They are popular among people because of their easy feeding and portability.

[0003] With the development of science and technology, the technology in various industries has been innovated, among which electronic pets have gradually risen to mechanical pets. Mechanical pets are pets made by machines. They can imitate real pets, have realistic appearance and behavior, and even have some functions that surpass real pets. With the development of science and technology, the types of mechanical pets are becoming more and more abundant, from small pets to large pets, from land to underwater.

[0004] The rise of mechanical pets is due to the advancement of science and technology. On the one hand, the continuous improvement of mechanical manufacturing technology makes the appearance and behavior of mechanical pets more and more realistic; on the other hand, the rapid development of artificial intelligence technology enables mechanical pets to better imitate the behavior of real pets, and even to conduct some self-learning, so as to better accompany people.

[0005] However, current mechanical pets mainly rely on people to adjust the companionship mode themselves. They cannot automatically adjust according to the user's emotions and are not intelligent enough. Therefore, people have invented some systems that can recognize user emotions.

[0006] The existing patent authorization announcement number is CN113506586B, and the patent name is a method and system for user emotion recognition. The invention patent records that the method for user emotion recognition includes: obtaining voice data, and extracting voice features based on the voice data; converting the voice data to obtain text data, and extracting text features based on the text data; inputting voice features and text features into a user emotion recognition model, and outputting the user's emotion label, wherein in the user emotion recognition model: using a convolutional neural network to characterize voice features to obtain a first feature, using a long short-term memory network to characterize the first feature to obtain a second feature, using a convolutional neural network to characterize text features to obtain a third feature, using a long short-term memory network to characterize the third feature to obtain a fourth feature, fully connecting the second feature and the fourth feature to determine the user's emotion label. Through this application, the problem of low accuracy of user emotion recognition in the related art is solved, which is conducive to improving the accuracy of user emotion recognition;

[0007] The existing patent authorization announcement number is CN113380271B, and the patent name is the invention patent of emotion recognition method, system, device and medium. It records that it includes: obtaining the voice data and face data of the user to be tested; using voice recognition technology to process the voice data to generate text data; analyzing the text data to obtain the emotional characteristics of the user to be tested, extracting the tone characteristics of the user to be tested in the voice data, and extracting the expression characteristics of the user to be tested in the face data; constructing an emotion recognition model based on a distilled neural network to identify the emotions of the user to be tested; inputting the emotional characteristics, tone characteristics and expression characteristics into the emotion recognition model for identification, and obtaining the emotion category of the user to be tested. Compared with the existing emotion recognition based solely on the user's text content, the present invention recognizes from multiple dimensions such as tone, expression and emotion, which greatly improves the accuracy of emotion recognition;

[0008] The solutions recorded in the above patents have certain drawbacks. The solutions recorded in the above patents are mainly based on sound data. One solution is complete detection of speech, and the other is a combination of image recognition and speech to determine which specific user is speaking. They are suitable for scenarios where multiple people are applicable, but they are all achieved through speech comparison under different emotions. However, there are major defects in actual use. For people with strong control, the fluctuation of their speech characteristics is very small, and for people with weak control, the amplitude of their emotions changes is very large, so the characteristics of their speech change greatly, and the speech is usually longer. Therefore, the efficiency of emotion recognition is not high and does not meet people's requirements. To this end, we have developed a user emotion recognition method, system and device based on multi-sensor technology. Summary of the invention

[0009] 1. Technical issues to be resolved

[0010] In view of the shortcomings of the prior art, the present invention provides a user emotion recognition method, system and device based on multi-sensor technology, which analyzes the emotion data recognized by the user, and the subsequent comparison is also based on the user's data. Therefore, the system screens emotional characteristics based on the user's personal data, so that the accuracy of emotion recognition is very high, the accuracy of emotion recognition is relatively high, the use effect is good, and it has good application prospects, which solves the problems raised in the background technology.

[0011] (II) Technical solution

[0012] To achieve the above objectives, the present invention is implemented through the following technical solutions:

[0013] A user emotion recognition system based on multi-sensor technology includes a data acquisition module, a data analysis module, a feature screening module, an emotion recognition module, a comprehensive analysis module and a feedback processing module:

[0014] Data collection module: used to collect user emotional data authenticated by the user and collect user emotional data in real time, and set it as standard emotional data. User emotional data includes facial image data and voice data of users in different emotions;

[0015] Data analysis module: used to classify standard emotion data, assign corresponding emotion levels according to the process of emotion fluctuation, and refine standard emotion data to obtain emotion feature types;

[0016] Feature screening module: used to evaluate emotional features, obtain the feature data of the maximum and minimum emotional levels of the emotional feature types corresponding to the facial images under different emotions, calculate the feature change, and compare the feature change with the feature setting value, delete the emotional features below the feature setting value, obtain the categories of the remaining emotional features after deletion, and extract the feature data of the remaining emotional features under different emotions and different emotional levels to form a feature level calculation set, and then screen the emotional features in the feature level calculation set for proximity, summarize the screened emotional features and the feature data corresponding to the emotional features, and obtain a preliminary emotional recognition feature set;

[0017] Emotion recognition module: used to extract the user emotion data collected in real time, obtain the real-time feature data corresponding to the emotion features recorded in the emotion preliminary recognition feature set, compare the feature data in the emotion preliminary recognition feature set with the real-time feature data, perform similarity analysis, and obtain the user emotion category;

[0018] Comprehensive analysis module: used to obtain user emotion categories, and perform similarity analysis between the user emotion data collected in real time and the feature data of each emotion feature recorded in the feature level calculation set, and summarize and calculate the obtained similarities to obtain the user emotion level;

[0019] Feedback processing module: used to obtain the user's emotion category and user emotion level, and feed back the results to the robot pet. The robot pet executes the corresponding companionship strategy according to the feedback results.

[0020] Furthermore, the types of emotional features include eyebrow data, eye data, nose data, mouth data analyzed from facial image data, amplitude data, speech rate data, spectrum data, resonance peak parameters extracted from speech data, and text data from speech recognition.

[0021] Further, extracting the user's emotion data to obtain feature data corresponding to each emotion feature includes extracting the facial image data in the standard emotion data and extracting the voice data;

[0022] The steps to extract facial image data from standard emotion data are as follows:

[0023] Obtain facial recognition feature points used in facial recognition technology;

[0024] Filter the facial recognition feature points and remove the feature points outside the eyebrows, eyes, nose and mouth areas;

[0025] The screened feature points are classified and summarized to obtain eyebrow point set, eye point set, nose point set and mouth point set;

[0026] Use curves to connect the classified and summarized feature points, analyze the slope, curvature and curvature radius of the curve, and the analysis results constitute feature data;

[0027] The steps to extract voice data are as follows:

[0028] Performing audio analysis and spectrum analysis on the acquired speech data to obtain amplitude data and spectrum data;

[0029] The speech data is analyzed and processed by using the cepstrum method to extract the formant parameters;

[0030] Using an algorithm based on a statistical model to recognize speech data, and obtaining text data from speech recognition;

[0031] The duration of the voice data and the total number of characters in the text data obtained through voice recognition are obtained, and the speech rate data is calculated using the total number of characters in the text data obtained through voice recognition / the duration of the voice data.

[0032] Furthermore, the steps of obtaining the difference data between the maximum and normal emotional levels of each category and calculating the feature change are as follows:

[0033] Obtaining the emotion feature type data when the set emotion level is the maximum and the feature data of different emotion classifications when the emotion level is the minimum;

[0034] Analyze the feature data, calculate the difference ratio of each data, and then calculate the feature change amount based on the difference ratio;

[0035] ;

[0036] In the formula, is the characteristic variation of the emotional feature, n is the number of data types for identifying the emotional feature, is the i-th item of data when the emotion level is the maximum, is the i-th item of data when the emotion level is the minimum, is the weight coefficient corresponding to the i-th data type;

[0037] The calculation formula of the characteristic setting value TZs is: .

[0038] Furthermore, the steps of screening the degree of proximity of the emotional features in the feature level calculation set are as follows:

[0039] When the emotion level is the highest, the difference between the emotion data of this emotion classification and the emotion data of any other emotion classification is calculated one by one. The specific formula is as follows:

[0040] ;

[0041] In the formula, is the difference between the emotion data of this emotion classification and the emotion data of any other emotion classification, is the i-th item of data of any other emotion when the emotion level is the maximum, is the i-th item of data of any other emotion when the emotion level is the minimum;

[0042] Summarize the calculated L groups of difference values, and compare the summarized difference values ​​with the set standard difference values ​​one by one;

[0043] Get the number of difference values ​​k that are less than the set standard difference value, and calculate the degree of closeness PROx. The specific calculation formula is as follows:

[0044] ;

[0045] Determine the interval to which the proximity degree PROx belongs, which includes the proximity interval (0, A) and the identification interval If the proximity degree PROx is in the proximity interval (0, A), the emotional feature is removed. If the proximity degree PROx is in the identification interval If the emotion feature is in the range of 0 to 1, then the emotion feature is retained.

[0046] Furthermore, the steps of comparing the feature data in the preliminary emotion recognition feature set with the real-time feature data and performing similarity analysis are as follows:

[0047] Extract C emotion features extracted from facial image data from the emotion preliminary recognition feature set, and retrieve feature data of the C emotion features at different emotion levels to form a feature comparison set of different emotions;

[0048] Randomly extract K pieces of face image data collected in real time and extract K groups of data;

[0049] K groups of data are compared with feature comparison sets of different emotions one by one, and the similarity with different emotions is calculated. After each group of data is compared with the feature comparison sets of different emotions one by one, the maximum similarity value is extracted, thereby obtaining the maximum similarity value of K groups;

[0050] If the emotions corresponding to the maximum values ​​of K groups of similarities are the same, the user's emotion category is that emotion.

[0051] Furthermore, the formula for calculating the similarity between real-time emotion data and different emotions is as follows:

[0052] ;

[0053] In the formula, SIM is the calculated similarity of different emotions, is the similarity of the i-th emotional feature, is the weight coefficient of the i-th emotional feature, is the jth data in the i-th emotion feature in the real-time collected data, is the jth data item in the i-th emotion feature in the feature comparison set, is the maximum difference of the j-th data in the set i-th emotion feature.

[0054] Furthermore, if the emotions corresponding to the maximum values ​​of the K groups of similarities are different or the emotion data collected in real time does not contain facial image data, a voice data comparison is performed at this time, and the voice data comparison includes comparing the text data obtained by voice recognition with the text library recorded in the standard emotion data, comparing the waveform corresponding to the amplitude data with the amplitude waveforms corresponding to different emotions in the standard emotion data, and comparing the waveform corresponding to the spectrum data with the spectrum waveforms corresponding to different emotions in the standard emotion data. The maximum value of the similarity obtained by each comparison is taken to determine whether the emotions corresponding to the maximum value of the similarity are the same;

[0055] If the emotions corresponding to the three groups of maximum similarities are the same, the user's emotion category is that emotion;

[0056] If the emotions corresponding to the three groups of maximum similarity values ​​are different, a speech rate similarity comparison is added to determine the types of emotions corresponding to the four groups of maximum similarity values. If there is only one emotion mode at this time, the user emotion category is the mode emotion. If there are two emotion modes at this time, a resonance peak similarity comparison is added to transmit the emotion corresponding to the resonance peak similarity comparison result.

[0057] Furthermore, the steps of performing similarity analysis on the feature data collected in real time and the feature data of each emotion feature recorded in the feature level calculation set are as follows:

[0058] A facial image is sampled at every T time interval, a total of G facial images are obtained, and G groups of data are extracted from the G facial images;

[0059] The extracted G group data are compared with the feature data of different emotion levels corresponding to the user emotion categories in the feature level calculation set;

[0060] The formula for calculating the similarity with different emotion level feature data is as follows;

[0061] ;

[0062] In the formula, is the similarity of different emotion level feature data, h is the number of types of emotion features recorded in the feature level calculation set, is the similarity of the i-th emotional feature, is the j-th data in the i-th emotional feature corresponding to the emotional level;

[0063] Get the emotion level corresponding to the maximum similarity value of group G data, and calculate the average emotion level , where is the average emotional level, is the emotion level corresponding to the maximum similarity value of the i-th group of data;

[0064] The output user sentiment level is .

[0065] Furthermore, a user emotion recognition method based on multi-sensor technology includes the following steps:

[0066] Collect user emotion data authenticated by the user and collect user emotion data in real time, and set it as standard emotion data, the user emotion data includes facial image data and voice data of the user in different emotions;

[0067] Classify the standard emotion data, assign corresponding emotion levels according to the process of emotion fluctuation, and refine the standard emotion data to obtain emotion feature types;

[0068] Emotional features are evaluated, and feature data of the maximum and minimum emotion levels of the emotion feature types corresponding to the facial images under different emotions are obtained, and the feature change amount is calculated, and the feature change amount is compared with the feature setting value, and the emotion features below the feature setting value are deleted, and the categories of the remaining emotion features after deletion are obtained, and the feature data of the remaining emotion features under different emotions and different emotion levels are extracted to form a feature level calculation set, and then the emotion features in the feature level calculation set are screened for proximity, and the screened emotion features and the feature data corresponding to the emotion features are summarized to obtain a preliminary emotion recognition feature set;

[0069] Extracting the user emotion data collected in real time to obtain the real-time feature data corresponding to the emotion features recorded in the emotion preliminary identification feature set, comparing the feature data in the emotion preliminary identification feature set with the real-time feature data, performing similarity analysis, and obtaining the user emotion category;

[0070] Perform similarity analysis on the feature data collected in real time and the feature data of each emotion feature recorded in the feature level calculation set, and summarize and calculate the obtained similarities to obtain the user emotion level;

[0071] The user's emotion category and user's emotion level are fed back to the robot pet, and the robot pet executes the corresponding companionship strategy according to the feedback result.

[0072] Furthermore, a user emotion recognition device based on multi-sensor technology includes a memory and a processor:

[0073] The memory stores a computer processing program;

[0074] The processor is used to execute the computer processing program in the memory to implement a user emotion recognition method based on multi-sensor technology.

[0075] (III) Beneficial effects

[0076] The present invention provides a user emotion recognition method, system and device based on multi-sensor technology, which has the following beneficial effects:

[0077] 1. The present invention provides a user emotion recognition method, system and device based on multi-sensor technology, which analyzes the emotion data recognized by the user, and the subsequent comparison is also based on the user's data. Therefore, the system screens the emotion characteristics based on the user's personal data, so that the accuracy of emotion recognition is very high, the accuracy of emotion recognition is relatively high, the use effect is good, and it has good application prospects.

[0078] 2. The present invention provides a user emotion recognition method, system and device based on multi-sensor technology, which analyzes the emotion data recognized by the user, studies the change amount of each emotion feature when the user's emotions fluctuate, and eliminates the emotion features with small changes to avoid interference during recognition. Moreover, this step can greatly reduce the amount of comparison of emotion feature data during the emotion recognition process, thereby speeding up the emotion recognition process. It has good use effect and good application prospects.

[0079] 3. The present invention provides a method, system and device for user emotion recognition based on multi-sensor technology, which further studies the emotion data recognized by the user, screens out the emotion features that change greatly when the user's emotions fluctuate, and uses them for preliminary screening. In addition, it adopts a method of combining voice and face to quickly realize the recognition of user emotions, and further analyzes after recognition to judge the user's emotion level, and executes corresponding companionship strategies according to the user's emotion level, which can provide users with accurate services, has better use effect, and has good application prospects. BRIEF DESCRIPTION OF THE DRAWINGS

[0080] Figure 1 This is a flowchart of a user emotion recognition system based on multi-sensor technology of the present invention;

[0081] Figure 2 This is an extraction diagram of facial feature points of a happy emotion when a user emotion recognition system based on multi-sensor technology is used in the present invention. DETAILED DESCRIPTION

[0082] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0083] Research and development concept: Existing emotion recognition systems are all implemented through voice comparison under different emotions, but there are major defects in actual use. For people with strong self-control, the fluctuation of their voice characteristics when their emotions change is very small, and the recognition accuracy is not high. For people with weak self-control, the amplitude of their emotions changes is very large, so the characteristics of their voice change greatly, and the voice is usually longer. Therefore, the efficiency of emotion recognition is not high.

[0084] In the early stages of research and development, the main focus was on finding a more convenient solution for emotion recognition. In order to improve efficiency, a method for identifying user emotions based on facial features was developed. Since facial features show different states in different emotions, emotions can be recognized quickly based on this.

[0085] However, there are certain disadvantages in actual use. For example, it is difficult to extract the eye features of people with small eyes, and there are some other emotional features that are similar under different emotions. At this time, the above method is used to directly perform emotion recognition, and the efficiency of emotion recognition is relatively low, which does not meet people's usage requirements.

[0086] Therefore, in the middle stage of research and development, a solution for individual identification of individual users was developed. Therefore, data based on user emotion identification was developed, the data was analyzed, and comparison was made based on the data to judge the differences between features. Emotional features with small differences were deleted, which can reduce the amount of data comparison, thereby effectively improving the efficiency of emotion recognition and achieving high usage efficiency.

[0087] However, there are certain drawbacks in actual use. When it comes to emotion recognition, it cannot accurately show the specific degree of emotion, and the effect of using a consistent companionship plan when a single emotion has different degrees is poor. Therefore, the use effect of the mechanical pet is not good.

[0088] Therefore, in the later stage of research and development, the judgment of user emotions was added to determine the emotional level of the user emotions, and the corresponding plan was adjusted according to the user's emotional level, which resulted in better use effect. In addition, the previous plan was further optimized to make the use of the entire system more flexible. For specific plans, see Examples 1 and 2.

[0089] Example 1

[0090] See also Figure 1 , this embodiment provides a user emotion recognition system based on multi-sensor technology, including a data acquisition module, a data analysis module, a feature screening module, an emotion recognition module, a comprehensive analysis module and a feedback processing module:

[0091] This system is mainly composed of two parts: hardware and software. Regarding the hardware part, it is mainly a server, which is connected to the mechanical pet through the server. The sensors on the mechanical pet will collect data. The sensors on the mechanical pet include recording equipment for collecting voice data and camera devices for collecting user facial image data.

[0092] The software part consists of multiple modules that constitute the system and a database that stores user data.

[0093] The real-time nature of this system is mainly based on sufficient user emotion data, and the user emotion data needs to be authenticated by the user. This is mainly because during normal use, different users exhibit different states under different emotions. In order to understand the user's characteristic data more clearly and completely, the collected data must be authenticated by the user, and this process is based on the data collection module.

[0094] The data collection module collects user emotion data authenticated by the user and collects user emotion data in real time, and sets it as standard emotion data. The user emotion data includes facial picture data and voice data of the user when he / she has different emotions.

[0095] The facial image data of different emotions are facial image data uploaded by users under different emotions from the initial state to the peak state of emotions.

[0096] Voice data is mainly data uploaded by users under different emotions, mainly different emotions, and it is difficult to collect voice data of emotions at different emotional levels. At the same time, different sentences have different amplitudes, so voice is only suitable for emotion recognition, and it is too difficult to recognize emotional levels.

[0097] After collecting standard emotion data, in order to better compare and analyze it later, the collected data needs to be further processed. The processing process includes grouping the corresponding emotion data together to facilitate subsequent comparison, and then dividing the summarized emotion feature data. The specific process is based on the data analysis module.

[0098] The data analysis module is used to classify standard emotion data, assign corresponding emotion levels according to the process of emotion fluctuation, and refine the standard emotion data to obtain emotion feature types;

[0099] The data analysis module is used to classify standard emotion data, that is, to classify standard emotion data according to user recognition, for example, into the six common emotion categories of happiness, surprise, fear, anger, disgust and discomfort;

[0100] Then the different degrees of emotional expression in the six categories are divided into levels. For example, the same happy emotion, but reaching the peak from the beginning and with different degrees of happiness, will show different characteristics in the data. For example, a slight smile and a big laugh, both of which show happiness, have characteristics at completely different levels. They can be divided into levels with an odd number, such as 5 or 7 levels, so that the subsequent data comparison will be more accurate.

[0101] The types of emotional features include eyebrow data, eye data, nose data, mouth data analyzed from facial image data, amplitude data, speech rate data, spectrum data, resonance peak parameters extracted from speech data, and text data from speech recognition.

[0102] After research, the emotional characteristics reflected by facial image data under different emotions change as follows:

[0103]

[0104] The above are intuitive expressions from the outside world, and there are obvious differences between them. Therefore, we can analyze these emotional features to identify emotions.

[0105] Extracting user emotion data to obtain feature data corresponding to each emotion feature includes extracting facial image data and voice data in standard emotion data;

[0106] The steps to extract facial image data from standard emotion data are as follows:

[0107] Obtain facial recognition feature points used in facial recognition technology;

[0108] The facial recognition feature points are screened and the feature points outside the eyebrows, eyes, nose and mouth areas are removed. The removal results are as follows Figure 2 As shown;

[0109] The screened feature points are classified and summarized to obtain eyebrow point set, eye point set, nose point set and mouth point set;

[0110] Use curves to connect the classified and summarized feature points, analyze the slope, curvature and curvature radius of the curve, and the analysis results constitute feature data;

[0111] Analyze the slope, curvature and curvature radius of all curves separately. For example, a single eyebrow is a curve, and a mouth is two curves. The accuracy is high in this case, but the calculation is difficult. This multi-group curve analysis method will only be activated for people whose emotional fluctuations cause small curve fluctuations.

[0112] Therefore, in actual use, the curve of the upper part of the eye is analyzed, while the lower curve of the nose and mouth is analyzed. This method can achieve an accuracy rate of more than 90% for emotion recognition and emotion level recognition, so all curves are not analyzed.

[0113] The steps to extract voice data are as follows:

[0114] Performing audio analysis and spectrum analysis on the acquired speech data to obtain amplitude data and spectrum data;

[0115] The speech data is analyzed and processed by using the cepstrum method to extract the formant parameters;

[0116] Using an algorithm based on a statistical model to recognize speech data, and obtaining text data from speech recognition;

[0117] The duration of the voice data and the total number of characters in the text data obtained through voice recognition are obtained, and the speech rate data is calculated using the total number of characters in the text data obtained through voice recognition / the duration of the voice data.

[0118] The method of extracting voice data can be directly implemented by using existing technology, and this technology is relatively mature. Those skilled in the art know how to implement it clearly, so no further description is given.

[0119] The present invention provides a user emotion recognition method, system and device based on multi-sensor technology, which analyzes the emotion data recognized by the user, and the subsequent comparison is also based on the user's data. Therefore, the system screens the emotion characteristics based on the user's personal data, so that the accuracy of emotion recognition is very high, the accuracy of emotion recognition is relatively high, the use effect is good, and it has good application prospects.

[0120] After analyzing the emotional feature data and obtaining the corresponding standards, it needs to be further tested to screen out the appropriate emotional features based on the user's own situation, and then identify the emotional data collected in real time based on the screened emotional features. This method can achieve accurate recognition of emotions with high accuracy, and this process is based on the feature screening module.

[0121] Feature screening module: used to evaluate emotional features, obtain the feature data of the maximum and minimum emotional levels of the emotional feature types corresponding to the facial images under different emotions, calculate the feature change, and compare the feature change with the feature setting value, delete the emotional features below the feature setting value, obtain the category of the remaining emotional features after deletion, and extract the feature data of the remaining emotional features under different emotions and different emotional levels to form a feature level calculation set, and then screen the emotional features in the feature level calculation set for proximity, summarize the screened emotional features and the feature data corresponding to the emotional features, and obtain a preliminary emotion recognition feature set.

[0122] The steps to obtain the difference data between the maximum emotional level and the normal state (the calm state authenticated by the user, which can also be replaced by the corresponding lowest emotional level) of each category and calculate the feature change are as follows:

[0123] Obtaining the emotion feature type data when the set emotion level is the maximum and the feature data of different emotion classifications when the emotion level is the minimum;

[0124] Analyze the feature data, calculate the difference ratio of each data, and then calculate the feature change amount based on the difference ratio;

[0125] ;

[0126] In the formula, is the characteristic variation of the emotional feature, n is the number of data types for identifying the emotional feature, is the i-th item of data when the emotion level is the maximum, is the i-th item of data when the emotion level is the minimum, is the weight coefficient corresponding to the i-th data type;

[0127] The weight coefficient is determined by the coefficient of variation method, which is a method of weighting each indicator according to the degree of variation between the current value of each evaluation indicator and the target value. If the numerical difference of an indicator is large, it can clearly distinguish the evaluated objects, indicating that the indicator has rich discrimination information, and thus the indicator should be given a larger weight. On the contrary, if the numerical difference of each evaluated object on a certain indicator is small, then the ability of this indicator to distinguish the evaluation objects is weak, and thus the indicator should be given a smaller weight. This method directly uses the information contained in each indicator and obtains the weight of the indicator through calculation, so it is objective.

[0128] For example, if the slope, curvature and curvature radius of a curve are used as data types for calculation, then n=3. The length of the curve can also be increased. Although the accuracy increases after the increase, the amount of calculation will increase. Therefore, the slope, curvature and curvature radius of the curve can be used for calculation.

[0129] The above can calculate whether the change of the emotional feature is obvious when the emotion fluctuates. If it is not obvious, it is difficult to judge, so the emotional feature is deleted.

[0130] For example, for users with small eyes, their emotional fluctuations have very little impact on their eye features. If eye features are still used for analysis, the difficulty will be very high and it will affect the accuracy.

[0131] The calculation formula of the characteristic setting value TZs is: .

[0132] The characteristic setting value, as a screening standard value, can well implement the screening of emotional features, thereby ensuring that the remaining emotional features have a large variation and are easy to identify.

[0133] In the above case, the emotional features corresponding to the six emotions are all calculated in this step. For example, the eyebrow features are analyzed separately under the six emotions, and each emotional feature is analyzed six times. Only when the corresponding feature setting values ​​are met six times can the requirements of the feature level calculation set be met.

[0134] The steps for screening the proximity of the emotional features in the feature level calculation set are as follows:

[0135] When the emotion level is the highest, the difference between the emotion data of this emotion classification and the emotion data of any other emotion classification is calculated one by one. The specific formula is as follows:

[0136] ;

[0137] In the formula, is the difference between the emotion data of this emotion classification and the emotion data of any other emotion classification, is the i-th item of data of any other emotion when the emotion level is the maximum, is the i-th item of data of any other emotion when the emotion level is the minimum;

[0138] This step determines the performance difference of the same emotional feature under different maximum emotion levels. If the calculated value is smaller, it means that the performance of the emotional feature under different emotions is more similar, and errors are more likely to occur when using this emotional feature in recognition.

[0139] For example, the difference between the eyebrow features at the maximum level of happy emotions and the eyebrow features at the maximum level of sad emotions.

[0140] Summarize the calculated L groups of difference values, and compare the summarized difference values ​​with the set standard difference values ​​one by one;

[0141] Get the number of difference values ​​k that are less than the set standard difference value, and calculate the degree of closeness PROx. The specific calculation formula is as follows:

[0142] ;

[0143] Determine the interval to which the proximity degree PROx belongs, which includes the proximity interval (0, A) and the identification interval If the proximity degree PROx is in the proximity interval (0, A), the emotional feature is removed. If the proximity degree PROx is in the identification interval If the emotion feature is in the range of 0 to 1, then the emotion feature is retained.

[0144] For example, some users like to frown and develop a habit of frowning when they are in different emotions. In this case, the recognition of eyebrow features is inaccurate. Therefore, if this emotional feature is used in the initial recognition, it will increase the difficulty of emotion recognition.

[0145] However, most users have some specific habits. If the habit only has two emotions and the other emotions are expressed in different forms, the emotional characteristics can still be retained, so a screening method is used instead.

[0146] For example, if there are 6 types of emotions, that is, L is 5, and k is 2, then the calculated PROx is 0.6. If A is 0.6, the emotion feature meets the requirements of the preliminary emotion recognition feature set. If A is 0.7, the emotion feature does not meet the requirements of the preliminary emotion recognition feature set.

[0147] After obtaining the preliminary recognition feature set, the preliminary recognition feature set can be used to process the user emotion data collected in real time to determine the user's specific emotion classification, and this step is based on the emotion recognition module.

[0148] Emotion recognition module: used to extract user emotion data collected in real time, obtain real-time feature data corresponding to the emotion features recorded in the emotion preliminary recognition feature set, compare the feature data in the emotion preliminary recognition feature set with the real-time feature data, perform similarity analysis, and obtain the user emotion category.

[0149] The steps for comparing the feature data in the preliminary emotion recognition feature set with the real-time feature data and performing similarity analysis are as follows:

[0150] Extract C emotion features extracted from facial image data from the emotion preliminary recognition feature set, and retrieve feature data of the C emotion features at different emotion levels to form a feature comparison set of different emotions;

[0151] The extraction method that does not use all features can shorten the time of emotion recognition and improve efficiency.

[0152] C is set according to the user's daily recognition settings. For example, if two emotion features can be used for accurate recognition, then C is 2. If the results of the recognition of two emotion features contain two emotions, it means that the recognition accuracy is not high and additional emotion features need to be added.

[0153] Randomly extract K pieces of face image data collected in real time, extract K groups of data, 3≤K≤5, and the multiple recognition results are relatively accurate;

[0154] K groups of data are compared with feature comparison sets of different emotions one by one, and the similarity with different emotions is calculated. After each group of data is compared with the feature comparison sets of different emotions one by one, the maximum similarity value is extracted, thereby obtaining the maximum similarity value of K groups;

[0155] The data corresponding to each picture is identified once to obtain a set of maximum similarities (there will be a similarity with each emotion, so each set of data will produce 6 similarities, and the maximum similarity is taken).

[0156] If the emotions corresponding to the maximum values ​​of K groups of similarities are the same, the user emotion category is that emotion. For example, if the emotions corresponding to the maximum values ​​of K groups of similarities are all happy, then the user emotion category is determined to be happy.

[0157] The formula for calculating the similarity between real-time sentiment data and different sentiments is as follows:

[0158] ;

[0159] In the formula, SIM is the calculated similarity of different emotions, is the similarity of the i-th emotional feature, is the weight coefficient of the i-th emotional feature, is the jth data in the i-th emotion feature in the real-time collected data, is the jth data item in the i-th emotion feature in the feature comparison set, is the maximum difference of the j-th data in the set i-th emotion feature.

[0160] The similarity analysis in this step is to compare the C group of emotional feature data in the K pieces of real-time collected facial image data with the C group of emotional feature data corresponding to the emotional levels between different emotions.

[0161] Each emotion feature will get a similarity data, and the similarity data obtained from C emotion features are weighted to get the similarity with the emotion.

[0162] In actual use, sometimes the user's face cannot be collected (the user covers his face). In this case, the above method cannot achieve rapid emotion recognition. In this case, the following backup plan is used.

[0163] If the emotions corresponding to the maximum values ​​of the K groups of similarities are different or the real-time collected emotion data does not contain facial image data, then voice data comparison is performed. The voice data comparison includes comparing the text data obtained by voice recognition with the text library recorded in the standard emotion data, comparing the waveform corresponding to the amplitude data with the amplitude waveforms corresponding to different emotions in the standard emotion data, and comparing the waveform corresponding to the spectrum data with the spectrum waveforms corresponding to different emotions in the standard emotion data. The maximum value of the similarity obtained by each comparison is taken to determine whether the emotions corresponding to the maximum value of the similarity are the same;

[0164] If the emotions corresponding to the three groups of maximum similarities are the same, the user's emotion category is that emotion;

[0165] If the emotions corresponding to the three groups of maximum similarity values ​​are different, a speech rate similarity comparison is added to determine the types of emotions corresponding to the four groups of maximum similarity values. If there is only one emotion mode at this time, the user emotion category is the mode emotion. If there are two emotion modes at this time, a resonance peak similarity comparison is added to transmit the emotion corresponding to the resonance peak similarity comparison result.

[0166] The present invention provides a user emotion recognition method, system and device based on multi-sensor technology, which analyzes emotion data recognized by the user, studies the change amount of each emotion feature when the user's emotion fluctuates, and eliminates emotion features with small changes to avoid interference during recognition. Moreover, this step can greatly reduce the amount of comparison of emotion feature data during the emotion recognition process, thereby speeding up the emotion recognition process, has good use effect, and has good application prospects.

[0167] When in use, the emotions are sent to the mechanical pet, which executes the general companionship plan corresponding to the emotions, and then conducts a comprehensive analysis of the subsequent emotional levels.

[0168] When the user's emotions are obtained, in order to better accompany the user, it is necessary to further analyze the level of the user's emotions so as to implement a better accompanying strategy. The process of calculating the level of the user's emotions is based on the comprehensive analysis module.

[0169] Comprehensive analysis module: used to obtain user emotion categories, and perform similarity analysis between the user emotion data collected in real time and the feature data of each emotion feature recorded in the feature level calculation set, and summarize and calculate the obtained similarities to obtain the user emotion level;

[0170] The steps for performing similarity analysis between the feature data collected in real time and the feature data of each emotional feature recorded in the feature level calculation set are as follows:

[0171] A facial image is sampled at every T time interval, a total of G facial images are obtained, and G groups of data are extracted from the G facial images;

[0172] 1 sec ≤ T ≤ 3 sec, 5 ≤ G ≤ 7, to obtain sufficient data analysis.

[0173] The extracted G group data are compared with the feature data of different emotion levels corresponding to the user emotion categories in the feature level calculation set;

[0174] The formula for calculating the similarity with different emotion level feature data is as follows;

[0175] ;

[0176] In the formula, is the similarity of different emotion level feature data, h is the number of types of emotion features recorded in the feature level calculation set, is the similarity of the i-th emotional feature, is the j-th data in the i-th emotional feature corresponding to the emotional level;

[0177] Get the emotion level corresponding to the maximum similarity value of group G data, and calculate the average emotion level , where is the average emotional level, is the emotion level corresponding to the maximum similarity value of the i-th group of data;

[0178] The output user sentiment level is .

[0179] The similarity analysis in this step is to compare the h groups of emotion feature data in the G group of data with the h groups of emotion feature data corresponding to different emotion levels of the identified emotions.

[0180] The two similarity analyses have different purposes.

[0181] Since user emotions are subject to a changing process and the time for the maximum order of normal emotion levels is relatively short, the average emotion level can be used.

[0182] For example, G takes 6, and the emotion levels corresponding to the maximum similarity values ​​of the 6 groups of data are 3, 4, 5, 5, 4, and 4 respectively. After calculation, we get The value is 4.16, and the strategy implemented by the robot pet is the companionship strategy with an emotion level of 4.

[0183] Feedback processing module: used to obtain the user's emotion category and user emotion level, and feed back the results to the robot pet. The robot pet executes the corresponding companionship strategy according to the feedback results.

[0184] The present invention provides a user emotion recognition method, system and device based on multi-sensor technology, which further studies the emotion data recognized by the user, screens out the emotion features that change greatly when the user's emotions fluctuate, and uses them for preliminary screening. In addition, it adopts a method of combining voice and face to quickly realize the recognition of user emotions, and further analyzes after recognition to judge the user's emotion level, and executes the corresponding companionship strategy according to the user's emotion level, which can provide accurate services for users, has better use effect, and has good application prospects.

[0185] When in use, the user's emotion category is first fed back to the robot pet, and the robot pet executes the general strategy of the corresponding emotion. The entire user emotion category analysis process takes 0.3-1.8 seconds. In the process of executing the general strategy, data is collected, the user's emotion level is calculated, and then the corresponding companionship strategy is executed according to the calculation result. This step takes 8-12 seconds. This method can achieve rapid companionship, shorten the reaction time, and has good use effect.

[0186] Example 2

[0187] Based on Example 1, this embodiment further provides a user emotion recognition method based on multi-sensor technology, including the following specific steps:

[0188] Collect user emotion data authenticated by the user and collect user emotion data in real time, and set it as standard emotion data, the user emotion data includes facial image data and voice data of the user in different emotions;

[0189] Classify the standard emotion data, assign corresponding emotion levels according to the process of emotion fluctuation, and refine the standard emotion data to obtain emotion feature types;

[0190] Emotional features are evaluated, and feature data of the maximum and minimum emotion levels of the emotion feature types corresponding to the facial images under different emotions are obtained, and the feature change amount is calculated, and the feature change amount is compared with the feature setting value, and the emotion features below the feature setting value are deleted, and the categories of the remaining emotion features after deletion are obtained, and the feature data of the remaining emotion features under different emotions and different emotion levels are extracted to form a feature level calculation set, and then the emotion features in the feature level calculation set are screened for proximity, and the screened emotion features and the feature data corresponding to the emotion features are summarized to obtain a preliminary emotion recognition feature set;

[0191] Extracting the user emotion data collected in real time to obtain the real-time feature data corresponding to the emotion features recorded in the emotion preliminary identification feature set, comparing the feature data in the emotion preliminary identification feature set with the real-time feature data, performing similarity analysis, and obtaining the user emotion category;

[0192] Perform similarity analysis on the feature data collected in real time and the feature data of each emotion feature recorded in the feature level calculation set, and summarize and calculate the obtained similarities to obtain the user emotion level;

[0193] The user's emotion category and user's emotion level are fed back to the robot pet, and the robot pet executes the corresponding companionship strategy according to the feedback result.

[0194] A user emotion recognition device based on multi-sensor technology, comprising a memory and a processor:

[0195] The memory stores a computer processing program; the processor is used to execute the computer processing program in the memory to implement a user emotion recognition method based on multi-sensor technology.

[0196] In the application, the several formulas involved are all calculated by taking their numerical values ​​after removing the dimensions, and the formulas are established by collecting a large amount of data and performing software simulation to obtain a formula for the most recent actual situation. Some coefficients or weights in the formulas are set by technical personnel in this field according to actual conditions, so they will not be elaborated here.

[0197] The above embodiments may be implemented in whole or in part by software, hardware, firmware or any other combination thereof. When implemented using software, the above embodiments may be implemented in whole or in part in the form of a computer program product. A person of ordinary skill in the art may appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein may be implemented in electronic hardware, or in a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution.

[0198] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, and may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0199] The above description is only a specific implementation manner of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application.

Claims

1. A user emotion recognition system based on multi-sensor technology, characterized by: include: Data collection module: used to collect user emotional data authenticated by the user and collect user emotional data in real time, and set it as standard emotional data; Data analysis module: used to classify standard emotion data, assign corresponding emotion levels according to the process of emotion fluctuation, and refine standard emotion data to obtain emotion feature types; Feature screening module: used to evaluate emotional features, obtain the feature data of the maximum and minimum emotional levels of the emotional feature types corresponding to the facial images under different emotions, calculate the feature change amount, screen the feature change amount, form the feature level calculation set with the screened feature data, and further perform proximity screening to obtain the preliminary emotion recognition feature set; The steps to calculate the feature variation are as follows: Obtaining emotion feature type data when the emotion level is set to be the maximum and different emotion classification feature data when the emotion level is set to be the minimum; Analyze the feature data, calculate the difference ratio of each data, and then calculate the feature change amount based on the difference ratio; In the formula, is the characteristic variation of the emotional feature, n is the number of data types for identifying the emotional feature, is the i-th item of data when the emotion level is the maximum, is the i-th item of data when the emotion level is the minimum, is the weight coefficient corresponding to the i-th data type; Emotion recognition module: used to extract the user emotion data collected in real time, obtain the real-time feature data corresponding to the emotion features recorded in the emotion preliminary recognition feature set, compare the feature data in the emotion preliminary recognition feature set with the real-time feature data, perform similarity analysis, and obtain the user emotion category; Comprehensive analysis module: used to obtain user emotion categories, and perform similarity analysis between the user emotion data collected in real time and the feature data of each emotion feature recorded in the feature level calculation set, and summarize and calculate the obtained similarities to obtain the user emotion level; Feedback processing module: used to obtain the user's emotion category and user emotion level, and feed back the results to the robot pet. The robot pet executes the corresponding companionship strategy according to the feedback results.

2. The user emotion recognition system based on multi-sensor technology according to claim 1, characterized in that: User emotion data includes facial image data and voice data of users in different emotions. Emotional feature types include eyebrow data, eye data, nose data, mouth data analyzed from facial image data, amplitude data, speech rate data, spectrum data, formant parameters extracted from voice data, and text data from voice recognition; Extracting user emotion data to obtain feature data corresponding to each emotion feature includes extracting facial image data in standard emotion data and extracting voice data.

3. The user emotion recognition system based on multi-sensor technology according to claim 2, characterized in that: The steps to extract facial image data from standard emotion data are as follows: Obtain facial recognition feature points used in facial recognition technology; Filter the facial recognition feature points and remove the feature points outside the eyebrows, eyes, nose and mouth areas; The screened feature points are classified and summarized to obtain eyebrow point set, eye point set, nose point set and mouth point set; Use curves to connect the classified and summarized feature points, analyze the slope, curvature and curvature radius of the curve, and the analysis results constitute feature data; The steps to extract voice data are as follows: Performing audio analysis and spectrum analysis on the acquired speech data to obtain amplitude data and spectrum data; The speech data is analyzed and processed by using the cepstrum method to extract the formant parameters; Using an algorithm based on a statistical model to recognize speech data, and obtaining text data from speech recognition; The duration of the voice data and the total number of characters in the text data obtained through voice recognition are obtained, and the speech rate data is calculated using the total number of characters in the text data obtained through voice recognition / the duration of the voice data.

4. The user emotion recognition system based on multi-sensor technology according to claim 3, characterized in that: When screening the feature change amount, the feature change amount is compared with the feature set value, and the emotional features below the feature set value are deleted, the categories of the remaining emotional features after deletion are obtained, and the feature data of the remaining emotional features under different emotions and different emotional levels are extracted to form a feature level calculation set; The calculation formula of the characteristic setting value TZs is: .

5. The user emotion recognition system based on multi-sensor technology according to claim 4, characterized in that: Further performing proximity screening is to perform proximity screening on the emotional features in the feature level calculation set, and aggregating the screened emotional features and feature data corresponding to the emotional features to obtain a preliminary emotion recognition feature set; The steps for screening the proximity of the emotional features in the feature level calculation set are as follows: When the emotion level is the highest, the difference between the emotion data of this emotion classification and the emotion data of any other emotion classification is calculated one by one. The specific formula is as follows: In the formula, is the difference between the emotion data of this emotion classification and the emotion data of any other emotion classification, is the i-th item of data of any other emotion when the emotion level is the maximum, is the i-th item of data of any other emotion when the emotion level is the minimum; Summarize the calculated L groups of difference values, and compare the summarized difference values ​​with the set standard difference values ​​one by one; Get the number of difference values ​​k that are less than the set standard difference value, and calculate the degree of closeness PROx. The specific calculation formula is as follows: Determine the interval to which the proximity degree PROx belongs, which includes the proximity interval (0, A) and the identification interval If the proximity degree PROx is in the proximity interval (0, A), the emotional feature is removed. If the proximity degree PROx is in the identification interval If the emotion feature is in the range of 0 to 1, then the emotion feature is retained.

6. The user emotion recognition system based on multi-sensor technology according to claim 5, characterized in that: The steps for comparing the feature data in the preliminary emotion recognition feature set with the real-time feature data and performing similarity analysis are as follows: Extract C emotion features extracted from facial image data from the emotion preliminary recognition feature set, and retrieve feature data of the C emotion features at different emotion levels to form a feature comparison set of different emotions; Randomly extract K pieces of face image data collected in real time and extract K groups of data; K groups of data are compared with feature comparison sets of different emotions one by one, and the similarity with different emotions is calculated. After each group of data is compared with the feature comparison sets of different emotions one by one, the maximum similarity value is extracted, thereby obtaining the maximum similarity value of K groups; If the emotions corresponding to the maximum values ​​of K groups of similarities are the same, the user's emotion category is that emotion.

7. The user emotion recognition system based on multi-sensor technology according to claim 6, characterized in that: The formula for calculating the similarity between real-time sentiment data and different sentiments is as follows: In the formula, SIM is the calculated similarity of different emotions, is the similarity of the i-th emotional feature, is the weight coefficient of the i-th emotion feature, is the jth data in the i-th emotion feature in the real-time collected data, is the jth data item in the i-th emotion feature in the feature comparison set, is the maximum difference of the j-th data in the set i-th emotion feature.

8. The user emotion recognition system based on multi-sensor technology according to claim 7, characterized in that: If the emotions corresponding to the maximum values ​​of the K groups of similarities are different or the emotion data collected in real time does not contain facial image data, then voice data comparison is performed; The voice data comparison includes comparing the text data obtained by voice recognition with the text library recorded in the standard emotion data, comparing the waveform corresponding to the amplitude data with the amplitude waveform corresponding to different emotions in the standard emotion data, and comparing the waveform corresponding to the spectrum data with the spectrum waveform corresponding to different emotions in the standard emotion data; Take the maximum similarity value obtained from each comparison and determine whether the emotions corresponding to the maximum similarity value are the same; If the emotions corresponding to the three groups of maximum similarities are the same, the user's emotion category is that emotion; If the emotions corresponding to the three groups of maximum similarities are different, then a speech speed similarity comparison is added to determine the types of emotions corresponding to the four groups of maximum similarities; If there is only one emotion mode at this time, the user emotion category is the majority emotion; If there are two modes of emotions at this time, a resonance peak similarity comparison is added, and the emotion corresponding to the resonance peak similarity comparison result is transmitted.

9. The user emotion recognition system based on multi-sensor technology according to claim 8, characterized in that: The steps of performing similarity analysis between the feature data collected in real time and the feature data of each emotional feature recorded in the feature level calculation set are as follows: A facial image is sampled at every T time interval, a total of G facial images are obtained, and G groups of data are extracted from the G facial images; The extracted G group data are compared with the feature data of different emotion levels corresponding to the user emotion categories in the feature level calculation set; The formula for calculating the similarity with different emotion level feature data is as follows; In the formula, is the similarity of different emotion level feature data, h is the number of types of emotion features recorded in the feature level calculation set, is the similarity of the i-th emotional feature, is the j-th data in the i-th emotional feature corresponding to the emotional level; Get the emotion level corresponding to the maximum similarity value of group G data, and calculate the average emotion level , where is the average emotional level, is the emotion level corresponding to the maximum similarity value of the i-th group of data; The output user sentiment level is .

10. A user emotion recognition method based on multi-sensor technology, using any system described in claims 1 to 9, characterized in that: The steps include: Collect user emotion data authenticated by the user and collect user emotion data in real time, and set it as standard emotion data, the user emotion data includes facial image data and voice data of the user in different emotions; Classify the standard emotion data, assign corresponding emotion levels according to the process of emotion fluctuation, and refine the standard emotion data to obtain emotion feature types; Emotional features are evaluated, and feature data of the maximum and minimum emotional levels of the emotional feature types corresponding to the facial images under different emotions are obtained, and the feature changes are calculated, and the feature changes are screened, and the screened feature data are used to form a feature level calculation set, and further closeness screening is performed to obtain a preliminary emotion recognition feature set, and a preliminary emotion recognition feature set is obtained; Extracting the user emotion data collected in real time to obtain the real-time feature data corresponding to the emotion features recorded in the emotion preliminary identification feature set, comparing the feature data in the emotion preliminary identification feature set with the real-time feature data, performing similarity analysis, and obtaining the user emotion category; Perform similarity analysis on the feature data collected in real time and the feature data of each emotion feature recorded in the feature level calculation set, and summarize and calculate the obtained similarities to obtain the user emotion level; The user's emotion category and user's emotion level are fed back to the robot pet, and the robot pet executes the corresponding companionship strategy according to the feedback result.

11. A user emotion recognition device based on multi-sensor technology, characterized in that: include: a memory having a computer processing program stored thereon; A processor is used to execute the computer processing program in the memory to implement the method of claim 10.

Images