A question and answer interaction method and device combined with facial expressions of a person

Abstract

The application relates to the technical field of computer application, and provides a question and answer interaction method and device combined with facial expressions of characters. The purpose is to solve the problem that direct transcoding and merging processes in the existing scheme can significantly affect performance. The main scheme comprises the following steps: collecting facial expressions of users when the users answer questionnaire problems, obtaining facial expression images of a plurality of users, cutting the horizontal side and the vertical side of each facial expression image, then performing aggregation to obtain a sub-image, performing information extraction on all user facial expression sub-images to obtain an expression hidden representation vector; using a nonlinear function to perform mapping to obtain three types of emotions, obtaining branch fixed texts corresponding to all emotion categories, using a BERT model to respectively encode the fixed texts, user labels and original texts of the questionnaire problems to obtain hidden vectors, using a weight modification coefficient to map the obtained hidden vectors to a question space, and the next most suitable question can be obtained.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer application, and provides a question and answer interaction method and device combined with facial expressions of characters. BACKGROUND

[0002] The current work relies on NLP algorithm, CV algorithm

[0003] The current work needs to collect user expressions when the user answers the survey questionnaire, and recommend subsequent series of questions combined with user expressions. The main technical direction of such a process is facial expression analysis and text recommendation. Convolutional neural network is used as the algorithm basis for facial feature extraction, and the image of the conversion of different psychological states of people to facial expressions is analyzed to label the face in real time to obtain the corresponding semantic information. Such semantic information can build a path from computer vision to natural language processing. These facial expression classification labels are used as a set of input information for the text recommendation part. In the recommendation process, the GCN network is used to fuse expression data, user labels and question texts to generate a new set or multiple sets of questions to support subsequent content. In the fusion process, the expression data uses convolutional neural network CNN and fully connected neural network for information extraction, and the user labels and question labels use pre-trained model BERT as the encoder.

[0004] BACKGROUND (SOLUTION STRATEGY OF EXISTING WORK)

[0005] The general solution to this problem is the image-to-text process in the multi-modal case. The image-to-text process can be described as follows: after recognizing the facial expression, a certain label is given, then the label is converted into natural text using a template method, then the user label and the question text are encoded using a pre-trained model, finally the three are fused using GCN, and the obtained hidden vectors are spliced together using a decoder, a hierarchical softmax layer is used to map the result to a set of multi-layer classification question candidates, and finally the most suitable question is obtained. When evaluating whether the question selection is reasonable, the NDCG data relevance evaluation index is used instead of the traditional F value evaluation system. This construction strategy can convert an actual problem generation problem into a multi-modal recommendation system scene problem. In this way, the result can be based on a recommendation system to ensure the accuracy of the result and be persuasive.

[0006] TECHNICAL PROBLEM TO BE SOLVED BY THE PRESENT APPLICATION (MAIN INNOVATION POINT OF THE CURRENT WORK)

[0007] The current work mainly solves the problem that the fixed style questionnaire cannot adapt to specific users. Since the current shopping website questionnaire cannot generate a specific questionnaire according to a specific person, only one set of questionnaires or several sets of questionnaires can be maintained. Although such questionnaires can adapt to a certain group of people, they cannot subdivide problems for each individual to achieve the case of customized for specific people. Generally, there are individual differences between people, and the questionnaire is more about capturing the psychological problems of users, so the differentiated questionnaire meets this goal. The current work can meet the needs of this goal, so it can meet the problem of the specific user proposed.

[0008] The fusion strategy of image information, that is, the fusion strategy of expression image and text information. The difficulty in the multimodal technology when combining image and text is determined according to different scenes. In the scene of combining image and text, the most complex case is that the discrete degree of the numerical value of the image is not enough when the code conversion is calculated, and the vector space of the image is different from the vector space of the text, and cannot express the same semantic information, so the direct code conversion and merging process in the existing scheme will significantly affect the performance, so the original image will undergo different indirect conversion processes to overcome such difficulties for different scenes. In the current scene, the expression of the user in answering the question is an important basis for subsequent question generation, and this basis needs to be converted using a code conversion strategy. The template in the current scheme can accurately convert this information, thereby applying to subsequent tasks. The fusion strategy of text information uses a graph correlation algorithm for fusion, which also considers that a variety of hidden information from different sources can be constructed in a "node transfer" method.

[0009] Under normal circumstances, image-to-text conversion is to convert the whole image into a piece of description text, but this method may lose some detailed information in the image, and may also write some irrelevant information into the text. For example, in a portrait picture, the face of a person and the face of a pet (such as a dog face, a cat face, etc.) or the face of a certain cartoon character (a fake person) appear at the same time, if the image is not cropped, reconstructed, and boundary filtered, these information will be written into the text, thereby causing performance degradation. SUMMARY

[0010] The purpose of the present application is to solve the problem that in the scene of combining image and text, the most complex case is that the discrete degree of the numerical value of the image is not enough when the code conversion is calculated, and the vector space of the image is different from the vector space of the text, and cannot express the same semantic information, so the direct code conversion and merging process in the existing scheme will significantly affect the performance.

[0011] To solve the above technical problems, the present application adopts the following technical means:

[0012] The present application provides a question and answer interaction method combined with facial expressions of characters, comprising the following steps:

[0013] Step 1: Collect the expressions of users when answering the questionnaire, get several expression images of users , where i represents the ith image, a total of n groups, and only one picture is collected at the submission time when the user answers a question;

[0014] Step 2: The horizontal and vertical edges of each expression image are divided s times (here s = 3), and divided into m (m = 9) parts. Here, uniform division is used, and , where j represents the jth image, a total of m groups, and the m expression images are aggregated according to the 2 times of the edge length to become a larger sub-image composed of 4 sub-pictures, a total of k groups (here k = 4), denoted as , where p represents the pth image, a total of k groups, the original image has a huge amount of data due to the pixel problem of the camera, and not every pixel in the image can contain the expression of the person, so the division is performed, which is beneficial to reduce the amount of calculation. At the same time, after division, parallel calculation can be performed, which can save time.

[0015] Step 3: Use ResNet50 and Swin-Transformer network to extract information from all user expression images , get expression hidden representation vector ; at the same time, use ResNet50 convolutional neural network to find all expression images , get the output results of the corresponding face presence or absence and weight ; the formula of this process is as follows:

[0016]

[0017]

[0018] In the above formula, the selection standard of the face output is the minimum absolute value of the sigmoid function output result and 0, 1, where and are calculation parameters. When is determined, the calculation result is determined by the absolute value of the difference between the above output result and , the smaller the absolute value, the more accurate the prediction, so the weight calculation result is as follows:

[0019]

[0020] Step 4: Calculate in the same way as step 3;

[0021] Step 5: and The result of the above formula is used as a mask for the subsequent process, and the logical AND operation is performed, and finally the output result is returned to The formula is as follows:

[0022]

[0023] wherein, represents that all selected j needs to be in the p area after combination, that is, a "larger subgraph" containing four subgraphs.

[0024] Step 6: Map using a nonlinear function to obtain three types of emotions, i.e. "positive", "calm", and "negative", and obtain branch fixed texts corresponding to all emotion categories , and b are training parameters, and the formula is:

[0025]

[0026] In this process, multiple generated fuse the results together, select the user's attitude according to the priority of "positive", "calm", and "negative", and write it into the fixed text:

[0027]

[0028] Merge represents the priority selection merging process, represents that the current subgraph to be fused is derived from the original picture ;

[0029] Step 7: Since the three groups are different, fixed texts , user labels are used to find suitable n characteristic words (n here can be 10) as key information of the original text of the question using the text screening strategy. This idea is a main content extraction method for the original text of the question, which can delete relatively weakly related information, so as to improve the performance of the model. The formula involved here is:

[0030]

[0031]

[0032] =

[0033] In the above two formulas, ​The matrix generated at time t is used to guide the screening of characteristic words, and MaxPos represents the position binary process, that is, after the inner layer result is normalized using Softmax, the position of the maximum value is found, and the value of this position is recorded as 1, and the values of other positions are recorded as 0, so that It has the property of a mask, so when the inner layer is calculated, the Sigmoid operation result is subtracted from this value, that is, the result considered before time t is no longer considered. Here, the fixed text in the three groups of text , user label is weighted using the same parameter , and the original text of the problem weights another parameter , is the bias term. After the first formula is traversed n times, it is obtained , which is multiplied by the original text of the problem to obtain the screened original text of the problem , which replaces the original text of the problem in the subsequent calculation process.

[0034] The fixed text , user label and the original text of the problem of the questionnaire are encoded using the BERT model respectively to obtain the fused three groups of hidden vectors , then the three groups of hidden vectors are encoded using GCN to obtain a group of mapped hidden vectors , the formula is as follows:

[0035]

[0036] Step 8: The obtained hidden vector is mapped to the problem space using a weight modification coefficient and , and then the result is sorted using the NDCG result to obtain the ranking, and the most suitable question is ranked first, so that the next most suitable question can be obtained, and the formula calculation is as follows:

[0037] .

[0038] Step 9: The question is re-applied to step 1, and then the loop process of steps 1-8 is executed until the user completes all the questions.

[0039] The application also provides a question and answer interaction device combined with a person's facial expression, comprising the following modules

[0040] The collection module collects the facial expressions of the user when submitting the time of answering the questionnaire, and obtains an expression image;

[0041] The segmentation and aggregation module divides each expression image into m parts by cutting the horizontal and vertical edges s times Then, the m expression images are aggregated according to the 2 times of the edge length to obtain a user expression sub-image ;

[0042] The information extraction module uses ResNet50 and Swin-Transformer networks to extract information from all user expression sub-images and obtains an expression hidden representation vector ;

[0043] The face searching module uses the ResNet50 convolutional neural network to search for faces in all expression sub-images, and obtains the output results of the corresponding face presence or absence and weights ;

[0044] The face presence or absence output module uses the ResNet50 convolutional neural network to search for faces in all user expression sub-images, and obtains the output results of the corresponding face presence or absence ;

[0045] The selected sub-image module and The results are used as masks for subsequent processes, logical AND operation is performed, and the output results are finally returned to as selected sub-images;

[0046] The fusion module maps using a nonlinear function , and multiplies it with the previously obtained mask and weight to obtain three types of emotions, i.e., “positive”, “calm”, and “negative”, and obtains branch fixed texts corresponding to all emotion categories Multiple generated fuse the results together and write them into the fixed text ;

[0047] The mapping module maps this fixed text , user labels and question texts of the questionnaire to obtain hidden vectors ;

[0048] The question matching module uses a weight modification coefficient to modify the hidden vectors and Map to the problem space, get the most suitable question .

[0049] Because the application adopts the above technical scheme, it has the following beneficial effects:

[0050] First, the application uses a secondary conversion method to split the image and extract feature information to fuse image information. Then it is combined with text information, and then the graph correlation algorithm is used to find the correlation matrix between data in different categories and as the basis for fusion to get the fusion result. In addition, the real-time generation strategy in the present scheme is also the reason why a complete questionnaire can be continuously constructed.

[0051] Second, the image information fusion strategy, that is, the fusion strategy of expression images and text information. The difficulty in multi-modal technology when combining images with text needs to be determined according to different scenarios. In the scenario of combining images with text, the most complex case is that the discrete degree of the numerical value of the image is not enough when it is converted and calculated, and there is a difference between its vector space and the vector space of the text, which cannot express the same semantic information. Therefore, the direct conversion and merging process in the existing scheme will significantly affect the performance, so the original image will go through different indirect conversion processes to overcome such difficulties. In the current scenario, the expression of the user's answer to the question is an important basis for subsequent question generation, and this basis needs to be converted using a conversion strategy. The template in the current scheme can accurately convert this information to facilitate subsequent task applications. The text information fusion strategy uses a graph correlation algorithm for fusion, which also takes into account that a variety of different sources of hidden information can be constructed in a "node transfer" method.

[0052] Third, the application first equally crops the image, labels the meaningful part, and then uses a YoloV5 trained model to identify the area where a face exists in the current sub-image; then a Dense-Net network is used to train a refined identification model to identify such areas. The labels obtained using such a secondary conversion method can effectively reduce the amount of calculation while improving accuracy. BRIEF DESCRIPTION OF DRAWINGS

[0053] Figure 1 For CNN expression classification;

[0054] Figure 2 For flowchart;

[0055] Figure 3 For cutting diagram;

[0056] Figure 4 For example diagram. DETAILED DESCRIPTION

[0057] The embodiments of the present application will be described in detail below. Although the present application will be described and illustrated with reference to some specific embodiments, it is noted that the present application is not limited to only these embodiments. Rather, modifications or equivalent arrangements are intended to be encompassed within the scope of the present application.

[0058] In addition, in order to better illustrate the present application, numerous specific details are given in the following specific embodiments. Those skilled in the art will understand that the present application can also be implemented without these specific details.

[0059] The present application provides a question and answer interaction method combined with human facial expressions, comprising the following steps:

[0060] Step 1: Collect the expressions of the user when answering the questionnaire, and obtain a plurality of expression images of the user , where i represents the ith image, a total of n groups, and only one picture at the submission time is collected when the user answers a question;

[0061] Step 2: The horizontal and vertical edges of each expression image are divided s times (here s = 3), and divided into m (m = 9) parts. Here, uniform division is used, and , where j represents the jth image, a total of m groups, and the m expression images are aggregated according to the 2 times of the edge length to become a larger sub-image composed of 4 sub-images, a total of k groups (here k = 4), denoted as , where p represents the pth image, a total of k groups, the original image has a huge amount of data due to the pixel problem of the camera, and not every pixel in the image can contain the expression of the person, so the division is performed, which is beneficial to reduce the amount of calculation. At the same time, after the division, parallel calculation can be performed, which can save time.

[0062] Step 3: Use ResNet50 and Swin-Transformer network to extract information from all user expression images , and obtain expression hidden representation vectors ; at the same time, use ResNet50 convolutional neural network to find faces in all expression images , and obtain the output results of the corresponding face presence or absence and weights ; the formula of this process is as follows:

[0063]

[0064]

[0065] In the above formula, the selection criteria of the face output is the minimum absolute value between the sigmoid function output result and 0, 1, wherein, and are calculation parameters. When is determined, the calculation result is determined by the absolute value of the difference between the above output result and . The smaller the absolute value, the more accurate the prediction, and thus the weight calculation result is as follows:

[0066]

[0067] Step 4: Calculate in the same way as step 3;

[0068] Step 5: and results are used as masks for subsequent processes, logical AND operation is performed, and finally the output result is returned to as the selected subgraph, and the formula is as follows:

[0069]

[0070] wherein, indicates that all selected j need to be in the p region after combination, that is, a “larger subgraph” containing four subgraphs.

[0071] Step 6: Use a nonlinear function to map to obtain three types of emotions, i.e., “positive”, “calm”, and “negative”, and obtain the branch fixed text corresponding to all emotion categories , and b are training parameters, and the formula is expressed as:

[0072]

[0073] In this process, multiple generated fuse the results together, select the user's attitude according to the priority of “positive”, “calm”, and “negative”, and write into the fixed text:

[0074]

[0075] Merge represents the priority selection merging process, represents that the current subgraph to be fused needs to be derived from the original picture ;

[0076] Based on the technical level, because not every word in a text is useful, in many cases some function words are irrelevant in the language model, but from the human point of view, it is not a sentence without these words, so my innovation point is to delete some words including these words that are not much related to the final prediction result, which can reduce the input content that the model needs to consider, thereby reducing the content space size, making the mapping result of the vector more easily mapped to the result set, thereby improving performance.

[0077] Step 7: Because the three groups of sources are different, this fixed text is used here , user label to the original text of the question Use the strategy of text screening to find n suitable feature words (n here can be 10) as the keyword information of the original text of the question, this idea is the main content extraction method of the original text of the question, which can delete information with weak correlation, which can improve the performance of the model, the formula involved here is:

[0078]

[0079]

[0080] =

[0081] In the above two formulas, represents the matrix generated at time t to guide the screening of feature words, MaxPos represents the position binary process, that is, after using Softmax to normalize the inner layer result, find the position of the maximum value, and record the value of this position as 1, and the other positions as 0, so that has the property of mask, so the Sigmoid operation result will be subtracted from this value during calculation, that is, no longer consider the results considered before time t. Here, the fixed text , user label use the same parameter for weighting, while the original text of the question make another parameter weighted, is the bias term. After traversing n times in the first formula, we get , which can be multiplied by the original text of the question to get the screened original text of the question , instead of the original text of the question to perform the subsequent calculation process.

[0082] This fixed text , user label And the original text of the questionnaire survey question The three groups of hidden vectors after fusion are obtained by using the BERT model for encoding respectively Then the three groups of hidden vectors are encoded using GCN to obtain a group of mapped hidden vectors The formula is as follows:

[0083]

[0084] Step 8: the obtained hidden vector is mapped to the question space using a weight modification coefficient and , and then the result is sorted using NDCG to obtain the ranking, and the most suitable question is ranked first, so that the next most suitable question can be obtained The formula calculation is as follows:

[0085] .

[0086] Step 9: the question is re-applied to step 1, and then the loop process of steps 1-8 is executed until the user completes all the questions.

[0087] The application also provides a question and answer interaction device combined with facial expressions of characters, comprising the following modules

[0088] The collection module collects the facial expressions of the characters at the submission time when the user answers the questionnaire, and obtains expression images;

[0089] The segmentation and aggregation module divides each expression image into s times along the horizontal and vertical edges, and divides it into m expression subgraphs Then the m expression images are aggregated according to the 2 times of the edge length to obtain user expression subimages ;

[0090] The information extraction module uses ResNet50 and Swin-Transformer network to extract information from all user expression subimages to obtain expression hidden representation vectors ;

[0091] The face searching module simultaneously uses the ResNet50 convolutional neural network to search for faces in all expression subgraphs, and obtains the output results of the corresponding face presence or absence and the weight ;

[0092] The face presence or absence output module simultaneously uses the ResNet50 convolutional neural network to search for faces in all user expression subimages, and obtains the output results of the corresponding face presence or absence ;

[0093] selected sub-image module, and the result as a mask for the subsequent process, do logical and operation, and finally return the output result to selected sub-image;

[0094] fusion module, will use a nonlinear function mapping, and multiplying the mask and weight obtained before, get three kinds of emotion, namely "positive", "calm", "negative", get the corresponding branch fixed text , multiple generated fusion results together and write to fixed text ;

[0095] mapping module, this fixed text , user label and the original text of the questionnaire survey question mapping to get hidden vector ;

[0096] question matching module, the hidden vector use a weight modification coefficient and mapping to the question space, get the most suitable question .

[0097] The key point of the present proposal is to use the secondary conversion method to split the image and extract the feature information to fuse the image information. Then combine it with the text information, and then use the graph correlation algorithm to find the correlation matrix between the data of different categories and use it as the basis for fusion to get the fusion result. In addition, the real-time generation strategy in the present scheme is also the reason why a complete survey questionnaire can be continuously constructed.

[0098] Example: as follows Figure 4 , after the image is cut, it can be seen that the face part is mainly on the left, and the dog face on the right cannot be counted as expression content, so it can be proved that the image cutting part in the current scheme is effective. After generating information from this type of image, combining the label and question part content, the question result can be obtained by the above algorithm process.

[0099] The fusion strategy of image information, that is, the fusion strategy of expression image and text information. The difficulty in multimodal technology when combining image and text is determined according to different scenes. In the scene of combining image and text, the most complex case is that the discrete degree of the numerical value of the image is not enough when the code is calculated, and the vector space of the image is different from the vector space of the text, and cannot express the same semantic information, so the direct code conversion and merging process in the existing scheme will significantly affect the performance, so the original image will undergo different indirect conversion processes to overcome such difficulties for different scenes. In the current scene, the expression of the user's answer to the question is an important basis for subsequent question generation, which needs to be converted using a code strategy. The template in the current scheme can accurately convert this information, thereby applying to subsequent tasks. The fusion strategy of text information uses graph-related algorithms for fusion, which also considers that a variety of hidden information from different sources can be constructed in a "node transfer" method.

[0100] Advantages of template method: the image-to-text process will get a set of hidden codes (which refers to the intermediate calculation results obtained by inputting the neural network after the actual description transaction is vectorized, and the hidden codes are not in the output layer), but due to the large change range of such codes (the change range of hidden codes after direct conversion of the image is large, for example, a 1920*1080 image, the number of image pixels in this image is 1920*1080*3, and these values have large differences with each other, so the change range is very large after direct conversion to hidden vectors), therefore, it is not suitable for direct fusion with text information. After using a convolutional neural network, the idea of joint error can be used (the loss value generated by the main image conversion to the face sub-image, face sub-image label recognition, and text part reply information is superimposed, thereby realizing joint error, which has the advantage of being able to discover the independent implementation effect of the above three different modules through the error value size.), a mapping result is given in this step, and then the mapping result is encoded again, and fused with subsequent information, preventing irrelevant features in the image from affecting the final result, so the template method is better than the direct import method.

[0101] The purpose of the joint error is to use the image equal division scheme, use the "target detection model" to find the part containing the face, and at the same time as the first group of errors; Then use the "facial expression detection model" to detect the expression label of the face, as the second group of errors. Here the label is defined as: "positive expression", "negative expression", "neutral expression", a total of 3, so that this scheme can convert the excessive pixel point value information in the image information into these three groups of results. Finally, such information is combined with the information selected by the user when completing the previous question of the survey questionnaire, so as to calculate the next question that needs to be answered by the user, realize the question recommendation, and at the same time take the error of this result as the third group of errors. The three errors and the total error of the model training are used to correct the model, so that the model is more accurate.

[0102] The advantage of fusion coding is that the graph algorithm GCN can filter information of different dimensions as much as possible. Compared with the strategy of using weighted addition, this method can bring more accurate feedback results, thereby avoiding irrelevant problems selected by the model.

Claims

1. A method for question and answer interaction combined with facial expressions of a person, characterized in that, The method comprises the following steps: Step 1, collecting the facial expression of the user when answering the questionnaire, obtaining an expression image; Step 2, cutting the horizontal and vertical edges of each expression image s times, cutting into m expression subgraphs Then, m expression images are aggregated according to the 2 times of the edge length to obtain a user expression subgraph ; Step 3, using ResNet50 and Swin-Transformer network to all user expression sub-images Information extraction is performed to obtain an expression hidden representation vector ; Step 4, simultaneously using ResNet50 convolutional neural network on all expression subgraphs Face search is performed to obtain an output result of whether the corresponding face exists and weights ; Step 5, simultaneously using ResNet50 convolutional neural network to all household expression sub-images Face search is performed to obtain an output result of whether the corresponding face exists ; Step 6, and the result of the subsequent process as a mask, do a logical AND operation, and finally return the output result to as the selected sub-image; Step 7, the non-linear function is used to map and multiply with the previously obtained mask and weights to obtain three categories of emotions, i.e. "positive", "calm", "negative", to obtain the branch fixed text corresponding to all emotion categories , multiple generated results are fused together to write the fixed text ; Step 8, fix the text , user tags and the original text of the questionnaire questions the hidden vector obtained after mapping ; Step 9, the resulting hidden vector using a weight modification factor and mapping to the problem space, resulting in the most appropriate problem . 2.The method of claim 1, wherein, Step 1 comprises the following steps: Collect the expressions of users when answering the questionnaire, and obtain expression images of several users Wherein i represents the i-th image, a total of n, and only one picture at the submission time is collected when the user answers a question. 3.The method of claim 1, wherein, Step 2 comprises the following steps: Cutting the horizontal and vertical edges of each expression image s times, cutting into m expression subgraphs, using uniform cutting, obtaining expression image Corresponding expression subgraph , where j represents the jth image, a total of m, and m expression subgraphs are aggregated according to the 2 times of the edge length, becoming a larger subgraph image spliced together by 4 label subgraphs, a total of k, recorded as user expression subgraph image , where p represents the pth image, a total of k.

4. The method of claim 3, wherein the method further comprises: Step 3 comprises the following steps: Using ResNet50 and Swin-Transformer network on all user expression sub-images Performing information extraction to obtain expression hidden representation vectors : ； Wherein RS represents the calculation process of ResNet50 and Swin-Transformer network.

5. The method of claim 4, wherein the method further comprises: Step 4 comprises the following steps: Using ResNet50 convolutional neural network to all sub-graphs Face search to get the output result of the corresponding face And weight Where The binary result, 1 indicates the presence of a face and 0 indicates the absence of a face; Then it is a result between 0 and 1, using probability value to represent weight, the formula is as follows: In the above formula, R() represents the calculation process of ResNet50 network; and the selection standard of face output is sigmoid function The output result is the minimum absolute value of 0 and 1, wherein, And Both are calculation parameters, when After that, The calculation result is determined by the absolute value of the difference between the above output result and The smaller the absolute value, the more accurate the prediction, so the weight calculation result is as follows: 。 6. The method of claim 5, wherein the method further comprises: Step 6 comprises the following steps: and The result of the operation is used as a mask for the subsequent process, a logical AND operation is performed, and the output result is finally returned to as the selected sub-image. The maximum value of the image weight containing the human face part is selected as the fusion result of the selected sub-image weight, and the formula is as follows: wherein represents that all selected j's need to be in the p region after combination, i.e. a "larger subgraph" that contains four subgraphs.

7. The method of claim 6, wherein the method further comprises: Step 7 Comprise the following steps: Will Using a non-linear function Mapping and multiplying with the previously obtained mask and weights to obtain three categories of emotions, namely "positive", "calm", "negative", and obtaining the branch fixed text corresponding to all emotion categories , And b are training parameters, and the mask Using the "and" operation, the weight Using the element-wise multiplication calculation method, the formula is represented as: In this process, multiple generated Fusing the results together, the user's attitude is selected according to the priority of "positive", "calm", "negative", and written in fixed text : Merge represents a priority selection merge process, The subgraph representing the current need for fusion is to be derived from the expression image .

8. The method of claim 6, wherein the method further comprises: Step 8 comprises the following steps: Using fixed text , user tags To the original text of the question Using the strategy of text screening, find the appropriate n characteristic words as the keyword information of the original text of the question, delete the information with weak relative correlation degree, and the formula is as follows: In the above two formulas, represents the matrix generated at time t for guiding the screening of characteristic words, MaxPos represents the position binary process, that is, after normalizing the inner layer result using Softmax, finding the position of the maximum value, and recording the value of this position as 1 and the other positions as 0, so that It has the property of mask, so when the inner layer is calculated, the Sigmoid operation result is subtracted from this value, that is, the result considered before time t is no longer considered, and the fixed text in the three groups of text , user label uses the same parameter for weighting, while the original text of the question makes another parameter weighted, is the bias term, after the first formula is traversed n times, it is obtained , which can be multiplied by the original text of the question to obtain the screened original text of the question ; This fixed text , user label And the original text of the screening question of the questionnaire survey Use the BERT model to encode respectively to get the fused three groups of hidden vectors Then use GCN to encode the three groups of hidden vectors to get a group of mapped hidden vectors The formula is as follows: 。 9. The method of claim 6, wherein the method further comprises: Step 9 specifically comprises the following steps: The resulting hidden vector Using a weight modification coefficient And Mapping to the question space, resulting in This result is then used to rank the results using NDCG, with the most appropriate question ranked first, so that the next most appropriate question can be obtained The formula is as follows: 。 10. A question and answer interaction device incorporating a person's facial expression, characterized by, Comprise the following modules: The acquisition module collects the facial expression of the user when answering the questionnaire, and obtains an expression image. The segmentation and aggregation module divides each expression image into m parts of expression sub-images by cutting the horizontal and vertical edges s times Then, the m expression images are aggregated according to the 2 times of the edge length to obtain a user expression sub-image ; The information extraction module uses ResNet50 and Swin-Transformer network to all user expression sub-images Information extraction is performed to obtain an expression hidden representation vector ; The face searching module uses a ResNet50 convolutional neural network to search all the expression subgraphs to obtain an output result of whether the corresponding face exists and weights ; The face presence / absence output module uses a ResNet50 convolutional neural network to simultaneously process all of the facial expression sub-images The face search is performed to obtain an output result of the corresponding face presence / absence ; selected sub-image module, and the result of the selected sub-image module as a mask for the subsequent process, performing a logical AND operation, and finally returning the output result to as the selected sub-image; fusion module, to using a non-linear function mapping and multiplying with the previously obtained masks and weights, obtaining the three classes of sentiment, i.e. "positive", "calm", "negative", obtaining the branch fixed text corresponding to all the classes of sentiment , the plurality of generated fusing the results together writing in the fixed text ; mapping module, mapping the fixed text , user tags and the original text of the questionnaire questions the obtained hidden vector after mapping ; question matching module, the obtained hidden vector using a weight modification coefficient and mapping to a question space to obtain the most suitable question .

Images