Mental health evaluation device and method based on expression recognition and LLM

Abstract

The invention relates to the technical field of mental health assessment, and discloses a mental health assessment device and method based on expression recognition and LLM, and the device comprises a data collection module, a facial expression analysis module, a language analysis module, a fusion analysis module, a mental health assessment module, and a result display module. A camera of the data acquisition module acquires face video data of a testee in a specific situation, acquires voice data of the testee in a communication process, and converts the voice data into text data; the facial expression analysis module analyzes the collected facial video data frame by frame by using an advanced facial expression recognition algorithm, and extracts facial expression features. According to the method, the evaluation accuracy is improved, facial expressions and language information are comprehensively considered, the psychological health condition can be evaluated more comprehensively and accurately, and errors caused by a single factor are reduced; according to the method, objectivity is enhanced, interference of human factors in a traditional method is avoided, and the evaluation result is more objective.

Description

Technical Field

[0001] The present invention belongs to the technical field of mental health assessment. Specifically, it relates to a mental health assessment device and method based on facial expression recognition and LLM. Background Art

[0002] With the rapid development of society and the acceleration of the pace of life, the pressure faced by people is increasing day by day, and mental health problems have received more and more attention. Traditional mental health assessment methods usually rely on questionnaires and interviews, and these methods have limitations such as strong subjectivity, low efficiency, and being easily affected by human factors.

[0003] In recent years, with the development of computer technology and artificial intelligence, facial expression recognition technology and language models have shown certain application potential in the field of psychological analysis. However, the existing technologies still have deficiencies in comprehensively using facial expressions and language information for accurate and efficient mental health assessment.

[0004] In view of this, the present invention is specifically proposed. Summary of the Invention

[0005] To solve the above technical problems, the basic concept of the technical solution adopted by the present invention is:

[0006] A mental health assessment device based on facial expression recognition and LLM includes a data acquisition module, a facial expression analysis module, a language analysis module, a fusion analysis module, a mental health assessment module, and a result display module. The data acquisition module uses a camera to collect facial video data of the tested person in a specific scenario, collects voice data of the tested person during the communication process, and converts it into text data. The facial expression analysis module uses an advanced facial expression recognition algorithm to analyze the collected facial video data frame by frame, extracts facial expression features, and then classifies the facial expression features into common expression categories, including but not limited to happiness, sadness, anger, fear, and surprise, and counts the frequency and duration of each expression. The language analysis module uses natural language processing technology to analyze the collected text data. The fusion analysis module is electrically connected to the data acquisition module, the facial expression analysis module, and the language analysis module. The mental health assessment module conducts a mental health assessment based on the results of the fusion analysis using a trained large language model. The result display module displays the mental health assessment results to the user in an intuitive manner.

[0007] As a preferred embodiment of the present invention, the facial expression features include but are not limited to morphological changes in the eyebrows, eyes, and mouth parts.

[0008] As a preferred embodiment of the present invention, the language analysis module extracts key information in the text, including but not limited to emotional tendency, topic content, and language style.

[0009] As a preferred embodiment of the present invention, the fusion analysis module fuses the facial expression analysis result and the language analysis result, establishes the association between the two, and when the facial expression shows sadness and the language content also expresses negative emotions, enhances the judgment weight of the negative mental state.

[0010] As a preferred embodiment of the present invention, the large language model gives a mental health assessment report according to the input fusion features, in combination with a large amount of mental health data and knowledge, including the classification of mental health status, possible problems and corresponding suggestions.

[0011] As a preferred embodiment of the present invention, the classification of the mental health status includes but is not limited to normal, mild anxiety and moderate depression.

[0012] As a preferred embodiment of the present invention, the display methods of the result display module include but are not limited to reports and charts.

[0013] As a preferred embodiment of the present invention, the facial expression analysis module preprocesses the collected facial video data. The preprocessing includes but is not limited to denoising, enhancing contrast, and grayscale conversion, and then decomposes the video data into a series of image frames for frame-by-frame analysis; then uses a face detection algorithm to separate the face from the background of the video frame and accurately locate the face through convolutional neural network technology; after detecting the face, extracts features related to expressions. The features include but are not limited to the shape, size, position of the eyes, mouth, and eyebrows and the relative relationship between them. The feature extraction adopts the method of geometric features; then uses a machine learning algorithm to classify the extracted features to judge the expression expressed by the face, classifies the expression features into common expression categories such as happy, sad, angry, fearful, and surprised; counts the frequency and duration of each expression, which is achieved by counting and timestamp recording of the classified expression frames, and analyzes the change trend and pattern of the expressions to provide in-depth insights into the user's emotional state.

[0014] As a preferred embodiment of the present invention, the working process of the language analysis module includes text preprocessing, syntactic analysis, sentiment analysis, topic content extraction, language style analysis, and information integration and output. In text preprocessing, irrelevant characters, special symbols, and stop words in the text are removed, and then the text is segmented into individual words or phrases, and a part-of-speech tag such as noun, verb, and adjective is assigned to each word. In syntactic analysis, a syntactic analyzer is used to parse the text to identify the subject, predicate, and object of the sentence; in sentiment analysis, a sentiment analysis algorithm is used to judge the sentiment tendency of the text, such as positive, negative, or neutral; in topic content extraction, a topic model or keyword extraction algorithm is used to identify the main topic or theme in the text; in language style analysis, the language style of the text, such as formality, colloquialism, and sense of humor, is analyzed, and language style analysis can be achieved by counting the usage frequencies of specific words or phrases, sentence structures, and other features; in information integration and output, the above analysis results are integrated to form a report containing key information, and the report will be used as one of the inputs of the fusion analysis module and will be jointly used with the analysis results of other modules for mental health assessment.

[0015] As a preferred embodiment of the present invention, in the process of constructing the fuzzy relation matrix of the language large model, it is necessary to accurately capture the subtle connections between elements, ensure that each item of the matrix accurately reflects the actual relationship, so as to provide solid data support for subsequent analysis; The determination of the weight vector needs to be based on the fuzzy relation matrix, and the weights of each element are calculated through a rigorous algorithm to ensure that the vector can accurately reflect the relative importance between elements. Specifically, as follows:

[0016]

[0017] In the fuzzy matrix composite operation, through precise weight allocation, the system can efficiently process multi-dimensional data, and the output result accurately fits the expectation;

[0018] B = A * R = (b1, b2…, bn)

[0019] Among them, each element bi is composed of the sum of the products of the corresponding elements of matrix A and R, ensuring that each element bi strictly follows the operation rules and accurately maps complex relationships;

[0020] In the fuzzy membership degree operation, bj = (w1 * r1j) * (w2 * r2j) * … * (wp * rpj), j = 1, 2, …, m. Through layer-by-layer weighted product, bj accurately reflects the comprehensive influence of each factor, realizes multi-dimensional data fusion, and optimizes the decision-making model; the generated bj value of the evaluation result is obtained through the fuzzy matrix composite operation to ensure the accurate allocation of the weights of each factor;

[0021] W = B × J

[0022] W = (w1j, w2j, …, wmj), where each wj value is composed of the sum of the products of the corresponding elements of B and J, strictly following the operation logic, accurately reflecting the comprehensive evaluation result, and ensuring that the decision-making basis is scientific and reliable; the W value, as the final decision-making basis, effectively integrates multi-dimensional data through systematic operations.

[0023] A method for a mental health assessment device based on facial expression recognition and LLM, including the following working steps:

[0024] First step: The camera of the data acquisition module collects the facial video data of the test subject in a specific scenario, collects the voice data of the test subject during the communication process, and converts it into text data; the facial expression features include, but are not limited to, the morphological changes of the eyebrows, eyes, and mouth parts.

[0025] Second step: The facial expression analysis module uses advanced facial expression recognition algorithms to analyze the collected facial video data frame by frame, extracts facial expression features, and then classifies the facial expression features into common expression categories, including but not limited to happy, sad, angry, fearful, and surprised, and counts the frequency and duration of each expression; the facial expression analysis module preprocesses the collected facial video data, and the preprocessing includes, but is not limited to, denoising, enhancing contrast, and grayscale conversion, and then decomposes the video data into a series of image frames for frame-by-frame analysis; then uses a face detection algorithm to separate the face from the background of the video frame, and accurately locates the face through convolutional neural network technology; after detecting the face, extracts the features related to the expression, and the features include, but are not limited to, the shape, size, position of the eyes, mouth, and eyebrows, and the relative relationship between them, and the feature extraction uses the method of geometric features; then uses a machine learning algorithm to classify the extracted features to determine the expression expressed by the face, classifies the expression features into common expression categories, such as happy, sad, angry, fearful, and surprised; counts the frequency and duration of each expression, which is achieved by counting and timestamp recording of the classified expression frames, analyzes the change trends and patterns of the expressions to provide in-depth insights into the user's emotional state;

[0026] Step 3: The language analysis module uses natural language processing technology to analyze the collected text data; the language analysis module extracts key information from the text, including but not limited to sentiment tendency, topic content, and language style; the working process of the language analysis module includes text preprocessing, syntactic analysis, sentiment analysis, topic content extraction, language style analysis, and information integration and output. In text preprocessing, irrelevant characters, special symbols, and stop words in the text are removed, and then the text is segmented into individual words or phrases, and a part-of-speech tag is assigned to each word, such as noun, verb, and adjective. Syntactic analysis uses a syntactic analyzer to parse the text and identify the subject, predicate, and object of the sentence; sentiment analysis uses a sentiment analysis algorithm to judge the sentiment tendency of the text, such as positive, negative, or neutral; topic content extraction uses a topic model or keyword extraction algorithm to identify the main topic or theme in the text; language style analysis analyzes the language style of the text, such as formality, colloquialism, and sense of humor, and language style analysis can be achieved by counting the usage frequencies of specific words or phrases, sentence structures, and other features; information integration and output integrates the above analysis results to form a report containing key information, and the report will be used as one of the inputs of the fusion analysis module and, together with the analysis results of other modules, for mental health assessment;

[0027] Step 4: The fusion analysis module is electrically connected to the data collection module, the facial expression analysis module, and the language analysis module; the fusion analysis module fuses the facial expression analysis result and the language analysis result, establishes the association between the two, and when the facial expression shows sadness and the language content also expresses negative emotions, the judgment weight of the negative mental state is enhanced;

[0028] Step 5: The mental health assessment module conducts mental health assessment based on the results of the fusion analysis using a trained large language model; the large language model gives a mental health assessment report according to the input fusion features, combined with a large amount of mental health data and knowledge, including the classification of mental health status, possible problems, and corresponding suggestions;

[0029] Step 6: The result display module displays the mental health assessment results to the user in an intuitive way.

[0030] The present invention has the following beneficial effects compared with the prior art:

[0031] 1. The present invention improves the accuracy of the assessment: By comprehensively considering facial expressions and language information, it can more comprehensively and accurately evaluate the mental health status and reduce errors caused by single factors.

[0032] 2. The present invention enhances objectivity: It avoids the interference of human factors in traditional methods and makes the assessment results more objective.

[0033] 3. The present invention improves the efficiency of evaluation, can quickly collect and analyze data, and significantly shortens the evaluation time.

[0034] 4. The present invention can conduct personalized evaluations and can perform personalized mental health evaluations based on the unique performance of the test taker.

[0035] The following further describes in detail the specific implementation manners of the present invention with reference to the accompanying drawings. Description of the Drawings

[0036] In the drawings:

[0037] Figure 1 is a block diagram of the mental health evaluation device of the present invention. Specific Embodiments

[0038] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the following will clearly and completely describe the technical solutions in the embodiments of the present invention with reference to the accompanying drawings in the embodiments of the present invention. The following embodiments are used to illustrate the present invention.

[0039] A mental health evaluation device based on facial expression recognition and LLM, as Figure 1 shown, includes a data collection module, a facial expression analysis module, a language analysis module, a fusion analysis module, a mental health evaluation module, and a result display module. The data collection module uses a camera to collect facial video data of the test taker in a specific scenario, collects voice data of the test taker during communication, and converts it into text data; the facial expression analysis module uses advanced facial expression recognition algorithms to analyze the collected facial video data frame by frame, extracts facial expression features, and then the facial expression analysis module classifies the facial expression features into common expression categories, including but not limited to happy, sad, angry, fearful, and surprised, and counts the frequency and duration of each expression; the language analysis module uses natural language processing technology to analyze the collected text data; the fusion analysis module is electrically connected to the data collection module, the facial expression analysis module, and the language analysis module; the mental health evaluation module conducts a mental health evaluation based on the results of the fusion analysis using a trained large language model; the result display module displays the mental health evaluation results to the user in an intuitive manner. The display methods of the result display module include but are not limited to reports and charts.

[0040] The facial expression features include but are not limited to morphological changes in the eyebrows, eyes, and mouth parts.

[0041] The language analysis module extracts key information in the text, including but not limited to emotional tendency, topic content, and language style.

[0042] The fusion analysis module fuses the facial expression analysis results and the language analysis results, establishes the correlation between the two, and when the facial expression shows sadness and the language content also expresses negative emotions, enhances the judgment weight of the negative mental state.

[0043] The large language model gives a mental health assessment report based on the input fusion features, combined with a large amount of mental health data and knowledge, including the classification of mental health status, possible problems, and corresponding suggestions. The classification of the mental health status includes but is not limited to normal, mild anxiety, and moderate depression.

[0044] Furthermore, the facial expression analysis module preprocesses the collected facial video data. The preprocessing includes but is not limited to denoising, enhancing contrast, and grayscale conversion. Then, the video data is decomposed into a series of image frames for frame-by-frame analysis. Then, the face detection algorithm is used to separate the face from the background of the video frame, and the face is accurately located through convolutional neural network technology. After the face is detected, the features related to the expression are extracted. The features include but are not limited to the shape, size, position of the eyes, mouth, and eyebrows, and the relative relationship between them. The feature extraction uses the method of geometric features. Then, the machine learning algorithm is used to classify the extracted features to judge the expression expressed by the face, and the expression features are classified into common expression categories, such as happy, sad, angry, fearful, and surprised. The frequency and duration of each expression are counted, which is achieved by counting and timestamp recording of the classified expression frames, and the change trend and pattern of the expression are analyzed to provide in-depth insights into the user's emotional state.

[0045] Furthermore, the working process of the language analysis module includes text preprocessing, syntactic analysis, sentiment analysis, topic content extraction, language style analysis, and information integration and output. Among them, text preprocessing removes irrelevant characters, special symbols, and stop words in the text, and then cuts the text into individual words or phrases, and assigns a part-of-speech tag to each word, such as noun, verb, and adjective. Syntactic analysis uses a syntactic analyzer to parse the text and identify the subject, predicate, and object of the sentence. Sentiment analysis uses a sentiment analysis algorithm to judge the sentiment tendency of the text, such as positive, negative, or neutral. Topic content extraction uses a topic model or keyword extraction algorithm to identify the main topic or theme in the text. Language style analysis analyzes the language style of the text, such as formality, colloquialism, and sense of humor. Language style analysis can be achieved by counting the usage frequency of specific words or phrases, sentence structure, and other features. Information integration and output integrates the above analysis results to form a report containing key information, and the report will be used as one of the inputs of the fusion analysis module and jointly used with the analysis results of other modules for mental health assessment.

[0046] Furthermore, in the process of constructing the fuzzy relation matrix for the language large model, it is necessary to accurately capture the subtle connections between elements, ensure that each item in the matrix precisely reflects the actual relationship, and thus provide solid data support for subsequent analysis; The determination of the weight vector needs to be based on the fuzzy relation matrix. Through a rigorous algorithm, calculate the weights of each element to ensure that the vector can accurately reflect the relative importance between elements. Specifically, as follows:

[0047]

[0048] For the composite operation of the fuzzy matrix, through precise weight allocation, the system efficiently processes multi-dimensional data, and the output result precisely meets the expectations;

[0049] B = A * R = (b1, b2…, bn)

[0050] Among them, each element bi is composed of the sum of the products of the corresponding elements of matrix A and R, ensuring that each element bi strictly follows the operation rules and precisely maps complex relationships;

[0051] For the fuzzy membership operation, bj = (w1 * r1j) * (w2 * r2j) * … * (wp * rpj), j = 1, 2, …, m. Through layer-by-layer weighted product, bj precisely reflects the comprehensive influence of each factor, realizes multi-dimensional data fusion, and optimizes the decision-making model; The generated bj value of the evaluation result is obtained through the composite operation of the fuzzy matrix, ensuring the precise allocation of the weights of each factor;

[0052] W = B × J

[0053] W = (w1j, w2j, …, wmj), and each wj value is composed of the sum of the products of the corresponding elements of B and J, strictly following the operation logic, precisely reflecting the comprehensive evaluation result, and ensuring that the decision-making basis is scientific and reliable; The W value is used as the final decision-making basis, and through systematic operations, multi-dimensional data is effectively integrated.

[0054] The method of the mental health assessment device based on facial expression recognition and LLM includes the following working steps:

[0055] The first step: The camera of the data acquisition module collects the facial video data of the test subject in a specific scenario, collects the voice data of the test subject during the communication process, and converts it into text data; The facial expression features include, but are not limited to, the morphological changes of the eyebrows, eyes, and mouth parts.

[0056] Step 2: The facial expression analysis module uses advanced facial expression recognition algorithms to analyze the collected facial video data frame by frame, extract facial expression features, and then classify the facial expression features into common expression categories, including but not limited to happiness, sadness, anger, fear, and surprise, and count the frequency and duration of each expression; the facial expression analysis module preprocesses the collected facial video data, and the preprocessing includes but not limited to denoising, enhancing contrast, and grayscale conversion, and then decomposes the video data into a series of image frames for frame-by-frame analysis; then uses a face detection algorithm to separate the face from the background of the video frame, and accurately locates the face through convolutional neural network technology; after detecting the face, extract the features related to the expression, and the features include but not limited to the shape, size, position of the eyes, mouth, and eyebrows, and the relative relationship between them, and the feature extraction adopts the method of geometric features; then uses a machine learning algorithm to classify the extracted features to judge the expression expressed by the face, and classify the expression features into common expression categories, such as happiness, sadness, anger, fear, and surprise; count the frequency and duration of each expression, which is achieved by counting and timestamp recording the classified expression frames, analyze the change trends and patterns of the expressions to provide in-depth insights into the user's emotional state;

[0057] Step 3: The language analysis module uses natural language processing technology to analyze the collected text data; the language analysis module extracts the key information in the text, including but not limited to sentiment tendency, topic content, and language style; the working process of the language analysis module includes text preprocessing, syntactic analysis, sentiment analysis, topic content extraction, language style analysis, and information integration and output, where the text preprocessing removes the irrelevant characters, special symbols, and stop words in the text, and then cuts the text into individual words or phrases, and assigns a part-of-speech tag to each word, such as noun, verb, and adjective, and the syntactic analysis uses a syntactic analyzer to parse the text and identify the subject, predicate, and object of the sentence; the sentiment analysis uses a sentiment analysis algorithm to judge the sentiment tendency of the text, such as positive, negative, or neutral; the topic content extraction uses a topic model or keyword extraction algorithm to identify the main topic or theme in the text; the language style analyzes the language style of the text, such as the degree of formality, colloquialism, and sense of humor, and the language style analysis can be achieved by counting the usage frequency of specific words or phrases, sentence structure and other features; the information integration and output integrates the above analysis results to form a report containing key information, and the report will be used as one of the inputs of the fusion analysis module and jointly used with the analysis results of other modules for mental health assessment;

[0058] Step 4: The fusion analysis module is electrically connected to the data acquisition module, the facial expression analysis module, and the language analysis module; the fusion analysis module fuses the facial expression analysis results and the language analysis results, establishes the association between the two, and when the facial expression shows sadness and the language content also expresses negative emotions, increases the judgment weight of the negative mental state;

[0059] Step 5: The mental health assessment module conducts mental health assessment based on the results of the fusion analysis using the trained large language model; the large language model gives a mental health assessment report according to the input fusion features, in combination with a large amount of mental health data and knowledge, including the classification of mental health status, possible problems, and corresponding suggestions;

[0060] Step 6: The result display module displays the mental health assessment results to the user in an intuitive way.

[0061] It can be understood that the present invention is described through some embodiments. Those skilled in the art know that without departing from the spirit and scope of the present invention, various changes or equivalent substitutions can be made to these features and embodiments. Additionally, under the teaching of the present invention, these features and embodiments can be modified to adapt to specific situations and materials without departing from the spirit and scope of the present invention. Therefore, the present invention is not limited by the specific embodiments disclosed herein, and all embodiments falling within the scope of the claims of this application belong to the scope protected by the present invention.

Claims

1. A mental health assessment device based on facial expression recognition and LLM, characterized in that, It includes a data acquisition module, a facial expression analysis module, a language analysis module, a fusion analysis module, a mental health assessment module, and a result display module. The data acquisition module uses a camera to collect facial video data of the test subject in a specific scenario, collects voice data of the test subject during the communication process, and converts it into text data. The facial expression analysis module uses advanced facial expression recognition algorithms to analyze the collected facial video data frame by frame, extracts facial expression features, and then classifies the facial expression features into common expression categories, including but not limited to happiness, sadness, anger, fear, and surprise, and counts the frequency and duration of each expression. The language analysis module uses natural language processing technology to analyze the collected text data. The fusion analysis module is electrically connected to the data acquisition module, the facial expression analysis module, and the language analysis module respectively. Based on the results of the fusion analysis, the mental health assessment module uses a trained large language model to conduct a mental health assessment. The result display module presents the mental health assessment results to the user in an intuitive way.

2. The mental health assessment device based on facial expression recognition and LLM according to claim 1, wherein The facial expression features include but are not limited to morphological changes in the eyebrows, eyes, and mouth parts.

3. The mental health assessment device based on facial expression recognition and LLM according to claim 1, characterized in that, The language analysis module extracts key information in the text, including but not limited to emotional tendency, topic content, and language style.

4. The mental health assessment device based on facial expression recognition and LLM according to claim 1, wherein The fusion analysis module fuses the facial expression analysis results and the language analysis results, establishes the association between the two, and when the facial expression shows sadness and the language content also expresses negative emotions, it enhances the judgment weight of the negative mental state.

5. The mental health assessment device based on facial expression recognition and LLM according to claim 1, characterized in that, The large language model gives a mental health assessment report based on the input fusion features, combined with a large amount of mental health data and knowledge, including the classification of mental health status, possible problems, and corresponding suggestions; the classification of mental health status includes but is not limited to normal, mild anxiety, and moderate depression; the display methods of the result display module include but are not limited to reports and charts.

6. The mental health assessment device based on facial expression recognition and LLM according to claim 2, wherein, The facial expression analysis module preprocesses the collected facial video data. The preprocessing includes but is not limited to denoising, enhancing contrast, and grayscale conversion, and then decomposes the video data into a series of image frames for frame-by-frame analysis; then uses a face detection algorithm to separate the face from the background of the video frame, and accurately locates the face through convolutional neural network technology; after detecting the face, extracts features related to expressions, and the features include but are not limited to the shape, size, position of the eyes, mouth, and eyebrows, and the relative relationship between them. The feature extraction uses the method of geometric features; then uses machine learning algorithms to classify the extracted features to judge the expression expressed by the face, and classifies the expression features into common expression categories, such as happiness, sadness, anger, fear, and surprise. The frequency and duration of each expression are counted by counting and timestamp recording of the classified expression frames, and the change trends and patterns of the expressions are analyzed to provide in-depth insights into the user's emotional state.

7. The mental health assessment device based on facial expression recognition and LLM according to claim 3, wherein The working process of the language analysis module includes text preprocessing, syntactic analysis, sentiment analysis, topic content extraction, language style analysis, and information integration and output. In text preprocessing, irrelevant characters, special symbols, and stop words in the text are removed, and then the text is segmented into individual words or phrases, and a part-of-speech tag is assigned to each word, such as nouns, verbs, and adjectives. In syntactic analysis, a syntactic analyzer is used to parse the text to identify the subject, predicate, and object of the sentence. In sentiment analysis, a sentiment analysis algorithm is used to judge the sentiment tendency of the text, such as positive, negative, or neutral. In topic content extraction, a topic model or keyword extraction algorithm is used to identify the main topic or theme in the text. In language style analysis, the language style of the text is analyzed, such as the degree of formality, colloquialism, and sense of humor. Language style analysis can be achieved by counting the usage frequency of specific words or phrases, sentence structure, and other features. Information integration and output integrates the above analysis results to form a report containing key information. The report will be used as one of the inputs for the fusion analysis module and, together with the analysis results of other modules, will be used for mental health assessment.

8. The mental health assessment device based on facial expression recognition and LLM according to claim 1, characterized in that, When constructing the fuzzy relation matrix, the language large model needs to accurately capture the subtle connections between elements, ensuring that each item in the matrix precisely reflects the actual relationship, thereby providing solid data support for subsequent analysis; The determination of the weight vector is based on the fuzzy relation matrix. Through a rigorous algorithm, the weights of each element are calculated to ensure that the vector can accurately reflect the relative importance between elements. Specifically, it is as follows: Fuzzy matrix composition operation. Through precise weight allocation, the system efficiently processes multi-dimensional data, and the output result precisely matches the expectation. B = A * R = (b1, b2…, bn) Among them, each element bi is composed of the sum of the products of the corresponding elements of matrix A and R, ensuring that each element bi strictly follows the operation rules and accurately maps complex relationships. Fuzzy membership operation, bj = (w1 * r1j) * (w2 * r2j) *… * (wp * rpj), j = 1, 2,…, m. Through layer-by-layer weighted product, bj accurately reflects the comprehensive influence of each factor, realizes multi-dimensional data fusion, and optimizes the decision-making model. The evaluation result generation bj value is obtained through fuzzy matrix composition operation, ensuring the accurate allocation of the weights of each factor. W = B × J W = (w1j, w2j,…, wmj), and each wj value is composed of the sum of the products of the corresponding elements of B and J, strictly following the operation logic, accurately reflecting the comprehensive evaluation result, and ensuring the scientific reliability of the decision-making basis. The W value is used as the final decision-making basis, and through systematic operations, multi-dimensional data is effectively integrated.

9. Method of a mental health assessment device based on facial expression recognition and LLM, characterized in that, It includes the following working steps: The first step: The data acquisition module's camera collects the facial video data of the tested person in a specific scenario, collects the voice data of the tested person during the communication process, and converts it into text data. Facial expression features include, but are not limited to, the morphological changes of the eyebrows, eyes, and mouth parts. The second step: The facial expression analysis module uses advanced facial expression recognition algorithms to analyze the collected facial video data frame by frame, extracts facial expression features, and then classifies the facial expression features into common expression categories, including, but not limited to, happy, sad, angry, fearful, and surprised, and counts the frequency and duration of each expression. The facial expression analysis module preprocesses the collected facial video data. The preprocessing includes, but is not limited to, denoising, enhancing contrast, and grayscaling. Then, the video data is decomposed into a series of image frames for frame-by-frame analysis. Next, a face detection algorithm is used to separate the face from the background of the video frame, and accurate face localization is performed through convolutional neural network technology. After detecting the face, features related to expressions are extracted. The features include, but are not limited to, the shape, size, position of the eyes, mouth, and eyebrows, as well as the relative relationships between them. The feature extraction uses geometric feature methods. Then, machine learning algorithms are used to classify the extracted features to determine the expression expressed by the face, and the expression features are classified into common expression categories, such as happy, sad, angry, fearful, and surprised. The frequency and duration of each expression are counted by counting the classified expression frames and recording timestamps, and the changing trends and patterns of expressions are analyzed to provide in-depth insights into the user's emotional state. Step 3: The language analysis module analyzes the collected text data using natural language processing techniques. The language analysis module extracts key information from the text, including but not limited to sentiment tendency, topic content, and language style. The working process of the language analysis module includes text preprocessing, syntactic analysis, sentiment analysis, topic content extraction, language style analysis, and information integration and output. In text preprocessing, irrelevant characters, special symbols, and stop words in the text are removed, and then the text is segmented into individual words or phrases, and a part-of-speech tag is assigned to each word, such as noun, verb, and adjective. Syntactic analysis uses a syntactic analyzer to parse the text and identify the subject, predicate, and object of the sentence. Sentiment analysis uses sentiment analysis algorithms to judge the sentiment tendency of the text, such as positive, negative, or neutral. Topic content extraction uses topic models or keyword extraction algorithms to identify the main topics or themes in the text. Language style analysis analyzes the language style of the text, such as formality, colloquialism, and sense of humor. Language style analysis can be achieved by counting the usage frequencies of specific words or phrases, sentence structures, and other features. Information integration and output integrates the above analysis results to form a report containing key information. The report will be used as one of the inputs for the fusion analysis module and will be jointly used with the analysis results of other modules for mental health assessment. Step 4: The fusion analysis module is electrically connected to the data collection module, the facial expression analysis module, and the language analysis module. The fusion analysis module fuses the facial expression analysis results and the language analysis results, establishes the association between them. When the facial expression shows sadness and the language content also expresses negative emotions, the judgment weight for the negative mental state is enhanced. Step 5: The mental health assessment module conducts mental health assessment based on the results of the fusion analysis using a trained large language model. The large language model gives a mental health assessment report based on the input fusion features, combined with a large amount of mental health data and knowledge, including the classification of mental health status, possible problems, and corresponding suggestions. Step 6: The result display module presents the mental health assessment results to the user in an intuitive way.

Images