A multi-modal based artificial intelligence psychological assessment method and device

Abstract

The application discloses a kind of artificial intelligence psychological evaluation method and equipment based on multi-modal, specifically related to psychological evaluation analysis technical field, for solving the problem of high detection error rate caused by biological feature distortion due to privacy desensitization in existing psychological evaluation;By dynamically generating user behavior adaptive evaluation instruction, collecting facial micro-expression video and quantifying privacy interference level based on regional pixel segmentation, while retaining key micro-expression area features, non-sensitive area is blurred;Synchronously collect speech spectrum waveform and voiceprint modulation and reconstruct fundamental frequency waveform;Extract text sentiment keywords to construct cross-stage sentiment correlation chain to repair semantic break caused by local desensitization;According to the fusion weight of video and voice dynamically adjusted according to privacy interference level Generation stage evaluation parameter and timing label, finally construct psychological state evolution map and output evaluation report, significantly improve emotional feature fidelity and the accuracy of psychological state analysis.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of psychological assessment analysis, more particularly, the present application relates to a multi-modal based artificial intelligence psychological assessment method and device. BACKGROUND

[0002] In the psychological assessment of artificial intelligence, the existing technology generally adopts multi-source data collection methods such as video, voice, and text, and constructs a user psychological portrait through a mode combining biological feature recognition and semantic analysis; under the requirement of medical compliance, strict privacy protection specifications must be followed during data collection, such as desensitization of original data containing personal identity information through face blurring and voiceprint tuning techniques, to meet the storage and use restrictions on biological feature data.

[0003] However, the existing anonymization processing causes structural distortion of key biological features, and local desensitization of text semantics destroys the context emotional association features. This multi-modal feature degradation phenomenon caused by privacy constraints makes the artificial intelligence model unable to accurately capture the subtle changes in the user's psychological state, resulting in a significant increase in detection error rate. SUMMARY

[0004] In order to overcome the above-mentioned defects of the prior art, the embodiments of the present application provide a multi-modal based artificial intelligence psychological assessment method and device to solve the problems raised in the background art.

[0005] To achieve the above-mentioned purpose, the present application provides the following technical solutions:

[0006] A multi-modal based artificial intelligence psychological assessment method, comprising the following steps:

[0007] S1. Generating dynamic assessment instructions according to user behavior history and issuing the user to execute;

[0008] S2. Collecting the user's facial micro-expression video and performing regional pixel segmentation, and generating a privacy interference level based on the pixel activity intensity of the segmented region;

[0009] S3. Collecting the user's voice frequency spectrum waveform and performing voiceprint tuning processing to generate a tuned voice frequency spectrum waveform;

[0010] S4. Extracting emotional keywords from the user's text semantic data, and constructing a cross-stage emotional association chain based on task timing;

[0011] S5. Combining the tuned voice frequency spectrum waveform after the fundamental frequency waveform reconstruction with the cross-stage emotional association chain, adjusting the fusion weight of the video and voice according to the privacy interference level to generate a stage evaluation parameter and a timing label;

[0012] S6, constructing a psychological state evolution map according to the stage evaluation parameters and the timing labels and generating a psychological evaluation report.

[0013] In a preferred embodiment, dynamic evaluation instructions are generated according to the user behavior history and are issued for the user to execute, including:

[0014] Extracting the interaction frequency, task completion time and emotion label in the user historical interaction record as the user behavior history;

[0015] According to the interaction frequency, the difficulty level of the cognitive evaluation task is matched, according to the task completion time, the trigger threshold of the emotion evoking task is divided, and the dynamic evaluation instruction containing the difficulty level and the trigger threshold is generated combined with the emotion label;

[0016] The dynamic evaluation instruction is issued to the user terminal through the visual operation interface, and the task execution countdown module is loaded on the user terminal to start the evaluation process synchronously.

[0017] In a preferred embodiment, the user's facial micro-expression video is collected and regionally pixel segmented, and the privacy interference level is generated based on the pixel activity intensity of the segmented region, including:

[0018] The facial micro-expression video of the user when executing the dynamic evaluation instruction is collected through the camera, and the video segments containing the eye area, the mouth corner area and the forehead area are intercepted;

[0019] The edge detection algorithm is performed on the eye area and the mouth corner area to segment the micro-expression muscle movement unit area, the original pixel value of the micro-expression muscle movement unit area is retained, and the other facial areas are subjected to Gaussian blur encryption processing;

[0020] The pixel gray scale change frequency of the segmented region in the preset time window is calculated as the pixel activity intensity, and when the pixel gray scale change frequency exceeds the preset gray scale threshold, it is marked as a high activity region;

[0021] Based on the proportion and average gray scale change amplitude of the high activity region, the privacy interference level value is generated through a linear mapping function.

[0022] In a preferred embodiment, the user's speech spectrum waveform is collected and the voiceprint modulation is processed to generate a modulated speech spectrum waveform, including:

[0023] The speech signal of the user when executing the dynamic evaluation instruction is collected through the microphone array, and the original speech spectrum waveform is generated through Fourier transform;

[0024] The original speech spectrum waveform is subjected to voiceprint modulation processing, the fundamental frequency is adjusted to a preset gender-neutral frequency band, and the formant distribution is randomly disturbed to eliminate the voiceprint feature;

[0025] The pitch-adjusted speech spectrum waveform is reconstructed to retain the pitch change characteristics and generate a pitch-adjusted speech spectrum waveform which is stored in a buffer queue.

[0026] In a preferred embodiment, the emotional keywords in the text semantic data of the user are extracted, and a cross-stage emotional association chain is constructed based on the task timing, including:

[0027] The text semantic data when the user executes the dynamic evaluation instruction is segmented, the emotional keywords are matched based on the emotional dictionary, and an emotional keyword list is generated;

[0028] According to the task timing timestamp of the dynamic evaluation instruction, the emotional keyword list is divided into multiple subsets according to the task stage;

[0029] The co-occurrence frequency and position offset of emotional keywords between adjacent task stage subsets are counted, and when the co-occurrence frequency exceeds a preset threshold, a cross-stage emotional association chain is established.

[0030] In a preferred embodiment, the pitch-adjusted speech spectrum waveform after the fundamental frequency waveform reconstruction is combined with the cross-stage emotional association chain, the fusion weight of the video and the voice is adjusted according to the privacy interference level to generate a stage evaluation parameter and a timing label, including:

[0031] Based on the privacy interference level value, dynamic compensation is applied to the video feature vector after regional pixel segmentation;

[0032] Based on the co-occurrence frequency and position offset in the cross-stage emotional association chain, a semantic consistency parameter is generated, and the intonation feature vector of the pitch-adjusted speech spectrum waveform is contextually calibrated;

[0033] The compensated video feature vector and the calibrated intonation feature vector are aligned between modal events according to the task timing timestamp, the stage evaluation parameter is generated based on the continuity of the emotional feature evolution in the aligned events, and the task timing association label is marked according to the distribution density of the position offset in the cross-stage emotional association chain.

[0034] In a preferred embodiment, the compensation strength of the dynamic compensation is inversely related to the privacy interference level value, and the feature contribution of the eye area is enhanced to offset the blurring loss when the interference level is high.

[0035] The calibration direction of the context calibration is consistent with the keyword emotional polarity in the cross-stage emotional association chain.

[0036] In a preferred embodiment, a psychological state evolution map is constructed according to the stage evaluation parameter and the timing label, and a psychological evaluation report is generated, including:

[0037] Integrate the stage evaluation parameters to the time axis according to the task time sequence timestamp, and generate the node weight of the psychological state evolution graph based on the continuity of the emotional feature evolution;

[0038] Generate the connection strength between nodes according to the distribution density and standard deviation of the position offset in the cross-stage emotional association chain;

[0039] Generate the psychological assessment report based on the node weight and the connection strength between nodes. The psychological assessment report includes the depression tendency fluctuation curve and the anxiety state heat map.

[0040] In a preferred embodiment, the connection strength between nodes is dynamically adjusted by the ratio of the distribution density and the standard deviation. When the distribution density increases, the connection strength between nodes is enhanced. When the standard deviation increases, the connection strength between nodes is weakened.

[0041] In another aspect, the present application provides a multi-modal artificial intelligence psychological assessment device, comprising:

[0042] A behavior instruction generation module generates dynamic assessment instructions according to the user behavior history and issues the instructions for the user to execute;

[0043] A facial micro-expression grading module collects facial micro-expression videos of the user and performs regional pixel segmentation, and generates a privacy interference level based on the pixel activity intensity of the segmented regions;

[0044] A voiceprint spectrum modulation module collects the voice spectrum waveform of the user and performs voiceprint modulation processing to generate a modulated voice spectrum waveform;

[0045] A cross-stage emotional chain construction module extracts emotional keywords from the text semantic data of the user and constructs a cross-stage emotional association chain based on the task time sequence;

[0046] A multi-modal fusion weight adjustment module combines the modulated voice spectrum waveform after the fundamental frequency waveform reconstruction with the cross-stage emotional association chain, adjusts the fusion weight of the video and the voice according to the privacy interference level to generate stage evaluation parameters and time sequence labels;

[0047] A cardiogram evolution report module constructs a psychological state evolution graph according to the stage evaluation parameters and the time sequence labels and generates a psychological assessment report.

[0048] Compared with the prior art, the present application has the following beneficial effects:

[0049] 1. By regional pixel segmentation, the key muscle movement unit of facial micro-expression is accurately identified, the non-key area is locally encrypted while the core biological features are reserved, the identity features are eliminated but the tone emotion rules are maintained by combining the reconstructed technology of the fundamental frequency waveform after voiceprint tone conversion, the effective balance between privacy desensitization and feature fidelity is realized, the feature distortion caused by traditional global blurring is avoided, and the sensitivity of emotion state judgment is significantly improved.

[0050] 2. By constructing and dynamically fusing weight adjustment through cross-stage emotion association chain, the problems of multi-modal data semantic rupture and credibility imbalance are solved, based on task timing emotion keyword co-occurrence analysis and position offset statistics, the context association rupture caused by text desensitization is repaired to ensure the coherence of emotion context; at the same time, the fusion weight of video and voice is adaptively adjusted according to the privacy interference level, the feature contribution of high credibility data source is preferentially reserved, the error accumulation of traditional single-modal analysis is effectively suppressed, and combined with the dynamic modeling ability of psychological state evolution map, the spatio-temporal rules of emotion evolution can be intuitively presented, and the risk of misjudgment is greatly reduced. BRIEF DESCRIPTION OF DRAWINGS

[0051] Figure 1 A flowchart of a kind of multi-modal artificial intelligence psychological evaluation method according to the present application;

[0052] Figure 2 A structural schematic diagram of a kind of multi-modal artificial intelligence psychological evaluation device according to the present application. DETAILED DESCRIPTION

[0053] The technical solutions in the embodiments of the present application will be described clearly and completely in combination with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0054] Embodiment 1: Figure 1 A kind of multi-modal artificial intelligence psychological evaluation method according to the present application is given, which includes the following steps:

[0055] S1, generating dynamic evaluation instruction according to user behavior history and issuing user to execute;

[0056] S2, collect the facial micro-expression video of user and carry out regional pixel segmentation, generate privacy interference level based on the pixel activity intensity of segmented area;

[0057] S3, collect the voice frequency spectrum waveform of user and carry out voiceprint tone conversion to generate variable tone voice frequency spectrum waveform;

[0058] S4, extracting emotional keywords in the text semantic data of the user, and constructing a cross-stage emotional association chain based on a task time sequence;

[0059] S5, combining the spectrum waveform of the pitch-adjusted voice after reconstruction with the cross-stage emotional association chain, adjusting the fusion weight of the video and the voice according to the privacy interference level to generate a stage evaluation parameter and a time sequence label;

[0060] S6, constructing a psychological state evolution map according to the stage evaluation parameter and the time sequence label and generating a psychological evaluation report.

[0061] S1, generating a dynamic evaluation instruction according to a user behavior history and issuing the user to execute, including:

[0062] Extracting the interaction frequency, task completion time and emotion label in the user's historical interaction record as the user behavior history;

[0063] According to the interaction frequency, the difficulty level of the cognitive evaluation task is matched, according to the task completion time, the trigger threshold of the emotion induction task is divided, and the dynamic evaluation instruction containing the difficulty level and the trigger threshold is generated combined with the emotion label;

[0064] The dynamic evaluation instruction is issued to the user terminal through the visual operation interface, and the task execution countdown module is loaded on the user terminal to start the evaluation process synchronously.

[0065] The extraction of the user behavior history is realized by recording the user's interaction behavior data in the historical evaluation task, which includes but is not limited to the number of times the user triggers the evaluation task, the duration from the start to the submission of a single task, and the emotion state label annotated by the user. The statistical method of interaction frequency is to count the number of times the user initiates the evaluation task in the past 1 week, for example, when the user completes the evaluation task more than a preset threshold in the past 1 week, the interaction frequency is marked as high frequency, and below the threshold is marked as low frequency.

[0066] The calculation of the task completion time is obtained by recording the time stamp difference from the task interface loading completion to the final submission result, for example, the user's interval time from starting to answer to clicking the submit button in the cognitive evaluation task is 3 minutes and 20 seconds. The emotion label is obtained by checking the self-emotion state option through the interface when the user executes the historical task, or by emotion analysis result of the user's historical voice and text data through artificial intelligence model, for example, the user checks "anxiety" or the model identifies the rapid tone feature in the voice spectrum and marks it as anxiety tendency.

[0067] The difficulty level matching rule of the cognitive assessment task is dynamically adjusted according to the interaction frequency, when the interaction frequency is high, a cognitive assessment task containing high-order logical reasoning questions, such as figure rule identification or number sequence deduction questions, is generated; when the interaction frequency is low, a cognitive assessment task containing basic cognitive test questions, such as simple arithmetic or vocabulary memory questions, is generated.

[0068] The trigger threshold of the emotion induction task is divided according to the statistical result of the task completion time, when the task completion time is lower than 20% of the average completion time of the same type of task, a high-intensity emotion induction task is triggered, such as rapid decision-making stress test or sudden scene simulation; when the task completion time is higher than 20% of the average completion time of the same type of task, a low-intensity emotion induction task is triggered, such as relaxation scene guidance or soothing music playing. The generation of dynamic evaluation instructions is combined with the matched difficulty level and the trigger threshold, and is associated with the dominant emotion type in the user emotion label, for example, when the anxiety tendency in the user emotion label accounts for more than 50%, the guiding task content for relieving anxiety is added in the dynamic evaluation instructions.

[0069] The design of the visual operation interface includes a task instruction display area, a countdown display area and an operation button area, the task instruction display area presents the specific content of the dynamic evaluation instructions in the form of text and pictures, such as using flowchart to explain the answering steps of the cognitive assessment task, or demonstrating the scene setting of the emotion induction task through animation.

[0070] The loading of the task execution countdown module is realized by synchronizing the system clock of the user terminal, after the user clicks the start button, the countdown module displays the remaining time in seconds, and automatically resets the countdown time when switching between different task stages, for example, the cognitive assessment task stage is set to 5 minutes, and the emotion induction task stage is set to 3 minutes. The synchronous start of the evaluation process is realized by network time protocol calibration between the user terminal and the server, which ensures the consistency of the evaluation progress time axis of multiple terminal users, for example, when the time deviation between the user terminal and the server exceeds 500 milliseconds, the countdown module is automatically corrected based on the server time.

[0071] S2, collect the facial micro-expression video of the user and perform regional pixel segmentation, generate a privacy interference level based on the pixel activity intensity of the segmented region, including:

[0072] Collect the facial micro-expression video of the user when executing the dynamic evaluation instruction through the camera, and intercept the video clips containing the eye area, the corner of the mouth area and the forehead area;

[0073] Perform edge detection algorithm processing on the eye area and the corner of the mouth area to segment the micro-expression muscle movement unit area, retain the original pixel value of the micro-expression muscle movement unit area and perform Gaussian blur encryption processing on other facial areas.

[0074] The pixel gray level change frequency of the segmented region within a preset time window is calculated as the pixel activity intensity, and when the pixel gray level change frequency exceeds a preset gray level threshold, it is marked as a high activity region;

[0075] Based on the proportion and average gray level change amplitude of the high activity region, a linear mapping function is used to generate a privacy interference level value.

[0076] The collection of facial micro-expression videos is achieved by real-time shooting of facial dynamic images of the user performing dynamic evaluation instructions through the camera of the user terminal. The video capture range covers the preset angle of the user's face front and side, for example, the camera automatically focuses on the line connecting the user's eyes as the horizontal reference line, ensuring that the eye area, mouth area, and forehead area are completely in the shot. The video segment is selected according to the key frame sequence of the task type of the dynamic evaluation instruction, for example, in the emotion induction task stage, the video segment from 3 seconds after the task starts to 1 second before the task ends is captured, ensuring the integrity of the micro-expression features. The eye area is defined as a rectangular area with the pupils of the eyes as the center and extending outward by 2 cm, the mouth area is defined as a rectangular area with the center line of the lips as the reference and extending to both sides by 3 cm, and the forehead area is defined as a rectangular area from the hairline to the brow.

[0077] The edge detection algorithm is processed for the video frames of the eye area and the mouth area. First, the video frame is converted to a gray-scale image, and then the boundary profile of the micro-expression muscle movement unit is identified by detecting the gray value mutation position, such as the contraction boundary of the orbicularis oculi or the deformation boundary of the orbicularis oris. The segmentation of the micro-expression muscle movement unit area is achieved by retaining the pixels within the boundary profile, and the Gaussian blur encryption processing of the other facial areas uses a blur radius of 15 pixels to blur the non-segmented areas, such as the forehead and cheek areas, to eliminate identity features. The original pixel value retention range of the segmented area includes all channel values in the RGB color space, such as the red channel value of the eye area for blood vessel dilation state analysis.

[0078] The calculation of the pixel gray level change frequency is achieved by counting the gray value difference of the segmented region in consecutive video frames, and the preset time window is set to 30 frames of video data within 1 second, for example, the gray average value of the eye area is counted every frame within 1 second and the difference between adjacent frames is calculated. When the difference exceeds the preset gray level threshold and the change frequency reaches 30 times per second, the region is marked as a high activity region, for example, the user's frequent blinking causes the gray level change frequency of the eye area to reach 35 times per second. The average gray level change amplitude is calculated by accumulating all differences that exceed the preset gray level threshold and taking the average, for example, a certain segmented region has 20 times of difference exceeding the preset gray level threshold within 1 second, the cumulative value is 200, and the average amplitude is 10.

[0079] The privacy interference level value is generated by linear mapping function to weight and sum the proportion of high activity area and the average gray scale change amplitude in the interval of 0 to 100, for example, when the proportion of high activity area is 30% and the average gray scale change amplitude is 50, the privacy interference level value is (30% x 0.6 + 50 x 0.4) x 100 = 70. The weight coefficient of the linear mapping function is set according to the biological characteristics of the micro-expression muscle movement unit, for example, the weight coefficient of the eye area is higher than that of the mouth area to reflect its higher emotional expression sensitivity.

[0080] S3, collecting the voice spectrum waveform of the user and performing voiceprint pitch shifting processing to generate a pitch shifted voice spectrum waveform, comprising:

[0081] Collecting the voice signal of the user performing the dynamic evaluation instruction through the microphone array, and generating the original voice spectrum waveform through Fourier transform;

[0082] Performing voiceprint pitch shifting processing on the original voice spectrum waveform, adjusting the fundamental frequency to the preset gender-neutral frequency band and randomly disturbing the formant distribution to eliminate the voiceprint characteristics;

[0083] Reconstructing the fundamental frequency waveform of the pitch shifted voice spectrum waveform, retaining the intonation change characteristics and generating a pitch shifted voice spectrum waveform stored in the buffer queue.

[0084] The collection of voice signals is realized through the microphone array deployed on the user terminal. The layout of the microphone array adopts a ring-symmetric structure, ensuring that the user sound field range can be covered at an azimuth angle of 0 to 180 degrees when the user performs the dynamic evaluation instruction. For example, when the user faces the terminal screen, the microphone array preferentially collects voice signals within a 120-degree range in front. The sampling rate of the voice signal is set to 16 kHz to cover the fundamental frequency and formant frequency range of human voice. For example, the fundamental frequency range of adult males is usually 85 Hz to 180 Hz, and the fundamental frequency range of adult females is usually 165 Hz to 255 Hz. The Fourier transform processing adopts a Hanning window function for frame processing of the voice signal, with a frame length of 25 milliseconds and a frame shift of 10 milliseconds. The time domain voice signal is converted into a frequency domain voice spectrum waveform, for example, the time domain waveform of the user reading the emotional induction task text is converted into a frequency spectrum graph containing the fundamental frequency, formant, etc.

[0085] The fundamental frequency adjustment of the voiceprint pitch-shifting process is achieved through a linear scaling algorithm, which maps the original fundamental frequency to a pre-set gender-neutral frequency range of 180-220 Hz. This range covers the intersection of male and female voice fundamental frequencies. For example, when the user's original fundamental frequency is 200 Hz, it remains unchanged; when the original fundamental frequency is 150 Hz, it is raised to 190 Hz; and when the original fundamental frequency is 250 Hz, it is lowered to 210 Hz. The random perturbation of the formant distribution is achieved by shifting the formant peak frequency in the 1-3 kHz frequency range by ±50 Hz. For example, the original formant peak of 2500 Hz is randomly adjusted to 2450 Hz or 2550 Hz. The amount of shift is determined based on speech intelligibility experiments to ensure semantic integrity. The elimination of voiceprint features is achieved through double adjustment of the fundamental frequency and formants, so that the pitch-shifted speech cannot be associated with the user's identity through voiceprint recognition algorithms. For example, the pitch-shifted speech spectrum waveform generated by the same user in different tasks cannot be matched to the same identity in the voiceprint database.

[0086] The fundamental frequency waveform reconstruction process extracts the fundamental frequency profile from the pitch-shifted speech spectrum waveform and smooths it using a cubic spline interpolation algorithm to eliminate the jagged fluctuations introduced by pitch-shifting. For example, the user's voice tone rises when reading a question, and the corresponding fundamental frequency curve fluctuation is preserved, while the unnatural jumps caused by pitch-shifting are smoothed and corrected. The preservation of intonation changes is achieved by comparing the relative change rates of the fundamental frequency profile before and after reconstruction. For example, when the user's voice fundamental frequency rises from 200 Hz to 240 Hz within 1 second, the reconstructed fundamental frequency waveform still maintains a 25% relative increase. The storage of pitch-shifted speech spectrum waveform is managed by a buffer queue, which caches at least 5 seconds of speech data in timestamp order to ensure time-domain alignment with the facial micro-expression video. For example, when the video data lags behind due to network delay, the buffer queue provides the corresponding time window of speech data for synchronous analysis. The cache duration is set based on typical network delay statistics.

[0087] S4, extract the emotional keywords in the text semantic data of the user, and construct a cross-stage emotional association chain based on the task timing, including:

[0088] Perform word segmentation on the text semantic data when the user executes the dynamic evaluation instruction, match emotional keywords based on the emotional dictionary, and generate an emotional keyword list;

[0089] According to the task timing timestamp of the dynamic evaluation instruction, divide the emotional keyword list into multiple subsets according to the task stages;

[0090] Statistical co-occurrence frequency and position offset of emotional keywords between adjacent task stage subsets, and when the co-occurrence frequency exceeds a pre-set threshold, establish a cross-stage emotional association chain.

[0091] The extraction of text semantic data is achieved by the text content input by the user when executing the dynamic evaluation instruction, which includes but is not limited to the free description text of the user in the emotion induction task, the answer text of the selection question in the cognitive evaluation task, and the feedback opinion text in the task gap.

[0092] The implementation of word segmentation processing adopts a word segmentation tool based on dictionary matching, which divides the text content into independent lexical units according to the word granularity, for example, the user input "I feel anxious but try to calm down" is divided into "I / feel / anxious / but / try / calm down". The construction of the emotional dictionary is based on the publicly available emotional vocabulary library, such as the dictionary list containing positive emotional words (such as "happy" "relaxed"), negative emotional words (such as "anxious" "depressed") and degree adverbs (such as "very" "slightly"). The matching of emotional keywords is achieved by traversing the word segmentation results and the emotional dictionary entries for full matching, for example, when the word segmentation result is "anxious", the negative emotional keyword is matched from the emotional dictionary and added to the list.

[0093] The acquisition of task timing timestamp is based on the system timing after the dynamic evaluation instruction is issued to the user terminal, and the start timestamp and the end timestamp of each task stage are recorded by the server, for example, the timestamp of the emotion induction task stage is from the 10th second to the 50th second after starting, and the cognitive evaluation task stage is from the 55th second to the 100th second.

[0094] The division of the emotional keyword list is based on the timestamp range of the task stage to intercept the word segmentation results of the corresponding text content, for example, the emotional keywords in the 10th second to the 50th second are classified into the emotion induction task subset, and the keywords in the 55th second to the 100th second are classified into the cognitive evaluation task subset. The division rule of the subset also includes the logical association of the task type, for example, the word "difficult" filled in by the user in the cognitive evaluation task and the word "stress" in the subsequent emotion induction task are classified into adjacent subsets.

[0095] The co-occurrence frequency statistics of adjacent task stage subsets are achieved by traversing the emotional keywords of the two subsets, calculating the number of times the same keyword appears in adjacent stages, for example, "anxiety" appears 3 times in the emotion induction task subset, and "anxiety" appears 2 times in the cognitive evaluation task subset, then the co-occurrence frequency is 2 times. The calculation of the position offset is achieved by recording the difference value of the first appearance position of the same keyword in adjacent subsets, for example, "anxiety" appears for the first time in the 5th word position in the emotion induction task subset, and appears for the first time in the 3rd word position in the cognitive evaluation task subset, then the position offset is 2.

[0096] The setting of the preset threshold is dynamically adjusted according to the total number of keywords in the task stage, for example, when the total number of keywords in the adjacent subset exceeds 20, the co-occurrence frequency threshold is set to 3 times; when the total number of keywords is less than 20, the threshold is set to 2 times. The establishment of the cross-stage emotional association chain is realized by connecting the keywords and their offsets that meet the threshold condition, for example, when the co-occurrence frequency of "anxiety" is 2 times and the offset is 2, the association chain of "anxiety-emotion induction-cognitive assessment" is generated.

[0097] S5, the spectrum waveform of the pitch-adjusted voice after the reconstruction of the fundamental frequency waveform is combined with the cross-stage emotional association chain, the fusion weight of the video and the voice is adjusted according to the privacy interference level to generate a stage evaluation parameter and a time sequence label, including:

[0098] Based on the privacy interference level value, dynamic compensation is applied to the video feature vector after regional pixel segmentation, and the compensation strength is inversely related to the privacy interference level value. When the interference level is high, the feature contribution of the eye area is enhanced to offset the loss of blurring;

[0099] Based on the co-occurrence frequency and position offset in the cross-stage emotional association chain, a semantic consistency parameter is generated, and the tone feature vector of the pitch-adjusted voice spectrum waveform is contextually calibrated. The calibration direction of the context calibration is consistent with the emotional polarity of the keywords in the cross-stage emotional association chain;

[0100] The compensated video feature vector and the calibrated tone feature vector are aligned between modalities according to the task time sequence timestamp, the stage evaluation parameter is generated based on the continuity of the emotional feature evolution in the aligned event, and the task time sequence association label is marked according to the distribution density of the position offset in the cross-stage emotional association chain.

[0101] The application of the privacy interference level value is realized by dynamically compensating the feature contribution of the eye area in the video feature vector after regional pixel segmentation. The strength of dynamic compensation is inversely related to the privacy interference level value, which means that the higher the privacy interference level value, the greater the feature compensation strength of the eye area.

[0102] The calculation rule of the feature compensation strength is to determine the compensation coefficient according to the percentile of the privacy interference level value, for example, when the privacy interference level value is 70, the feature contribution of the eye area is increased to 1.5 times of the original value to offset the loss of details caused by Gaussian blur processing. The execution of the compensation operation is realized by adjusting the weight distribution of the video feature vector, for example, in the video feature vector, the weight of the gray change frequency feature of the eye area is increased, and the weights of other regions are proportionally decreased according to the interference level.

[0103] The application of the co-occurrence frequency and the position offset in the cross-stage emotional association chain is realized by generating a semantic consistency parameter, the value of the semantic consistency parameter is positively correlated with the co-occurrence frequency and is negatively correlated with the position offset. The calculation method of the semantic consistency parameter is to divide the co-occurrence frequency by the result of the position offset plus 1, for example, when the co-occurrence frequency is 5 times and the position offset is 2, the semantic consistency parameter is 5 / (2+1)=1.67.

[0104] The semantic consistency parameter is used for context calibration of the intonation feature vector of the variable pitch speech spectrum waveform, the calibration direction is determined according to the keyword emotional polarity in the cross-stage emotional association chain, for example, when the proportion of negative emotional keywords in the emotional chain is more than 60%, the calibration of the pitch fluctuation amplitude of the intonation feature vector is increased by 20% to strengthen the consistency of emotional expression.

[0105] The inter-modal event alignment of the compensated video feature vector and the calibrated intonation feature vector is realized by accurate matching of the task timing timestamp, the accuracy of the task timing timestamp is set to millisecond level to ensure the accuracy of the event alignment.

[0106] The logic of event alignment is to identify the timing relationship between the micro-expression event (such as eye muscle contraction) in the video feature vector and the intonation event (such as pitch jump) in the intonation feature vector in the same or adjacent time window, for example, when the time difference between the eye muscle contraction event and the pitch jump event is less than 500 milliseconds, it is determined as an emotional synchronization event. The continuity analysis of emotional feature evolution is realized by statistical distribution density of synchronization events on the task timing axis, for example, in the emotional induction task stage, if more than 3 synchronization events are detected every 10 seconds, it is marked as an emotional high continuity stage.

[0107] The generation of the stage evaluation parameter is realized by the continuity index of the emotional feature evolution, the continuity index is the product of the synchronization event distribution density and the semantic consistency parameter, for example, the distribution density is 0.3 times / second and the semantic consistency parameter is 1.5, the continuity index is 0.45.

[0108] The marking rule of the task timing association label is based on the distribution density of the position offset in the cross-stage emotional association chain, the distribution density is calculated by counting the occurrence frequency of the position offset in the preset time window, for example, in the cognitive evaluation task stage, if the standard deviation of the position offset is less than 1, it is marked as “high association”, and if the standard deviation is greater than or equal to 1, it is marked as “low association”.

[0109] S6, constructing a psychological state evolution map and generating a psychological test report according to the stage evaluation parameter and the timing label, including:

[0110] The stage evaluation parameters are integrated into a time axis according to task time sequence timestamps, and the node weights of the psychological state evolution graph are generated based on the continuity of the emotional feature evolution;

[0111] The connection strength between nodes is generated according to the distribution density and standard deviation of the position offset in the cross-stage emotional association chain, and the connection strength between nodes is dynamically adjusted through the ratio of the distribution density and the standard deviation. When the distribution density increases, the connection strength between nodes is enhanced, and when the standard deviation increases, the connection strength between nodes is weakened.

[0112] The psychological assessment report is generated based on the node weights and the connection strength between nodes. The psychological assessment report includes a depression tendency fluctuation curve and an anxiety state heat map.

[0113] The integration of the stage evaluation parameters is realized through the order of the task time sequence timestamps, and the accuracy of the task time sequence timestamps is set to milliseconds to ensure the accuracy of data alignment. In the integration process, the evaluation parameters corresponding to different task stages are arranged in the time axis according to the timestamp order, for example, the timestamp of the emotional induction task stage is from the 10th second to the 50th second, the cognitive assessment task stage is from the 55th second to the 100th second, and the evaluation parameters are filled in sequence according to this time range.

[0114] The node weights of the psychological state evolution graph are generated based on the continuity index of the emotional feature evolution in the stage evaluation parameters. The higher the value of the continuity index, the greater the node weight, for example, when the continuity index of the emotional induction task stage is 0.8, the weight of the corresponding node is set to 0.8, and when the continuity index of the cognitive assessment task stage is 0.5, the node weight is set to 0.5. The specific value of the node weight is converted to the interval of 0 to 1 through a linear mapping function, for example, the continuity index 0.45 is mapped to the node weight 0.45.

[0115] The calculation of the distribution density and standard deviation of the position offset in the cross-stage emotional association chain is realized by counting the position difference of the same emotional keyword in different task stages. The calculation of the distribution density is to count the number of position offsets in a preset time window, for example, in the emotional induction task stage, if the word "anxiety" appears 3 times in adjacent subsets with position offsets of 1, 2, and 3, the distribution density is 3 times / 10 seconds. The calculation of the standard deviation is based on the dispersion degree of the position offset, for example, the standard deviation of the offsets 1, 2, and 3 is about 0.82. The generation rule of the connection strength between nodes is the ratio of the distribution density and the standard deviation. When the distribution density increases or the standard deviation decreases, the connection strength is enhanced, for example, when the distribution density is 3 times / 10 seconds and the standard deviation is 0.82, the connection strength is 3 / 0.82≈3.66; if the distribution density is 2 times / 10 seconds and the standard deviation is 1.5, the connection strength is 2 / 1.5≈1.33.

[0116] The generation of the psychological evaluation report is realized by combining the node weight and the connection strength into a visual chart. The drawing of the depression tendency fluctuation curve is based on the attenuation trend of the node weight with the timestamp interval, and the attenuation trend is determined by calculating the difference value of the adjacent node weights, for example, if the node weight decreases from 0.8 to 0.5 within a timestamp interval of 10 seconds, the attenuation rate is (0.8-0.5) / 10=0.03 / s.

[0117] The rendering of the anxiety state heat map is based on the spatial distribution density of the node weight on the task timing time axis, and the calculation of the distribution density is the cumulative value of the node weight within a statistical unit time, for example, the node weight is accumulated to 2.4 within 10 seconds in the emotional induction task stage, and the distribution density is 0.24 / s. The color gradient of the heat map is dynamically adjusted according to the distribution density value, and the higher the density, the deeper the color, for example, the density of 0.24 / s corresponds to light red, and the density of 0.5 / s corresponds to dark red.

[0118] Embodiment 2: Figure 2 A structural schematic diagram of an artificial intelligence psychological evaluation device based on multiple modes is given, an artificial intelligence psychological evaluation device based on multiple modes, comprising:

[0119] The behavior instruction generation module generates dynamic evaluation instructions according to the user behavior history and issues the user to execute;

[0120] The face micro-privacy grading module collects the user's face micro-expression video and performs regional pixel segmentation, and generates a privacy interference level based on the pixel activity intensity of the segmented region;

[0121] The voiceprint spectrum modulation module collects the user's voice spectrum waveform and performs voiceprint modulation processing to generate a modulated voice spectrum waveform;

[0122] The cross-stage emotion chain construction module extracts emotional keywords from the user's text semantic data, and constructs a cross-stage emotional association chain based on the task timing;

[0123] The multi-modal fusion weight adjustment module combines the modulated voice spectrum waveform after the reconstruction of the fundamental frequency waveform with the cross-stage emotional association chain, adjusts the fusion weight of the video and the voice according to the privacy interference level to generate a stage evaluation parameter and a timing label;

[0124] The cardiogram evolution report module constructs a psychological state evolution graph according to the stage evaluation parameter and the timing label and generates a psychological evaluation report.

[0125] The above formulas are dimensionless values, and the formulas are obtained by software simulation of a large amount of data to obtain a formula closest to the actual situation, and the preset parameters and threshold values in the formula are set by the person skilled in the art according to the actual situation.

[0126] It should be noted that the application can be deployed in the device itself to realize embedded application, or run on PC terminal or other terminal with user interface, so as to meet various hardware environment and use requirements.

[0127] The above-described embodiments can be implemented in whole or in part by software, hardware, firmware, or any combination thereof. When implemented by software, the above-described embodiments can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, the processes or functions described in the embodiments of the present application are generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another, for example, the computer instructions can be transmitted from one website, computer, server or data center to another website, computer, server or data center by wired (for example, infrared, wireless, microwave, etc.) mode. The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server, data center, etc. containing one or more available medium collections. The available medium can be a magnetic medium (for example, floppy disk, hard disk, magnetic tape), an optical medium (for example, DVD), or a semiconductor medium. The semiconductor medium can be a solid state disk.

[0128] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the above-described system, device and module can refer to the corresponding process in the foregoing method embodiments, which will not be described here.

[0129] In several embodiments provided in the present application, it should be understood that the disclosed system, device and method can be implemented in other ways. For example, the above-described device embodiments are only schematic, for example, the division of the modules is only a logical function division, and actual implementation can have another division manner, for example, a plurality of modules or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the displayed or discussed each other can be indirect coupling or communication connection through some interfaces, devices or modules, which can be electrical, mechanical or other forms.

[0130] The modules described as separate components may or may not be physically separate, and the components shown as modules may or may not be physical modules, and may be located in one place or distributed on multiple network modules. Part or all of the modules can be selected according to actual needs to achieve the purpose of the embodiment scheme.

[0131] In addition, the functional modules in each embodiment of the present application can be integrated in one processing module, or each module can exist physically alone, or two or more modules can be integrated in one module.

[0132] The functions, if realized in the form of software functional modules and sold or used as independent products, can be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the present application essentially or the part that contributes to the prior art or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium, including a plurality of instructions for causing a computer device (which can be a personal computer, a server or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The foregoing storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), magnetic disk or optical disk and various program code storage media.

[0133] The above is only a specific embodiment of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art can easily think of changes or replacements within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

[0134] Finally: the above is only the preferred embodiment of the present application, and is not used to limit the present application, any modification, equivalent replacement, improvement, etc. within the spirit and principles of the present application, should be included in the protection scope of the present application.

Claims

1. A multimodal artificial intelligence psychological assessment method, characterized in that: The steps include: S1. Generate dynamic evaluation instructions based on user behavior history and send them to users for execution; S2. Collect the user's facial micro-expression video and perform pixel segmentation. Generate a privacy interference level based on the pixel activity intensity of the segmented area, including: The camera collects the user's facial micro-expression video when executing the dynamic assessment command, and captures the video clips including the eye area, mouth corner area and forehead area; The eye area and mouth corner area are processed by edge detection algorithm to segment the micro-expression muscle motor unit area, retaining the original pixel value of the micro-expression muscle motor unit area and performing Gaussian blur encryption on other facial areas; Calculate the pixel grayscale change frequency of the segmented area within the preset time window as the pixel activity intensity. When the pixel grayscale change frequency exceeds the preset grayscale threshold, it is marked as a high-activity area. Based on the proportion of high-activity areas and the average grayscale change amplitude, a privacy interference level value is generated through a linear mapping function; S3. Collect the user's voice spectrum waveform and perform voiceprint pitch shifting processing to generate a pitch-shifted voice spectrum waveform; S4, extract emotional keywords from the user's text semantic data and build a cross-stage emotional association chain based on the task sequence; S5. Combine the pitch-shifted speech spectrum waveform reconstructed from the fundamental frequency waveform with the cross-stage emotional association chain, and adjust the fusion weight of video and speech based on the privacy interference level to generate stage-by-stage evaluation parameters and time series labels, including: Apply dynamic compensation to the video feature vector after regional pixel segmentation based on the privacy interference level value; Semantic consistency parameters are generated based on the co-occurrence frequency and position offset in the cross-stage emotional association chain, and the intonation feature vector of the pitch-shifted speech spectrum waveform is contextually calibrated. The compensated video feature vector and the calibrated intonation feature vector are aligned with each other according to the task timing timestamp. The stage-by-stage evaluation parameters are generated based on the continuity of the emotional feature evolution in the aligned events. The task timing association labels are then marked according to the distribution density of the position offset in the cross-stage emotional association chain. S6. Construct a psychological state evolution map based on the stage-by-stage evaluation parameters and time series labels and generate a psychological assessment report.

2. The multimodal artificial intelligence psychological assessment method according to claim 1, characterized in that: Generate dynamic evaluation instructions based on user behavior history and send them to users for execution, including: Extract interaction frequency, task completion time, and emotional labels from user historical interaction records as user behavior history; Match the difficulty level of cognitive assessment tasks based on interaction frequency, divide the trigger threshold of emotion-inducing tasks based on task completion time, and generate dynamic assessment instructions containing difficulty level and trigger threshold based on emotion tags; The dynamic evaluation instructions are sent to the user terminal through the visual operation interface, and the task execution countdown module is loaded on the user terminal to synchronously start the evaluation process.

3. The multimodal artificial intelligence psychological assessment method according to claim 2, characterized in that: Collect the user's voice spectrum waveform and perform voiceprint pitch shifting processing to generate a pitch-shifted voice spectrum waveform, including: The microphone array is used to collect the user's voice signal when executing the dynamic assessment command, and the original voice spectrum waveform is generated through Fourier transform; Perform voiceprint pitch shifting on the original speech spectrum waveform, adjust the fundamental frequency to the preset gender-neutral frequency band, and randomly perturb the formant distribution to eliminate voiceprint characteristics; The fundamental frequency waveform of the pitch-shifted speech spectrum waveform is reconstructed, the tone change characteristics are retained, and the pitch-shifted speech spectrum waveform is generated and stored in the buffer queue.

4. The multimodal artificial intelligence psychological assessment method according to claim 3, characterized in that: Extract emotional keywords from the user's text semantic data and build a cross-stage emotional association chain based on the task sequence, including: Perform word segmentation on the text semantic data when the user executes the dynamic evaluation instruction, match the emotional keywords based on the emotional dictionary and generate an emotional keyword list; Based on the task timing timestamps of the dynamic evaluation instructions, the list of emotional keywords is divided into multiple subsets according to the task stages; The co-occurrence frequency and position offset of emotional keywords between adjacent task stage subsets are counted, and a cross-stage emotional association chain is established when the co-occurrence frequency exceeds a preset threshold.

5. The multimodal artificial intelligence psychological assessment method according to claim 4, characterized in that: The intensity of dynamic compensation is inversely correlated with the privacy interference level. When the interference level is high, the contribution of features in the periocular area is enhanced to offset the blurring loss. The calibration direction of contextual calibration is consistent with the keyword emotion polarity in the cross-stage emotion association chain.

6. The multimodal artificial intelligence psychological assessment method according to claim 5, characterized in that: Construct a psychological state evolution map based on phased assessment parameters and time series labels and generate a psychological assessment report, including: Integrate the phased evaluation parameters into the timeline according to the task sequence timestamps, and generate the node weights of the mental state evolution graph based on the continuity of the emotional feature evolution; Generate the connection strength between nodes based on the distribution density and standard deviation of the position offset in the cross-stage emotional association chain; Generate a psychological assessment report based on node weights and inter-node connection strengths; the psychological assessment report includes a depression tendency fluctuation curve and an anxiety state heat map.

7. The multimodal artificial intelligence psychological assessment method according to claim 6, characterized in that: The connection strength between nodes is dynamically adjusted by the ratio of distribution density to standard deviation. When the distribution density increases, the connection strength between nodes is enhanced, and when the standard deviation increases, the connection strength between nodes is weakened.

8. A multimodal artificial intelligence psychological assessment device, used to implement the multimodal artificial intelligence psychological assessment method according to any one of claims 1 to 7, characterized in that: include: Behavior instruction generation module: generates dynamic evaluation instructions based on user behavior history and sends them to users for execution; Facial Micro-Privacy Grading Module: This module collects user facial micro-expression videos and performs pixel segmentation, generating a privacy interference level based on the pixel activity intensity of the segmented regions. Voiceprint spectrum modulation module: collects the user's voice spectrum waveform and performs voiceprint pitch shifting to generate a pitch-shifted voice spectrum waveform; Cross-stage emotion chain construction module: extracts emotional keywords from users' text semantic data and constructs cross-stage emotion association chains based on task sequence; Multi-modal fusion weight adjustment module: This module combines the pitch-shifted speech spectrum waveform reconstructed from the fundamental frequency waveform with the cross-stage emotional association chain. It adjusts the fusion weight of video and speech based on the privacy interference level to generate stage-by-stage evaluation parameters and time series labels. Mental map evolution report module: Constructs a mental state evolution map based on stage evaluation parameters and time series labels and generates a psychological assessment report.

Images