A micro-short play user portrait analysis system and method based on multi-modal data

Abstract

The application relates to the technical field of data analysis processing, and discloses a micro-short drama user portrait analysis system and method based on multi-modal data, which comprises the following modules: a data acquisition module, which is used for acquiring multi-modal micro-short drama user data; a feature extraction module, which is used for performing feature extraction according to the multi-modal micro-short drama user data to obtain a plurality of user features; a fusion module, which is used for inputting the plurality of user features into a pre-trained hierarchical attention network model to obtain a joint feature vector; a dynamic portrait generation module, which is used for inputting the joint feature vector into a preset online deep forest model to generate a dynamic portrait; and a portrait analysis module, which is used for calculating Shapley values of each dimension of the dynamic portrait and generating a portrait analysis report according to the Shapley values. Compared with the single-modal data portrait commonly used in the prior art, the application has more dimensions, and the dynamic portrait can be modified in time according to user preferences by using a deep forest model.

Description

Technical Field

[0001] This invention relates to the field of data analysis and processing technology, and in particular to a micro-drama user profiling analysis system and method based on multimodal data. Background Technology

[0002] Traditional user profiling systems primarily rely on single-modal data (such as click-through rates and viewing durations), making it difficult to comprehensively depict the complex behavioral characteristics of users in the context of micro-dramas. Micro-dramas are characterized by fragmented content, high-frequency interaction, and intensive emotional expression; existing technologies face the following challenges:

[0003] Single-modal data cannot capture multi-dimensional feedback from users, such as facial expressions, voice, and bullet screen interactions;

[0004] Static tagging systems cannot dynamically reflect shifts in user interests, resulting in insufficient recommendation accuracy.

[0005] Multimodal data is highly heterogeneous, and there is a lack of efficient methods for cross-modal feature alignment and fusion. Summary of the Invention

[0006] In view of this, embodiments of the present invention provide a micro-drama user profiling analysis system and method based on multimodal data to solve at least one of the above-mentioned technical problems.

[0007] To achieve the above objectives, firstly, a micro-drama user profiling analysis system based on multimodal data is provided, comprising:

[0008] The data acquisition module is used to acquire user data for multimodal micro-dramas;

[0009] The feature extraction module is used to extract features based on the multimodal micro-drama user data to obtain several user features;

[0010] The fusion module is used to input several user features into a pre-trained hierarchical attention network model to obtain a joint feature vector;

[0011] The dynamic portrait generation module is used to input the joint feature vector into a preset online deep forest model to generate a dynamic portrait.

[0012] The portrait analysis module is used to calculate the Shapley value of each dimension of the dynamic portrait and generate a portrait analysis report based on the Shapley value.

[0013] Secondly, this invention provides a method for analyzing user profiles of short dramas based on multimodal data, including:

[0014] Acquire user data for multimodal micro-dramas;

[0015] Based on the multimodal short drama user data, feature extraction is performed to obtain several user features;

[0016] Several user features are input into a pre-trained hierarchical attention network model to obtain a joint feature vector;

[0017] The joint feature vector is input into a preset online deep forest model to generate a dynamic profile;

[0018] Calculate the Shapley value for each dimension of the dynamic profile, and generate a profile analysis report based on the Shapley value.

[0019] Thirdly, an electronic device is provided, comprising:

[0020] One or more processors;

[0021] Storage device for storing one or more programs.

[0022] When the one or more programs are executed by the one or more processors, the one or more processors implement a micro-drama user profiling analysis method based on multimodal data as described in the second aspect.

[0023] Fourthly, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements a method for analyzing micro-drama user profiles based on multimodal data as described in the second aspect.

[0024] The above technical solution has the following beneficial technical effects:

[0025] This invention generates user profiles by acquiring multimodal micro-drama user data. Compared with the single-modal data profiles commonly used in existing technologies, it has higher accuracy and richer dimensions. This invention uses a hierarchical attention network model to fuse several user features to obtain a joint feature vector, which greatly improves the fusion accuracy and lays a solid foundation for the subsequent generation of accurate profiles. This invention uses a deep forest model to generate dynamic profiles, which are more flexible than existing user profiles and can be modified in a timely manner according to user preferences. Attached Figure Description

[0026] The accompanying drawings are provided to better understand the invention and are not intended to unduly limit the scope of the invention. Wherein:

[0027] Figure 1 This is a structural block diagram of a micro-drama user profiling analysis system based on multimodal data in an embodiment of the present invention;

[0028] Figure 2 This is a structural block diagram of the data acquisition module in an embodiment of the present invention;

[0029] Figure 3This is a structural block diagram of the feature extraction module in an embodiment of the present invention;

[0030] Figure 4 This is a structural block diagram of the fusion module in an embodiment of the present invention;

[0031] Figure 5 This is a structural block diagram of the dynamic portrait generation module in an embodiment of the present invention;

[0032] Figure 6 This is a structural block diagram of the portrait analysis module in an embodiment of the present invention;

[0033] Figure 7 This is a flowchart of a method for analyzing user profiles of micro-dramas based on multimodal data, as described in an embodiment of the present invention.

[0034] Figure 8 This is a schematic diagram of the structure of a computer system according to an embodiment of the present invention. Detailed Implementation

[0035] The following description, in conjunction with the accompanying drawings, illustrates exemplary embodiments of the present invention, including various details to aid understanding. These details should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the invention. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0036] Example 1

[0037] like Figure 1 As shown, this embodiment provides a micro-drama user profiling analysis system based on multimodal data, including:

[0038] The data acquisition module is used to acquire user data for multimodal micro-dramas;

[0039] The feature extraction module is used to extract features based on the multimodal micro-drama user data to obtain several user features;

[0040] The fusion module is used to input several user features into a pre-trained hierarchical attention network model to obtain a joint feature vector;

[0041] The dynamic portrait generation module is used to input the joint feature vector into a preset online deep forest model to generate a dynamic portrait.

[0042] The portrait analysis module is used to calculate the Shapley Additive Ex Planations (SHAP) for each dimension of the dynamic portrait and generate a portrait analysis report based on the Shapley values.

[0043] Specifically, such as Figure 2 As shown, the data acquisition module includes: a data acquisition unit, a time alignment unit, and a first preprocessing unit.

[0044] The data acquisition unit is used to acquire initial multimodal data from different data sources;

[0045] A time alignment unit is used to synchronize the timestamps of the initial multimodal data from different sources according to the network time protocol, so as to obtain time-synchronized initial multimodal data.

[0046] The first preprocessing unit is used to process the initial multimodal data of the time synchronization using cubic spline interpolation and noise suppression algorithm to obtain multimodal data.

[0047] Specifically, in the data acquisition unit, the initial multimodal data includes initial behavioral data, initial user facial data, initial user voice data, and initial text data. The initial behavioral data includes click frequency (single / double click mode), swipe trajectory (direction / speed), pause / fast forward operation, and device gyroscope attitude data. The acquisition frequency of the initial behavioral data can be 10Hz. The initial user facial data is captured by the front-facing camera of the mobile phone at a frame rate of 30fps, and a 3D facial mesh with 62 key points is constructed in real time using the MediaPipe framework. Micro-expression parameters such as eyelid opening and closing, displacement of the glabella muscle, and upward angle of the corners of the mouth are extracted. The initial user voice data is acquired through a microphone. The initial user voice data is the user's real-time voice comment. After eliminating environmental noise using a noise suppression algorithm, it is saved as an audio segment in Pulse Code Modulation (PCM) format with a sampling rate of 16kHz. The initial text data is acquired in real time through the WebSocket protocol. The initial text data includes the content of the bullet comments sent by the user and the sending time.

[0048] Specifically, the 62 key points are not fixed; the number of key points can be adjusted according to the specific application scenario and algorithm model. MediaPipe's face mesh model offers different numbers of key points with varying precision, ranging from dozens to hundreds. Choosing the number of key points requires balancing computational complexity, detection accuracy, and real-time performance. More key points can provide more refined facial features, but they also increase the computational burden.

[0049] Specifically, the MediaPipe framework is used to build complex computer vision and machine learning pipelines. This framework provides a rich set of pre-trained models and tools, supporting various computer vision tasks such as face detection, pose estimation, and gesture recognition. Its greatest strength lies in its ability to rapidly deploy machine learning models and achieve real-time performance on mobile devices, web pages, and desktop platforms, enabling efficient multimedia processing through graph computing models. The MediaPipe framework can be replaced by the deep learning face detection module of the open-source computer vision library (OpenCV), the facial landmark detection library of Dlib, the face recognition software development kit of Face++, Apple's Vision framework, and machine learning toolkits. These alternatives each have their own characteristics in terms of real-time performance, keypoint accuracy, and cross-platform support. OpenCV provides high-performance deep learning models, Dlib is known for its accurate landmark localization, Face++ has advantages in commercial scenarios, Apple's Vision framework has high integration within the iOS ecosystem, and machine learning toolkits provide lightweight cross-platform solutions similar to MediaPipe. For the multimodal data acquisition scenario of this invention, these alternative solutions all support real-time face mesh construction, can extract key points of eyes, mouth and facial micro-expressions, and have low computational overhead and good mobile device adaptability, which can meet the technical requirements of real-time interactive data acquisition.

[0050] Specifically, pulse code modulation (PCM) converts a continuous analog signal into a discrete digital signal by periodically sampling, quantizing, and encoding the analog signal. There are several alternatives to PCM, such as Advanced Audio Coding (AAC), MP3, or Opus, which offer different balances between compression ratio and audio quality. Compared to PCM, these alternative formats offer higher compression efficiency and smaller storage space requirements.

[0051] Specifically, in the time alignment unit, the Network Time Protocol (NTP) is a network protocol used to synchronize the time of computer systems. During data acquisition, millisecond-level timestamp synchronization is performed using the Network Time Protocol, and a sliding window algorithm is used to slice data into 5-second time units.

[0052] Specifically, in the first preprocessing unit, when the initial multimodal data of the time synchronization is missing, cubic spline interpolation is used for time-series completion. Missing data points are estimated by constructing a piecewise cubic polynomial, thereby ensuring that the multimodal data is aligned in the spatiotemporal dimension. The noise suppression algorithm is a method used in signal processing to reduce background noise, including frequency domain filtering or spectral subtraction.

[0053] Specifically, such as Figure 3As shown, the feature extraction module specifically includes:

[0054] The first extraction unit is used to extract features from the behavioral data using a temporal convolutional network to obtain several first user features.

[0055] The second extraction unit is used to extract features from the user's facial data using a pre-trained target detection model based on a visual database, and obtain several second user features.

[0056] The third extraction unit is used to input the user's voice data into a pre-trained voice representation model for feature extraction, thereby obtaining several third user features.

[0057] The fourth extraction unit is used to input the text data into a pre-trained language model for feature extraction to obtain the fourth user feature.

[0058] Specifically, in the first extraction unit, the behavioral data is input into a preset temporal convolutional network for feature extraction, thereby obtaining several first user features. These first user features include viewing focus features (4-dimensional), content preference intensity features (4-dimensional), and interaction behavior features (4-dimensional). The first user features are obtained by weighted calculation of the viewing focus features, content preference intensity features, and interaction behavior features. The viewing focus features include operation interval variance, single viewing duration, frequency of interrupted viewing, and proportion of repeated playback segments. These indicators collectively reflect the user's level of focus and interest in the content. The content preference intensity features focus on the number of times the user repeatedly plays specific types of content, viewing completeness, content dwell time, and fast-forward / rewind frequency, thereby revealing the user's content preference patterns. The interaction behavior features include click / swipe frequency, device gyroscope attitude change rate, night / day viewing ratio, and social sharing willingness index, comprehensively expressing the user's viewing behavior characteristics. The temporal convolutional network is a deep learning model specifically designed for processing sequential data, directly processing time-series information through convolutional operations. Compared to traditional recurrent neural networks, the temporal convolutional network (TCNN) offers advantages such as parallel computation, long-term dependency modeling, and better gradient flow. By employing one-dimensional convolutional kernels to extract features from time-series data, the TCNN captures dependencies across different time scales through dilated convolution mechanisms while maintaining low computational complexity. In fields such as audio processing, video analysis, and user behavior prediction, the TCNN effectively captures temporal patterns and long-term dependencies in sequential data.

[0059] Specifically, the viewing focus characteristics are determined based on the operation interval variance, the duration of a single viewing session, the frequency of interrupted viewing, and the proportion of repeated playback segments. The operation interval variance is calculated by recording the time interval between every two consecutive operations (such as pause, fast forward, and rewind) performed by the user during viewing, and then calculating the standard deviation of these intervals. The formula for calculating the operation interval variance is as follows:

[0060]

[0061] In the formula, σ represents the variance of the operating interval, and x i σ represents the interval between each operation, μ represents the average interval, and n represents the number of operations. The smaller σ is, the more regular the user's operations and the higher their focus.

[0062] Specifically, the continuous viewing duration is calculated by statistically analyzing the user's uninterrupted viewing time and determining the duration of the continuous viewing interval. The continuous viewing duration is defined as max(t_end - t_start), where t_end and t_start are the end and start timestamps of the continuous viewing interval, respectively. A longer duration indicates higher focus.

[0063] Specifically, the frequency of interrupted viewing refers to the number of times a user pauses, switches, or exits during viewing. The frequency of interrupted viewing is calculated using `count(interruption_events)`, which counts the total number of interruption events. Fewer interruptions indicate higher focus.

[0064] Specifically, the proportion of repeatedly played segments is determined by calculating the percentage of times a user repeatedly plays a specific video segment. The formula for calculating the repeatedly played segments is [(duration of repeatedly played segment / total viewing time) * 100%]. By analyzing whether users repeatedly watch certain segments, it reflects their level of interest in the content.

[0065] Specifically, the content preference intensity feature is calculated by weighting the number of replays, viewing completion rate, content dwell time, and fast-forward / rewind frequency. The number of replays is calculated by counting the total number of times a user replays a specific type or content. The calculation method is sum(replay_count), which is to sum the replay counts of a certain type of content. The more replays, the stronger the user's preference for that content.

[0066] Specifically, the viewing completion rate is used to measure whether a user has watched the content in its entirety. The formula for calculating the viewing completion rate is [(actual viewing time / total video time) * 100%]. The higher the viewing completion rate, the stronger the user's interest in the content.

[0067] Specifically, the content dwell time is often used to record the total time a user spends on a particular content type or video. The calculation method for content dwell time is sum(stay_duration), which is the sum of the time a user spends on a certain type of content. A longer dwell time indicates greater user interest.

[0068] Specifically, the fast-forward / rewind frequency is used to count the number of times a user fast-forwards or rewinds during viewing. The formula for calculating the fast-forward / rewind frequency is [(number of fast-forwards + number of rewinds) / total viewing time]. The lower the frequency, the more focused the user is on the content.

[0069] Specifically, the interactive behavior characteristics are calculated based on a weighted average of click / swipe frequency, device gyroscope attitude change rate, nighttime / daytime viewing ratio, and social sharing willingness index. The click / swipe frequency records the frequency with which users click or swipe during viewing. The formula for calculating the click / swipe frequency is [(number of clicks + number of swipes) / total viewing time], and this frequency reflects the user's level of interactive activity.

[0070] Specifically, the device gyroscope attitude change rate is calculated by analyzing the device gyroscope data to determine the frequency and amplitude of attitude changes during viewing. The formula for calculating the device gyroscope attitude change rate is as follows:

[0071]

[0072] In the formula, Δθ i is the value of the i-th attitude angle change, n is the total number of attitude changes, and T is the total viewing time (in seconds or minutes). The attitude change rate of the device gyroscope reflects the user's focus and comfort during viewing.

[0073] Specifically, the nighttime / daytime viewing ratio refers to the percentage of time users spend watching at different times (nighttime / daytime). The calculation formula is (nighttime viewing time / total viewing time) * 100%, indicating users' viewing habits and time preferences.

[0074] Specifically, the social sharing willingness index calculates users' approval of content and their willingness to spread it by analyzing their sharing behavior. The formula for calculating the social sharing willingness index is: [(number of shares / number of views) * 100%], which reflects users' assessment of the social value of content.

[0075] Specifically, in the second extraction unit, the visual database includes a simplified computer vision library or an open-source library focused on human pose recognition. A continuous inter-frame optical flow matrix is ​​calculated using the visual database, and then this matrix is ​​input into the pre-trained target detection model (e.g., a residual neural network-18 model, a lightweight convolutional neural network model, a transformer vision model, or other pre-trained target detection models) to extract second user features such as the pupil diameter change rate and blink frequency of the eye region (ROI_eye), and the muscle tremor amplitude of the mouth region (ROI_mouth). The second user features include eye region features (4D), mouth region features (3D), and comprehensive facial features (2D).

[0076] Specifically, the eye region features are calculated by weighting the pupil diameter change rate, blink frequency, eye muscle tension, and gaze duration. The mouth region features are calculated by weighting the corner of the mouth angle, muscle tremor amplitude, and mouth micro-movement frequency. The comprehensive facial features are calculated by weighting the facial muscle tension index and emotional expression intensity. The comprehensive facial features provide an overall assessment of the user's emotional state and provide important non-verbal interaction information for multimodal user profiling.

[0077] Specifically, the pupil diameter change rate is obtained by extracting changes in the pupil region across consecutive frames using edge detection and contour recognition algorithms from the visual database. By tracking minute changes in pupil size, it reflects fluctuations in the user's emotions and attention. The formula for calculating the pupil diameter change rate is as follows:

[0078]

[0079] Specifically, the blink frequency is determined by identifying blinking events using a target detection model within consecutive video frames. The formula for calculating the blink frequency is: Blink frequency = Total blinks / Observation time. The observation time is preferably measured in seconds, and a higher blink frequency indicates tension, fatigue, or distraction.

[0080] Specifically, the eye muscle tension is calculated by analyzing minute deformations of the periorbital muscles and using optical flow to calculate the standard deviation of pixel displacement in the eye region. The formula for calculating the eye muscle tension is as follows:

[0081]

[0082] In the formula, MT represents the tension of the eye muscles, and Δp i is the displacement difference of the i-th pixel, and N is the total number of pixels. A higher eye muscle tension value indicates higher muscle tension, reflecting stress or emotional fluctuations.

[0083] Specifically, the change in fixation duration is obtained by tracking the stable fixation time of the eye in consecutive frames and calculating the variance of the fixation duration. The formula for calculating the change in fixation duration is as follows:

[0084]

[0085] In the formula, T i T is the gaze duration of the i-th frame. μ is the average fixation duration, and n is the total number of frames. Changes in fixation duration can reflect changes in cognitive load and attention span.

[0086] Specifically, the upward angle of the corners of the mouth can be calculated by extracting the coordinates of the corners of the mouth using a key point detection algorithm and calculating the angle change between the corners of the mouth and the horizontal line. The formula for calculating the upward angle of the corners of the mouth is as follows:

[0087]

[0088] In the formula, θ up The angle of the corners of the mouth is calculated using the coordinates of two key points at the corners of the mouth (x1, y1) and (x2, y2), where arctan is the arctangent function. This angle can reflect emotional states, such as smiling or tension.

[0089] Specifically, the muscle tremor amplitude is analyzed using optical flow to measure the displacement intensity of pixels in the mouth region across consecutive frames. The formula for calculating the muscle tremor amplitude is: Muscle tremor amplitude = max(pixel displacement vector length). A larger muscle tremor amplitude indicates more intense muscle activity, suggesting emotional fluctuations.

[0090] Specifically, the frequency of mouth micro-movements is obtained by identifying minute movements in the mouth region between consecutive frames and counting the number of movements per unit time. The calculation method for the frequency of mouth micro-movements is: Frequency of mouth micro-movements = Total number of micro-movements / Observation time length. The unit of the observation time length is preferably seconds, and the frequency of mouth micro-movements can reflect the degree of tension and emotional expression.

[0091] Specifically, the facial muscle tension index calculates the comprehensive standard deviation of total facial muscle displacement by integrating muscle activity data from the eye and mouth areas. The formula for calculating the facial muscle tension index is as follows:

[0092]

[0093] In the formula, MTI is the facial muscle tension index, Δq i The displacement difference of the i-th facial pixel is M, where M is the total number of facial pixels. The larger the facial muscle tension index value, the higher the overall muscle tension.

[0094] Specifically, the intensity of the emotional expression is determined by combining...

[0095] A comprehensive emotion intensity scoring model is constructed using eye and mouth region features. This model can be trained using a machine learning model (e.g., Support Vector Machine), outputting an emotion intensity value between 0 and 1. A higher emotion intensity value indicates a more pronounced emotional expression. First, the eye and mouth region features are preprocessed to unify their dimensions. Specifically, a min-max normalization method can be used to map each feature to the [0,1] interval, with the formula X_normalized = (X - X_min) / (X_max - X_min). This standardization eliminates scale differences between features, allowing the subsequent machine learning model to more fairly evaluate the importance of each feature, improving the model's learning efficiency and generalization ability. Second, Support Vector Machine is chosen as the algorithm for the emotion intensity scoring model because it can effectively handle multi-dimensional nonlinear features. The emotion intensity scoring model learns from a labeled training dataset to construct the optimal classification hyperplane. Then, the emotion intensity output by the model is within the [0,1] interval, where 0 represents extremely weak emotion expression, close to a state of no perception; and 1 represents extremely strong emotion expression, almost out of control. The median value reflects the gradual change in emotional intensity. This quantitative method provides an objective measure of emotional intensity.

[0096] Specifically, in the third extraction unit, the user's voice data is first segmented into effective voice segments. These effective voice segments are then converted into 128-dimensional Mel spectrograms and input into a pre-trained voice representation model (e.g., a Visual Geometry Group-based Feature Extraction for Audio (VGGish) network) to obtain 6-dimensional voice emotion features. These voice emotion features are the third user features, which extract the user's multi-dimensional emotional state through in-depth analysis of the audio signal. Excitement level and disappointment intensity reflect the user's immediate emotional response to the content, while the surprise index captures emotional fluctuations brought about by unexpected or groundbreaking content. Anger level and fear level reveal the user's negative emotional feedback to the content, while the neutral emotion ratio provides an indicator of the balance of emotional expression. These six features together constitute a comprehensive voice emotion profile, capable of deeply analyzing subtle changes in the user's emotions.

[0097] Specifically, the VGGish network can be replaced by the wav2vec 2.0 model or a hidden-unit bidirectional encoder model (HuBERT) based on transformers, where the wav2vec 2.0 model exhibits superior performance in speech feature extraction. Through self-supervised pre-training on the original audio signal, it can capture richer and deeper speech features while maintaining high accuracy even with limited labeled data. Compared to VGGish, wav2vec 2.0 has advantages in feature representation and generalization ability. The HuBERT model, through clustering and a mask prediction strategy similar to BERT, can learn powerful speech features from unlabeled audio data. HuBERT performs excellently in speech emotion recognition tasks and has a more flexible model structure, making it well-suited for different speech emotion feature extraction needs.

[0098] Specifically, the voice emotion features are obtained by weighting features of excitement level, disappointment intensity, surprise index, anger level, fear level, and neutral emotion ratio. The formula for calculating the excitement level feature is as follows:

[0099] Excitement level characteristic = α * speech energy + β * fundamental frequency change rate + γ * speech rate;

[0100] In the formula, α, β, and γ are weighting coefficients, dynamically adjusted using machine learning methods such as gradient descent. Speech energy reflects sound intensity, fundamental frequency change rate reflects pitch fluctuations, and speech rate characterizes the rhythm of speech. The higher the level of excitement, the larger the values ​​of these three indicators. The three indicators of excitement level are obtained through complex acoustic signal processing techniques. Speech energy is calculated by performing a Short-Time Fourier Transform (STFT) on the audio signal, calculating the sum of squares of the signal amplitude within each time window, and taking the logarithm, reflecting the physical intensity and amplitude changes of the sound. The fundamental frequency change rate is extracted using a pitch detection algorithm (such as autocorrelation or frequency domain analysis) to extract the fundamental frequency (F0) of each speech frame. By calculating the rate and amplitude of change of the fundamental frequency between adjacent frames, the dynamic fluctuation of pitch is quantified. Speech rate measurement involves speech segmentation and phoneme recognition. First, an endpoint detection algorithm (such as energy thresholding or zero crossover rate) is used to segment continuous speech into independent speech segments. Then, the number of speech segments or syllables per unit time is counted, and the results are standardized by combining the speech duration. These three metrics are dynamically adjusted using machine learning methods (such as gradient descent) to construct a multi-dimensional, dynamically adaptive excitement level assessment model that can accurately capture subtle emotional changes in speech.

[0101] Specifically, the formula for calculating the disappointment intensity characteristic is as follows:

[0102]

[0103] In the formula, DI represents the disappointment intensity feature, σ represents the pitch variance, D represents the amplitude decay coefficient, and e is the base of the natural logarithm. δ and ε are adjustment parameters, and the Sigmoid function is used to map the features to the [0,1] interval. The smaller the pitch variance, the faster the amplitude decays, and the higher the disappointment intensity. Obtaining the pitch variance and amplitude decay coefficient involves complex acoustic signal processing techniques. The pitch variance is obtained through statistical analysis of the fundamental frequency (F0). First, a pitch extraction algorithm (such as autocorrelation or Halbert-Huang transform) is used to accurately locate the fundamental frequency in each speech frame, and then the statistical variance of these fundamental frequencies is calculated. Specifically, the original speech signal needs to be preprocessed, including denoising, normalization, and framing, with each frame being 20-40 milliseconds long and overlapping by 50% to ensure continuity. A pitch extraction algorithm is applied to each frame to obtain a pitch frequency sequence. The standard deviation is calculated; a smaller variance indicates a more uniform pitch variation, reflecting monotony and a lack of emotional fluctuation in the speech. The speech amplitude attenuation coefficient is obtained by analyzing the exponential decay characteristics of the speech signal envelope. First, a Hilbert envelope transform is performed on the speech signal to extract the instantaneous amplitude. Then, an exponential decay model is used to fit the amplitude's trend over time. Specifically, the logarithm of the amplitude is taken followed by linear regression; the slope is the amplitude attenuation coefficient, reflecting the degree to which speech intensity decreases rapidly over time. These two indicators, combined with a Sigmoid nonlinear mapping, construct a disappointment intensity assessment model capable of sensitively capturing subtle emotional changes in speech.

[0104] Specifically, the surprise index is calculated as follows: Surprise Index = max(|Fundamental Frequency Sudden Change Rate|, |Speech Energy Sudden Change Rate|) * Sudden Change Duration. This quantifies the intensity of unexpected or unforeseen emotions by capturing sudden changes in the speech signal. A higher sudden change rate and a longer duration result in a higher surprise index. Obtaining the fundamental frequency sudden change rate and speech energy sudden change rate requires advanced signal processing and feature extraction techniques. The calculation of the fundamental frequency sudden change rate first involves accurate fundamental frequency (F0) extraction, employing a multi-algorithm fusion method, such as autocorrelation function, cepstral analysis, and harmonic matching, to ensure accurate fundamental frequency localization in complex speech contexts. Fundamental frequency detection is performed on consecutive speech frames (each frame 20-30 milliseconds), and the instantaneous rate of change of the fundamental frequency between adjacent frames is calculated. Specifically, this is obtained by dividing the absolute value of the difference between the fundamental frequencies of two consecutive frames by the inter-frame time interval. The speech energy sudden change rate is based on Short-Time Fourier Transform (STFT) and energy envelope analysis. First, the speech signal is decomposed into a time-frequency representation, and the signal energy of each time window is calculated. Quantification is achieved by calculating the relative rate of change of energy between adjacent time windows. The mutation detection algorithm combines wavelet transform and adaptive thresholding to identify significant energy jumps and eliminate background noise interference. These two mutation rate calculations not only focus on instantaneous changes but also consider the duration and magnitude of those changes. Through multi-scale analysis, it captures subtle yet significant surprise features in speech signals, thereby constructing a computational model that sensitively reflects the intensity of surprise.

[0105] Specifically, the formula for calculating the anger level characteristic is as follows:

[0106] Anger level = ζ * speech roughness + η * speech rate fluctuation rate + θ * fundamental frequency instability;

[0107] In the formula, ζ, η, and θ are adjustment weights. Speech roughness reflects the degree of hoarseness, speech rate variability reflects emotional instability, and fundamental frequency instability characterizes drastic pitch changes. Obtaining speech roughness, speech rate variability, and fundamental frequency instability requires the comprehensive application of multiple acoustic signal processing techniques. Speech roughness is quantified through glottal closure irregularity and harmonic-to-noise ratio (HNR). First, linear predictive cepstral coefficients (LPCC) and Mel-frequency cepstral coefficients (MFCC) are used to extract vocal tract features. Then, combined with a glottal impulse analysis algorithm, the proportion of non-periodic components in the speech signal is accurately evaluated. Specifically, by calculating the modulation spectrum and harmonic structure of each speech frame, the degree of harmonic distortion of the sound is quantified; higher roughness means a hoarser and harsher sound. Speech rate variability is determined through statistical analysis of the time intervals and syllable durations of speech segments. The Dynamic Time Warping (DTW) algorithm is used to accurately extract the temporal features of speech segments, calculating the coefficient of variation of adjacent syllable durations to reflect the irregularity of speech rate. Fundamental frequency instability is obtained through multidimensional pitch trajectory analysis. First, an improved YIN algorithm (a fundamental frequency detection algorithm that accurately extracts fundamental frequency or pitch from audio signals) and harmonic matching techniques are used to accurately extract the fundamental frequency sequence. Then, the short-time variance and peak difference of the fundamental frequency are calculated to quantify the degree of pitch fluctuation. These three features are combined linearly with weights to construct a computational model that sensitively captures subtle changes in anger emotion in speech, considering not only instantaneous features but also the overall dynamic characteristics of the speech signal.

[0108] Specifically, the formula for calculating the level of fear is as follows:

[0109]

[0110] In the formula, F represents the tremor frequency. A represents the volume weakness coefficient. e is the base of the natural logarithm. κ and λ are adjustment parameters, normalized using the Sigmoid function. The higher the tremor frequency, the weaker the volume, and the greater the degree of fear. Obtaining the tremor frequency and volume weakness coefficient requires complex acoustic signal processing and psycholinguistic analysis techniques. The calculation of the tremor frequency is first based on time-frequency analysis methods such as Short-Time Fourier Transform (STFT) and Hilbert-Huang Transform (HHT) to decompose the speech signal into multi-scale oscillatory components. By extracting the envelope and instantaneous frequency of the speech signal, the involuntary tremor features in the speech are identified. Specific methods include: calculating the rate of change of the envelope of the speech signal, extracting the tremor frequency band (in the range of 4-8Hz), and analyzing the frequency distribution and duration of the tremor. Simultaneously, by combining wavelet transform and Empirical Mode Decomposition (EMD) techniques, subtle flutter components in the speech signal are accurately extracted, eliminating background noise interference. The speech volume weakness coefficient is comprehensively evaluated through multi-dimensional acoustic features. First, logarithmic energy normalization and perceptual sound level calculation methods are used to extract the short-time energy and sound pressure level of the speech signal. Then, combined with a psychoacoustic model, the relative weakness of the speech volume is quantified. Specific calculations include: calculating the normalized energy envelope of the speech segment and analyzing its deviation from the standard speech energy distribution; using a perceptual weighting function to consider the sensitivity of the human ear to different frequencies of sound; and introducing a dynamic range compression algorithm to highlight the characteristics of weak sounds. These two features, through meticulous signal processing and psychoacoustic modeling, can sensitively capture the subtle acoustic features reflecting fear in speech, providing a meticulous computational foundation for the quantification of emotion intensity.

[0111] Specifically, the formula for calculating the neutral emotion ratio characteristic is as follows:

[0112] Neutral emotion ratio characteristic = 1 - (excitement level characteristic + disappointment intensity characteristic + surprise index characteristic + anger level characteristic + fear level characteristic);

[0113] The neutral emotion ratio characteristic is calculated by subtraction, reflecting the proportion of non-extreme emotional states. The closer the ratio is to 1, the more stable and neutral the emotional expression.

[0114] Specifically, in the fourth extraction unit, the text data line, after being segmented by Jieba, is input into a Bidirectional Encoder Representations from Transformers-Whole Word Masking (BERT-wwm) model. A 768-dimensional semantic vector is obtained through Classification Tokens (CLS), and then dimensionality is reduced to 8 dimensions using Principal Component Analysis (PCA). Each feature extractor is connected to an intra-modal attention layer, and learnable parameters are used to calculate feature importance weights. For example, in facial features, the indicative role of blink frequency in interest attenuation is automatically enhanced. The fourth user feature is the aforementioned bullet screen sentiment feature.

[0115] The aforementioned bullet screen sentiment features extract collective emotional characteristics through in-depth analysis of user group interactions. Positive and negative sentiment intensity features directly reflect users' overall evaluation of the content, while the density of popular online terms reflects the content's popularity and the user group's linguistic characteristics. Sentiment fluctuation frequency and interaction activity features reveal the user group's engagement, while sentiment consistency and group emotional synchronization reflect the degree of collective emotional resonance. Sentiment depth features are also included. These eight dimensions not only capture individual emotions but also present the collective emotional characteristics of the user group. In other embodiments, BERT-wwm can be replaced by Robust Optimized BERT Pre-training Method (RoBERTa), Knowledge Augmentation Representation Model ERNIE, or other pre-trained language models to extract and optimize the semantic features of the text data.

[0116] Specifically, the Jieba word segmentation can be replaced by the HanLP Chinese Natural Language Processing Toolkit or the THULAC Chinese word segmentation tool. HanLP offers more refined segmentation modes, supports custom dictionaries, and has stronger ambiguity resolution capabilities. HanLP performs particularly well in online text and colloquial corpora. HanLP features a high-precision segmentation algorithm and low computational overhead. It supports multiple segmentation modes and has a built-in dictionary trained on a large-scale corpus, demonstrating excellent performance when processing complex Chinese text. Compared to Jieba, THULAC has advantages in segmenting professional and academic texts.

[0117] Specifically, the formula for calculating the positive emotional intensity feature is as follows:

[0118]

[0119] In the formula, PS represents the intensity of positive emotion, and W...i + F represents the weight of the i-th positive sentiment word. These weights are determined through a pre-trained sentiment lexicon and indicate the degree to which the sentiment word contributes to the overall emotion. i Let be the word frequency of the i-th sentiment word, i.e., the number of times the sentiment word appears in the text. N is the total number of words, representing the total number of words in the text, used to normalize the sentiment intensity.

[0120] Specifically, the formula for calculating the intensity of negative emotions is as follows:

[0121]

[0122] In the formula, NS represents the intensity of negative emotion, and W... i - The weight of the i-th negative sentiment word is also determined through the sentiment dictionary, representing the impact of the negative sentiment word on the emotion.

[0123] Specifically, the formula for calculating the density feature of the aforementioned trending online words is as follows:

[0124] The density characteristic of trending online words = (number of times trending online words appear / total number of words) * log(the spread range of trending online words);

[0125] The frequency of trending online terms is calculated by dividing the number of occurrences of these terms by the total number of terms. This frequency is then combined with the reach of these terms to reflect the popularity of the content and the linguistic characteristics of the user group. A logarithmic function is used to mitigate the influence of extreme values, making the features more representative. The reach of trending online terms is obtained through comprehensive analysis of multi-dimensional indicators from online social media, primarily including the number of reposts, comments, interactions, and cross-platform mentions. The specific methods utilize web crawlers, big data analytics platforms, and social network application programming.

[0126] This interface statistically analyzes the spread of trending internet terms across different platforms, regions, and user groups. The calculation of the spread of trending internet terms integrates factors such as the number of reposts, the number of platforms, the diversity of user groups, and the time span, and can be expressed as a continuous floating-point number or a standardized discrete value to accurately reflect the intensity and influence of the spread of trending internet terms. This quantitative method not only objectively measures the popularity of trending terms but also reveals the linguistic characteristics of user groups and the dynamics of information dissemination.

[0127] Specifically, the formula for calculating the frequency characteristics of the emotional fluctuations is as follows:

[0128] Emotional fluctuation frequency characteristic = number of consecutive emotional polarity changes / total number of bullet comments;

[0129] The emotional fluctuation frequency feature quantifies the instability of user group emotions by tracking the rapid switching of emotional polarity (positive / negative / neutral) between consecutive bullet comments. The more frequent the fluctuation, the higher the feature value.

[0130] Specifically, the calculation formula for the interactive enthusiasm characteristic is as follows:

[0131] Interaction activity characteristic = (number of replies * weight of interactive words) / total number of bullet comments;

[0132] The interactive engagement characteristics are determined by considering both the number of bullet screen replies and the frequency of use of interactive words (such as "666" and "awesome"). By quantifying the degree of user active participation and positive interaction, the group's enthusiasm for participation is reflected.

[0133] Specifically, the formula for calculating the emotional consistency feature is as follows:

[0134] Emotional consistency characteristic = 1 - (Standard deviation of emotional polarity / Mean of emotional polarity);

[0135] The emotional consistency feature is calculated by assessing the dispersion of the emotional polarity of the user group within the bullet screen. A smaller standard deviation indicates greater emotional consistency among the user group. The normalized dispersion coefficient is subtracted from 1, resulting in a larger feature value representing higher consistency.

[0136] Specifically, the calculation formula for the group emotional synchronization feature is as follows:

[0137] Group emotional synchronicity characteristics = Pearson correlation coefficient (emotional polarity sequence 1, emotional polarity sequence 2);

[0138] The group emotion synchronization feature quantifies the degree of synchronization of user group emotions by calculating the Pearson correlation coefficient of the bullet screen emotion polarity sequence within different time windows. The closer the correlation coefficient is to 1, the more synchronized the group emotions are. The Pearson correlation coefficient is used to measure the degree of linear correlation between two variables, and its value ranges from -1 to 1. When the coefficient is close to 1, it indicates a positive correlation, that is, the two variables change in the same direction; close to -1 indicates a negative correlation, that is, the variables change in opposite directions; close to 0 means that there is almost no linear correlation between the variables. In the group emotion synchronization analysis, the emotion polarity sequence is obtained through natural language processing (NLP) technology. Specifically, the bullet screen text is subjected to sentiment analysis, and each bullet screen is divided into positive, negative, or neutral emotions using sentiment classification algorithms (such as Naive Bayes, support vector machines, or deep learning models), and assigned a corresponding value (e.g., positive +1, negative -1, neutral 0), forming an emotion polarity sequence arranged in chronological order. By calculating the Pearson correlation coefficient between these two sequences, the consistency of emotional changes among different user groups within a specific time window can be quantified, thereby revealing the dynamic characteristics and synchronization mechanisms of collective emotions.

[0139] Specifically, the formula for calculating the emotional depth feature is as follows:

[0140] Sentiment depth feature = (Number of sentiment words / Total number of words) * log(Average weight of sentiment words);

[0141] Specifically, the sentiment depth feature reflects the depth and complexity of emotional expression in bullet comments by combining the proportion of sentiment words and the average weight of sentiment words. A logarithmic function is used to balance extreme values ​​and quantify the richness of user group sentiment expression. The full-word masking BERT model is an important pre-trained model in the field of natural language processing, improving the model's language understanding ability through an improved masking strategy. The classification tag is located at the beginning of the input sequence and aggregates the semantic information of the entire sequence through an attention mechanism to generate a global semantic representation. During text feature extraction, the classification tag can capture the overall semantic features of the input text, providing a semantic foundation for subsequent sentiment analysis and user profile construction. During the pre-training and fine-tuning of the full-word masking BERT model, the classification tag is placed at the beginning of the input sequence. Its role is to aggregate the semantic information of the entire input sequence through a self-attention mechanism to generate a fixed-length global semantic representation vector. This tag was initially designed for classification tasks, capturing the overall semantic features of the input text and integrating the contextual information of each word in the sequence through a multi-head attention mechanism, thus providing a semantically rich and compact representation for subsequent text classification, sentiment analysis, and other tasks. In multimodal user profiling systems, classification tags can not only extract global semantic features of text, but also serve as a bridge for cross-modal feature fusion, helping the system understand the deep semantic intent behind user behavior.

[0142] Specifically, the contribution of the first, second, third, and fourth user features, as well as the features subordinate to each user feature, is dynamically adjusted through learning parameters. The weight calculation employs a softmax function, which non-linearly transforms the original features using a learnable weight matrix W and a bias term b to obtain the importance distribution of each feature. This adaptive weight learning mechanism enables the model to automatically adjust the weights of each modality feature based on different scenarios and user characteristics, achieving a more accurate user profile representation.

[0143] Specifically, such as Figure 4 As shown, the fusion module specifically includes:

[0144] The second preprocessing unit is used to preprocess several user features to obtain several preprocessed features;

[0145] The intramodal attention unit is used to divide the preprocessed features into several modalities, calculate the attention score of each preprocessed feature in each modal, and select key sub-features from each modal based on the attention score.

[0146] Cross-modal attention units are used to calculate the pairwise correlation between modalities using cross-modal attention algorithms;

[0147] The fusion unit is used to generate a joint feature vector based on the key sub-features and the correlation degree.

[0148] Specifically, in the second preprocessing unit, the preprocessing includes standardization and normalization. Since user features include data from multiple modalities (e.g., text, speech, facial expressions, etc.), for each modality m, its feature vector... Batch normalization or layer normalization can be used to eliminate scale differences in the feature spaces of different modalities, ensuring the fairness and stability of subsequent attention mechanism calculations. Normalization helps reduce the bias in feature distribution between modalities.

[0149] Specifically, in the intra-modal attention unit, when dividing the preprocessed features into several modalities, the division is based on the data source of the preprocessed features. For example, speech-related preprocessed features are divided into a unified speech modality. For each modality m, a learnable weight matrix W is used. m and V m Calculate attention score α m The formula for calculating the attention score is as follows:

[0150]

[0151] In the formula, a m W is the attention score for modality m, representing the importance weights of each dimension of the modality's features. m V is the weight matrix of mode m, used to calculate the weighted sum of preprocessed features within mode m. m It is another learnable matrix of mode m, used for preprocessing input features h. m Perform a nonlinear transformation. m is the preprocessed feature within modality m, representing the input features of the modality. tanh is a non-linear activation function used to introduce non-linear characteristics. softmax is used to normalize the obtained attention scores, ensuring that the attention scores of all modalities sum to 1. It is W m The transpose of .

[0152] Specifically, based on the attention score, the preprocessed feature with the highest weight is selected as the key sub-feature from each modality.

[0153] Specifically, in the cross-modal attention unit, the semantic correlation degree between different modalities is calculated through learnable query matrix Q (Query), key matrix K (Key), and value_matrix V (Value). Specifically, for modality i and modality j, their correlation degree β ij is calculated through the dot product attention mechanism, and the calculation formula for the correlation degree is as follows:

[0154]

[0155] In the formula, β ij is the correlation degree between modality i and modality j. q i is the query vector (Query) of modality i. k j is the key vector (Key) of modality j. k k represents the key vector (Key Vector) of the k-th modality. d is the dimension of the query vector and the key vector, which is used to scale the dot product to prevent the value from being too large. exp represents the exponential function, which is used to calculate the exponential value after the dot product. ∑k is the weighted sum of all modalities, ensuring the normalization of the correlation degree value. K is the total number of modalities.

[0156] Specifically, in the fusion unit, according to the correlation degree between modalities and several key sub-features, the key sub-features of different modalities are integrated through weighted summation or a more complex fusion strategy. The joint feature vector comprehensively reflects the semantic information of each modality and the interaction relationship between modalities, providing rich multi-modal representations for subsequent sentiment classification tasks.

[0157] Specifically, it further includes a dynamic feature enhancement unit, which is used to detect several of the key sub-features. When a special field appears in the several key sub-features, the weight of the key sub-feature with the special field when generating the joint feature vector is adjusted. The special field, for example, is a bullet screen containing "reverse wonderful". When there is a special field, a dynamic feature enhancement mechanism is activated, such as automatically adjusting the weight coefficient of the key sub-feature related to bullet screen feedback, so as to achieve semantic enhancement of the text-audio modality. This mechanism dynamically adjusts the weight allocation of cross-modal feature fusion based on the real-time feedback of user interaction, improving

[0158] the adaptability of the model to complex interaction scenarios.

[0159] Specifically, it further includes a post-processing unit, which is used to post-process the joint feature vector (128-dimensional). The post-processing includes but is not limited to: dropout regularization, layer normalization, or residual connection. These techniques are beneficial to reducing overfitting, stabilizing model training, and preparing high-quality feature representations for the subsequent sentiment classification layer.

[0160] Specifically, such as Figure 5 As shown, the dynamic image generation module specifically includes:

[0161] A subset building unit is used to select several feature subsets of different granularities from the joint feature vector;

[0162] The attenuation unit is used to adjust the influence weights of several feature subsets according to a preset time attenuation factor.

[0163] The profile generation unit is used to input several feature subsets and influence weights into the pre-trained online deep forest model to obtain an initial user profile;

[0164] The update unit is used to perform incremental learning at preset time intervals to update the user profile and obtain a dynamic profile.

[0165] Specifically, in the subset construction unit, the joint feature vector is processed using a random subspace method, feature importance evaluation method, or principal component analysis method to select feature subsets of different granularities. This method can effectively reduce dimensionality, prevent overfitting, and enhance the generalization ability of the model, enabling each base learner to capture subtle changes in user interests from different perspectives.

[0166] Specifically, in the attenuation unit, the attenuation factor is expressed as: λ = 0.98 t Here, t represents the data time difference in hours. By setting a decay factor, the weights of historical samples are dynamically adjusted, allowing the online deep forest model (ODF-MT) to focus more on recent behavioral patterns. This exponentially decays the influence of historical data; for example, data from 1 hour ago has a weight of approximately 0.98, data from 24 hours ago has a weight of 0.376, and data from 72 hours ago has a weight of only 0.014. Through this fine-grained weight adjustment strategy, the model can quickly respond to real-time changes in user interests, retaining the reference value of historical behavior while improving its sensitivity and adaptability to recent behavioral patterns, thereby achieving dynamic and accurate updates to user profiles.

[0167] Specifically, in the user profile generation unit, the online deep forest model includes three incremental random forest models and two dynamic gradient boosting tree models. Each base learner receives feature subsets of different granularities. The base learner is a single classification or regression model in ensemble learning. By inputting the feature subsets and the influence weights corresponding to each feature subset into different models, an initial user profile is obtained.

[0168] Specifically, in the update unit, the preset time interval is preferably 30 minutes. The leaf node distribution is updated using a staggered binning strategy, and nodes are automatically split when the KL divergence of the new data distribution exceeds the threshold θ = 0.15. Simultaneously, the system maintains the user state matrix U∈R.128 The model integrates short-term and long-term interest features through an LSTM update gate mechanism. Leaf nodes are the lowest-level nodes in the decision tree / forest model; they do not split further and represent the final classification or regression result. The misaligned binning strategy is an advanced data distribution adjustment method that divides the data space into multiple overlapping sub-intervals. When a significant shift occurs in the new data distribution, leaf node splitting is automatically triggered. This strategy allows the model to dynamically adjust its internal structure, capturing subtle changes and potential trends in user interests in real time. The user state matrix U∈R... 128 The system is derived from the joint feature vector and continuously updated and optimized using historical user interaction data. It integrates users' short- and long-term interest features through an LSTM (Long Short-Term Memory) update gate mechanism. The update gate controls the flow of information into the long-term memory unit, balancing the volatility of short-term interests with the stability of long-term interests. The LSTM update gate mechanism achieves dynamic fusion of short- and long-term interest features through a complex gating structure.

[0169] Specifically, such as Figure 6 As shown, the portrait analysis module specifically includes:

[0170] The Shapley value calculation unit is used to calculate the Shapley value of each dimension in the dynamic image;

[0171] The tag generation unit is used to generate several multi-dimensional tags based on each dimension and the corresponding Shapley value in the dynamic portrait.

[0172] The analysis unit is used to generate a profile analysis report based on several of the multi-dimensional tags.

[0173] Specifically, in the Shapley value calculation unit, when extracting features from the dynamic portrait, each dimension of the dynamic portrait corresponds to a portrait feature, resulting in several portrait features. These portrait features are then input into the Shapley library to calculate the Shapley value of each portrait feature.

[0174] Specifically, the working process of the Shapley value calculation unit includes the following steps:

[0175] Step 1: Feature Extraction. First, in the Shapley score calculation unit, each dimension of the dynamic profile corresponds to a profile feature. These features are extracted based on user behavioral or emotional data. For example, instantaneous sentiment score, sentiment fluctuation index, content preference, and consumption decision-making—each dimension represents a user's behavior or emotional state in a specific area. The feature extraction process involves identifying and extracting specific data related to each dimension from the raw user data and converting it into feature vectors that facilitate Shapley score calculation. The feature values ​​extracted for each dimension serve as model input, reflecting the user's specific performance in that dimension.

[0176] Step 2: After feature extraction, the next step is to define the feature set. Assume the dynamic profile has n features, each representing a dimension of the dynamic profile. For example: f1 represents the instantaneous sentiment value, f2 represents the sentiment fluctuation index, f3 represents content preference, f4 represents consumption decision, etc. The feature set N is the set of all these features: N = {f1, f2, ..., fn}; at this point, the feature vectors for each dimension have been extracted and formed into a feature set. Next, the system will use these features to calculate the Shapley value.

[0177] Step 3: Calculate the value of feature combinations. The Shapley value is calculated by evaluating the marginal contribution of each feature in different feature combinations. Specifically, the system needs to consider all feature combinations S and calculate the marginal contribution of each feature in these combinations. Assume S is a subset of the feature set N that does not contain the feature f whose Shapley value is to be calculated. i For each subset S, the system will evaluate the combination S and the combination S∪{f}. i The value difference of}. This difference represents the value difference of feature f. i The marginal contribution.

[0178] When calculating the value of a feature combination, the system inputs the feature set S into the pre-trained model and obtains the output value of the combination: v(S) = model output(S); where v(S) is the value of the feature combination S, representing the user profile result predicted by the model when the feature set is S.

[0179] Step 4: Calculate the marginal contribution. Once the value v(S) of the feature combination S is obtained, the system needs to calculate the current feature f. i The marginal contribution. Feature f i The marginal contribution is when feature f i The change in the model output value when added to the feature combination S. Specifically, the formula for calculating the marginal contribution is:

[0180] Marginal contribution i (S)=v(S∪{f i})-v(S);

[0181] Wherein, v(S∪{f i}) is to use feature f i The output value of the model after adding it to combination S represents the value of the model containing f. i The model predicts the results based on the given features.

[0182] Step 5: Calculate the Shapley Score. Once the marginal contribution of each feature in different combinations has been calculated, the next step is to calculate the Shapley score. The idea behind the Shapley score is to determine the contribution of each feature to the final user profile by taking a weighted average of the marginal contributions of all feature combinations. The purpose of weighting is to take into account the feature f i The frequency of occurrence in different combinations of features. In the calculation of the Shapley value, the marginal contribution of each combination is weighted according to the size of the combination. The calculation formula is:

[0183]

[0184] Where, φ i (v) is a feature f i The Shapley value represents the characteristic f i The contribution to the final portrait. S is a subset of the feature set N, and S does not contain element f. i v(S) is the value of the feature combination S (i.e., the model output). v(S∪{f i}) is a feature f i Feature combination S∪{f i The value of}. These are weight coefficients, representing the features f i The frequency of features appearing in combination S. In the Shapley value formula, lowercase n represents the total number of features. That is, n is the number of all features in the dynamic profile. For example, if the dynamic profile contains 5 features (instantaneous sentiment value, sentiment fluctuation index, content preference, consumption decision, etc.), then n = 5. The exclamation mark in the formula represents the factorial. The factorial is a mathematical operation that represents the product of a positive integer and all positive integers below it. In the Shapley value formula, the factorial is used to calculate the number of combinations, that is, the different permutations and combinations of features. For example, |S|! represents the factorial of the number of elements in the feature set S, representing the number of permutations of all features in the set. (n-|S|-1)! is the factorial of the remaining features, and n! is the total factorial of all features.

[0185] Step 6: Output Shapley Values. By weighting the marginal contributions of each feature, the Shapley value for each feature is calculated. A higher Shapley value indicates a more significant feature contribution.

[0186] The greater the contribution of a feature to the final user profile, the more accurate and detailed the analysis of the user profile can be. Through this calculation process, the system can quantify the impact of each feature on the dynamic user profile and generate a detailed analysis of the user profile. The final Shapley value can be used to analyze the importance of different features in the user profile, supporting further personalized recommendations and analysis report generation.

[0187] Specifically, in the tag generation unit, several multi-dimensional tags are generated based on the Shapley value and each dimension in the dynamic profile. The multi-dimensional tags include instantaneous sentiment value, sentiment fluctuation index, content preference degree, and consumption decision pattern.

[0188] Specifically, the instantaneous emotion value adopts a continuous real-time scoring mechanism, ranging from -1 to 1, with negative values ​​representing negative emotions and positive values ​​representing positive emotions.

[0189] Specifically, the sentiment fluctuation index is based on the variance statistics of sentiment values ​​within a certain time window, reflecting the stability and volatility of users' emotions. The formula for calculating the sentiment fluctuation index is as follows:

[0190]

[0191] In the formula, EI represents the sentiment fluctuation index, x i The sentiment value at each time point, μ is the average sentiment value, and n is the number of samples. The sentiment value is determined based on the Shapley value, which involves the sentiment dimension.

[0192] Specifically, the content preference is achieved through a nonlinear dimensionality reduction method. This involves compressing behavioral features (such as likes, views, and favorites) in the dynamic profile, reducing multidimensional features to a two-dimensional plane to achieve content tag clustering. Then, a nonlinear dimensionality reduction method is used to map high-dimensional features to a low-dimensional space, identifying cluster tags such as "suspense preference" and "urban sentiment." The preference intensity coefficient is calculated based on cluster density and behavioral frequency, normalized to the 0-1 range, where 0 represents complete disinterest and 1 represents extreme preference. The formula for calculating the content preference is as follows:

[0193]

[0194] In the formula, PI represents content preference, and w i f is the weight of the i-th cluster. i It is the behavior frequency of the i-th cluster, max(w) i ·f i ) is the maximum value of the product of the weights and behavior frequencies of all clusters.

[0195] Specifically, the consumption decision-making pattern is determined using Shapley values ​​from dimensions of the dynamic profile involving micro-expressions, eye tracking, and historical recharge records. For "impulsive" users, the system captures features such as decision time less than 3 seconds and a mouth corner upward angle greater than 15°; for "rational" users, it analyzes the stability of multiple look-back behaviors and pupil changes. Decision type identification uses a machine learning classifier combined with multimodal features to construct a probabilistic model of decision behavior. Key features include decision time, facial micro-expression angle, and eye movement trajectory stability, which are comprehensively judged by an ensemble learning algorithm to determine the user's decision type.

[0196] Specifically, the D3.js visualization library can be used to dynamically display the evolution path of user content preferences based on a timeline. Taking "preference for period dramas changing from 0.72 to 0.35, and preference for workplace dramas changing from 0.18 to 0.61" as an example, the dynamic migration process of user interests can be intuitively presented through color gradients and curve trajectories. In the visualization design, the change from cool to warm colors maps the change in emotional inclination, the smoothness of the curve reflects the degree of preference shift, and interactive charts allow users to explore the evolution of their personal profile in depth.

[0197] Specifically, in the analysis unit, the profile analysis report includes several multi-dimensional tags, the values ​​of each multi-dimensional tag, and the meaning of each multi-dimensional tag. The profile analysis report also includes the evolution path of content preferences to reflect the process of user preference changes.

[0198] Specifically, it also includes a dual-channel feedback mechanism, comprising an explicit feedback channel and an implicit feedback channel. The explicit feedback channel updates tag weights based on user-generated ratings (e.g., 1 to 5 stars); the implicit feedback channel monitors metrics such as viewing completion rate (>95% is considered a positive sample) and next-day retention rate. When a new user experiences a cold start, an initial profile is generated using a content similarity-based collaborative filtering algorithm, which is gradually replaced with multimodal analysis results over 24 hours. The model performance monitoring module continuously tracks the Area Under the Curve (AUC) and Normalized Discounted Cumulative Gain (NDCG) metrics. When NDCG@10 decreases by more than 5%, a full model retraining is automatically triggered to ensure continuous optimization of the profile system. The cold start refers to the generation of new user profiles, for which the system employs a content similarity-based collaborative filtering algorithm. In the initial stage, a sparse feature vector is constructed using user registration information and initial interest tags. Similarity calculation uses a cosine similarity algorithm, which matches the new user with the most similar historical user profiles to generate initial recommendations. Within 24 hours, the system gradually replaces the initial profile with multimodal deep analysis results, achieving rapid personalization.

[0199] Specifically, the dual-channel feedback mechanism also includes a performance monitoring mechanism to track key evaluation metrics, including AUC (Area Under the Curve) and NDCG (Normalized Depreciation Cumulative Gain). The NDCG@10 calculation formula is: NDCG@10 = DCG@10 / IDCG@10, where DCG (Depreciation Cumulative Gain) measures the relevance of the recommendation list, and IDCG is the optimal gain under ideal conditions. When the NDCG@10 metric drops by more than 5%, the system automatically triggers a full model retraining process. The retraining process includes incremental data acquisition, feature engineering reconstruction, model parameter resetting, multiple rounds of cross-validation, and canary release validation to ensure the continuous stability and improvement of model performance. Model iteration is based on a closed-loop feedback system, constructing a multi-dimensional, adaptive optimization framework. Strategies include dynamically adjusting feature weights, incremental learning mechanisms, multi-modal feature fusion, and model ensemble. By monitoring user interaction signals and model performance metrics in real time, the system can intelligently adjust the learning algorithm parameters. For example, when the contribution of certain features to user profile prediction decreases, their weights can be dynamically reduced using automatic feature selection algorithms (e.g., feature importance assessment based on mutual information). Model ensemble improves the robustness and generalization ability of the overall system by combining the prediction results of multiple different algorithms (e.g., deep neural networks, tree models, and probabilistic graphical models). During the user profile iteration process, the system strictly adheres to the principles of data desensitization and anonymization. Differential privacy technology is employed to introduce random noise during model training to protect individual privacy. Specifically, Gaussian noise is added during gradient updates to ensure that individual user data does not significantly affect the model output. The formula can be expressed as: Noisy gradient = Original gradient + N(0,σ²), where σ is the privacy budget control parameter.

[0200] Example 2

[0201] like Figure 7 As shown, this embodiment provides a method for analyzing user profiles of micro-dramas based on multimodal data, including:

[0202] S10: Acquire multimodal short drama user data;

[0203] S20: Based on the multimodal micro-drama user data, feature extraction is performed to obtain several user features;

[0204] S30: Input the aforementioned user features into a pre-trained hierarchical attention network model to obtain a joint feature vector;

[0205] S40: Input the joint feature vector into a preset online deep forest model to generate a dynamic profile;

[0206] S50: Calculate the Shapley value for each dimension of the dynamic portrait, and generate a portrait analysis report based on the Shapley value.

[0207] Specifically, step S10 includes the following steps:

[0208] S11: Obtain initial multimodal data from different data sources;

[0209] S12: Synchronize the timestamps of the initial multimodal data from different sources according to the network time protocol to obtain time-synchronized initial multimodal data;

[0210] S13: The initial multimodal data of the time synchronization is processed using cubic spline interpolation and noise suppression algorithm to obtain multimodal data.

[0211] Specifically, in step S10, the initial multimodal data includes initial behavioral data, initial user facial data, initial user voice data, and initial text data. The initial behavioral data includes click frequency (single / double click mode), swipe trajectory (direction / speed), pause / fast forward operation, and device gyroscope attitude data. The acquisition frequency of the initial behavioral data can be 10Hz. The initial user facial data is captured by the phone's front-facing camera at 30fps, and a 3D facial mesh with 62 key points is constructed in real time using the MediaPipe framework. Micro-expression parameters such as eyelid opening and closing, glabellar muscle displacement, and mouth corner upward angle are extracted. The initial user voice data is acquired through a microphone. The initial user voice data is the user's real-time voice comment. After eliminating environmental noise using a noise suppression algorithm, it is saved as a Pulse Code Modulation (PCM) format audio segment at a sampling rate of 16kHz. The initial text data is acquired in real time through the WebSocket protocol. The initial text data includes the content of the user's bullet comments and the sending time.

[0212] Specifically, step S20 includes the following steps:

[0213] S21: Use a temporal convolutional network to extract features from the behavioral data to obtain several first user features;

[0214] S22: Based on the visual database, a pre-trained target detection model is used to extract features from the user's facial data to obtain several second user features;

[0215] S23: Input the user's voice data into a pre-trained voice representation model for feature extraction to obtain several third-user features;

[0216] S24: Input the text data into a pre-trained language model for feature extraction to obtain the fourth user feature.

[0217] Specifically, in steps S21 to S24, the first user feature is obtained by weighting the viewing focus feature, content preference intensity feature, and interaction behavior feature;

[0218] The second user feature is obtained by weighting the features of the eye region, the mouth region, and the comprehensive facial features;

[0219] The third user characteristic is obtained by weighting the characteristics of excitement level, disappointment intensity, surprise index, anger level, fear level, and neutral emotion ratio.

[0220] The fourth user characteristic is obtained by weighting positive emotion intensity, negative emotion intensity, density of internet buzzword usage, frequency of emotion fluctuation, interaction activity, emotional consistency, group emotional synchronicity, and emotional depth.

[0221] Specifically, the contribution of the first, second, third, and fourth user features, as well as the features subordinate to each user feature, is dynamically adjusted through learning parameters. The weight calculation employs a softmax function, which non-linearly transforms the original features using a learnable weight matrix W and a bias term b to obtain the importance distribution of each feature. This adaptive weight learning mechanism enables the model to automatically adjust the weights of each modality feature based on different scenarios and user characteristics, achieving a more accurate user profile representation.

[0222] Specifically, step S30 includes the following steps:

[0223] S31: Preprocess several user features to obtain several preprocessed features;

[0224] S32: Divide the preprocessed features into several modalities, calculate the attention score of each preprocessed feature in each modal, and select key sub-features from each modal based on the attention scores;

[0225] S33: Use a cross-modal attention algorithm to calculate the pairwise correlation between modalities;

[0226] S34: Generate a joint feature vector based on the key sub-features and the correlation degree.

[0227] Specifically, in step S31, the preprocessing includes standardization and normalization. Since user features include data from multiple modalities (e.g., text, speech, facial expressions, etc.), for each modality m, its feature vector... Batch normalization or layer normalization can be used to eliminate scale differences in the feature spaces of different modalities, ensuring the fairness and stability of subsequent attention mechanism calculations. Normalization helps reduce the bias in feature distribution between modalities.

[0228] Specifically, in step S32, when dividing the preprocessed features into several modalities, the division is based on the data source of the preprocessed features. For example, speech-related preprocessed features are divided into a unified speech modal.

[0229] Specifically, in step S33, the semantic correlation between different modalities is calculated using the learnable query matrix Q, key matrix K, and value matrix V.

[0230] Specifically, in step S34, based on the correlation between modalities and several key sub-features, the key sub-features of different modalities are integrated through weighted summation or more complex fusion strategies. The joint feature vector comprehensively reflects the semantic information of each modality and the interaction relationships between modalities, providing rich multimodal representations for subsequent sentiment classification tasks.

[0231] Specifically, step S40 includes the following steps:

[0232] S41: Select several feature subsets of different granularities from the joint feature vector;

[0233] S42: Adjust the influence weights of several feature subsets according to a preset time decay factor;

[0234] S43: Input several feature subsets and influence weights into the pre-trained online deep forest model to obtain an initial user profile;

[0235] S44: Perform incremental learning at preset time intervals to update user profiles and obtain dynamic profiles.

[0236] Specifically, in step S41, the joint feature vector is processed using a random subspace method, feature importance evaluation method, or principal component analysis method to select feature subsets of different granularities. This method can effectively reduce dimensionality, prevent overfitting, and enhance the generalization ability of the model, enabling each base learner to capture subtle changes in user interests from different perspectives.

[0237] Specifically, in step S42, the attenuation factor is expressed as: λ = 0.98 tHere, t represents the data time difference in hours. By setting a decay factor, the weights of historical samples are dynamically adjusted, allowing the online deep forest model (ODF-MT) to focus more on recent behavioral patterns. This exponentially decays the influence of historical data; for example, data from 1 hour ago has a weight of approximately 0.98, data from 24 hours ago has a weight of 0.376, and data from 72 hours ago has a weight of only 0.014. Through this fine-grained weight adjustment strategy, the model can quickly respond to real-time changes in user interests, retaining the reference value of historical behavior while improving its sensitivity and adaptability to recent behavioral patterns, thereby achieving dynamic and accurate updates to user profiles.

[0238] Specifically, in step S43, the online deep forest model includes three incremental random forest models and two dynamic gradient boosting tree models. Each base learner receives a feature subset of different granularities. The base learner is a single classification or regression model in ensemble learning. By inputting the feature subset and the influence weights corresponding to each feature subset into different models, an initial user profile is obtained.

[0239] Specifically, in step S44, the preset time interval is preferably 30 minutes. The leaf node distribution is updated using a staggered binning strategy (please explain the staggered binning strategy). Nodes are automatically split when the KL divergence of the new data distribution exceeds the threshold θ = 0.15. Simultaneously, the system maintains the user state matrix U∈R. 128 Long-term and short-term interest features are fused through the LSTM update gate mechanism.

[0240] Specifically, step S50 includes the following steps:

[0241] S51: Calculate the Shapley value of each dimension in the dynamic image;

[0242] S52: Generate several multi-dimensional labels based on each dimension and the corresponding Shapley value in the dynamic image;

[0243] S53: Generate a profile analysis report based on the aforementioned multi-dimensional tags.

[0244] Specifically, in step S51, when extracting features from the dynamic portrait, each dimension of the dynamic portrait corresponds to a portrait feature, resulting in several portrait features. These features are then input into the Shapley library to calculate the Shapley value of each portrait feature.

[0245] Specifically, in step S52, several multi-dimensional tags are generated based on the Shapley value and the dimensions of the dynamic profile. These multi-dimensional tags include instantaneous sentiment value, sentiment fluctuation index, content preference, and consumption decision. The instantaneous sentiment value uses a continuous real-time scoring mechanism, ranging from -1 to 1, with negative values ​​representing negative emotions and positive values ​​representing positive emotions. The sentiment fluctuation index is based on the variance statistics of sentiment values ​​within a certain time window, reflecting the stability and volatility of user emotions. The content preference is achieved through a nonlinear dimensionality reduction method. By compressing behavioral features involved in the dynamic profile (e.g., likes, views, collections), the multi-dimensional features are compressed into a two-dimensional plane, achieving content tag clustering. Then, a nonlinear dimensionality reduction method is used to map high-dimensional features to a low-dimensional space, identifying cluster tags such as "suspense preference" and "urban sentiment." The preference intensity coefficient is calculated based on cluster density and behavioral frequency, normalized to the 0-1 range, where 0 represents complete disinterest and 1 represents extreme preference. The consumption decision is determined using Shapley values ​​from dimensions of the dynamic profile, including micro-expressions, eye tracking, and historical recharge records. For "impulsive" users, the system captures features such as decision time less than 3 seconds and a mouth corner upward angle greater than 15°; for "rational" users, it analyzes the stability of multiple look-back behaviors and pupil changes. Decision type identification uses a machine learning classifier combined with multimodal features to construct a probabilistic model of decision behavior. Key features include decision time, facial micro-expression angle, and eye movement trajectory stability, which are comprehensively judged using an ensemble learning algorithm to determine the user's decision type.

[0246] Specifically, in step S53, the profile analysis report includes several multi-dimensional tags, the values ​​of each multi-dimensional tag, and the meaning of each multi-dimensional tag. The profile analysis report also includes the evolution path of content preferences to reflect the process of user preference changes.

[0247] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0248] This invention also provides a computer-readable storage medium having a computer program stored thereon, wherein the program, when executed by a processor, implements any of the methods described above.

[0249] The present invention also provides an electronic device. The electronic device of this invention includes: one or more processors; and a storage device for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the micro-drama user profiling analysis method based on multimodal data provided by the present invention.

[0250] The following is for reference. Figure 8 It shows a schematic diagram of the structure of a computer system 800 suitable for implementing an electronic device according to embodiments of the present invention. Figure 8 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of use of the embodiments of the present invention.

[0251] like Figure 8 As shown, the computer system 800 includes a central processing unit (CPU) 801, which can perform various appropriate actions and processes based on programs stored in read-only memory (ROM) 802 or programs loaded from storage section 808 into random access memory (RAM) 803. The RAM 803 also stores various programs and data required for the operation of the computer system 800. The CPU 801, ROM 802, and RAM 803 are interconnected via a bus 804. An input / output (I / O) interface 805 is also connected to the bus 804.

[0252] The following components are connected to I / O interface 805: an input section 806 including a keyboard, mouse, etc.; an output section 807 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and speakers, etc.; a storage section 808 including a hard disk, etc.; and a communication section 809 including a network interface card such as a LAN card, modem, etc. The communication section 809 performs communication processing via a network such as the Internet. A drive 810 is also connected to I / O interface 805 as needed. A removable medium 811, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on drive 810 as needed so that computer programs read from it can be installed into storage section 808 as needed.

[0253] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can occur depending on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A micro-series user profiling analysis system based on multi-modal data, characterized in that, include: The data acquisition module is used to acquire user data for multimodal micro-dramas; The feature extraction module is used to extract features based on the multimodal micro-drama user data to obtain several user features; The fusion module is used to input several user features into a pre-trained hierarchical attention network model to obtain a joint feature vector; The dynamic portrait generation module is used to input the joint feature vector into a preset online deep forest model to generate a dynamic portrait. The portrait analysis module is used to calculate the Shapley value of each dimension of the dynamic portrait and generate a portrait analysis report based on the Shapley value; Specifically, the fusion module includes: The second preprocessing unit is used to preprocess several user features to obtain several preprocessed features; The intramodal attention unit is used to divide the preprocessed features into several modalities, calculate the attention score of each preprocessed feature in each modal, and select key sub-features from each modal based on the attention score. Cross-modal attention units are used to calculate the correlation between each pair of modalities using a cross-modal attention algorithm; A fusion unit is used to generate a joint feature vector based on the key sub-features and the correlation degree; The dynamic portrait generation module specifically includes: A subset building unit is used to select several feature subsets of different granularities from the joint feature vector; The attenuation unit is used to adjust the influence weights of several feature subsets according to a preset time attenuation factor. The profile generation unit is used to input several feature subsets and influence weights into the pre-trained online deep forest model to obtain an initial user profile; The update unit is used to perform incremental learning at preset time intervals to update the user profile and obtain a dynamic profile. The online deep forest model includes three incremental random forest models and two dynamic gradient boosting tree models. Each base learner receives feature subsets of different granularities. The base learner is a single classification or regression model in ensemble learning. By inputting the feature subsets and the influence weights corresponding to each feature subset into different models, an initial user profile is obtained.

2. The micro-series user profiling analysis system based on multi-modal data according to claim 1, characterized in that, The data acquisition module specifically includes: The data acquisition unit is used to acquire initial multimodal data from different data sources; A time alignment unit is used to synchronize the timestamps of the initial multimodal data from different sources according to the network time protocol, so as to obtain time-synchronized initial multimodal data. The first preprocessing unit is used to process the initial multimodal data of the time synchronization using cubic spline interpolation and noise suppression algorithm to obtain multimodal micro-drama user data.

3. The micro-series user profiling analysis system based on multi-modal data according to claim 1, characterized in that, The multimodal micro-drama user data includes behavioral data, user facial data, user voice data, and text data. The user features include several first user features, several second user features, several third user features, and several fourth user features. The feature extraction module specifically includes: The first extraction unit is used to extract features from the behavioral data using a temporal convolutional network to obtain several first user features. The second extraction unit is used to extract features from the user's facial data using a pre-trained target detection model based on a visual database, and obtain several second user features. The third extraction unit is used to input the user's voice data into a pre-trained voice representation model for feature extraction, thereby obtaining several third user features. The fourth extraction unit is used to input the text data into a pre-trained language model for feature extraction to obtain the fourth user feature.

4. The micro-series user profiling analysis system based on multi-modal data according to claim 1, wherein, The portrait analysis module specifically includes: The Shapley value calculation unit is used to calculate the Shapley value of each dimension in the dynamic image; The tag generation unit is used to generate several multi-dimensional tags based on each dimension and the corresponding Shapley value in the dynamic portrait. The analysis unit is used to generate a profile analysis report based on several of the multi-dimensional tags.

5. The micro-drama user profiling analysis system based on multimodal data according to claim 3, characterized in that, The first user characteristic is obtained by weighting viewing focus characteristics, content preference intensity characteristics, and interaction behavior characteristics; The second user feature is obtained by weighting the features of the eye region, the mouth region, and the comprehensive facial features; The third user characteristic is obtained by weighting the characteristics of excitement level, disappointment intensity, surprise index, anger level, fear level, and neutral emotion ratio. The fourth user characteristic is obtained by weighting positive emotion intensity, negative emotion intensity, density of internet buzzword usage, frequency of emotion fluctuation, interaction activity, emotional consistency, group emotional synchronicity, and emotional depth.

6. The micro-drama user profiling analysis system based on multimodal data according to claim 1, characterized in that, The dynamic portrait generation module further includes a dynamic feature enhancement unit, which is used to detect several key sub-features. When a special field appears in the several key sub-features, the weight of the key sub-feature with the special field is adjusted when generating the joint feature vector.

7. A method for analyzing user profiles of micro-dramas based on multimodal data, characterized in that, The method is based on the micro-drama user profiling analysis system based on multimodal data as described in any one of claims 1-6, and the method includes: S10: Acquire multimodal micro-drama user data; S20: Based on the multimodal micro-drama user data, feature extraction is performed to obtain several user features; S30: Input the aforementioned user features into a pre-trained hierarchical attention network model to obtain a joint feature vector; S40: Input the joint feature vector into a preset online deep forest model to generate a dynamic profile; S50: Calculate the Shapley value for each dimension of the dynamic portrait, and generate a portrait analysis report based on the Shapley value.

Images