Video character first impression personality analysis method and system based on large model

Abstract

The invention discloses a video character first impression personality analysis method and system based on a large model. The method comprises the following steps: extracting frames from a video, extracting visual features by using a Vision Transform, and obtaining initial visual semantic representation through a multi-modal large model; an Attn-LoRA module is used for carrying out self-adaptive adjustment on the expression; a two-stage training strategy is adopted, Attn-LoRA parameters are optimized through gender and expression recognition tasks in a pre-training stage, and a prompt template is constructed according to a big five personality theory in a fine tuning stage to improve personality prediction performance; and finally, inputting the processed features into a large language model, and outputting scores of a character in five dimensions of a large five personality. The system includes corresponding modules to perform the analysis. According to the method, personality related clues are enhanced through the Attn-LoRA module, and the feature fusion efficiency is improved; the two-stage strategy accelerates convergence and reduces overfitting; the special prompt template solves the discrete generation limitation of a large model in a regression task, and has the advantages of less parameter growth, low reasoning delay and high adaptability.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and personality analysis technology, and in particular to a method and system for first impression personality analysis of video characters based on a large model. Background Technology

[0002] Currently, with the rapid evolution of social media and short video platforms, video data recording human behavior is growing exponentially. Accurately analyzing personality traits from video content has become an important research direction in the interdisciplinary field of computer vision and psychology. Personality analysis, as a key approach to understanding individual behavioral patterns, has traditionally relied mainly on subjective report-based questionnaires or text analysis based on a single modality. However, these methods have significant shortcomings: subjective reports are easily influenced by participants' self-perception errors and response fatigue, making it difficult to objectively present true personality traits; while independent modality analysis methods can only obtain information from a single channel, which is insufficient to comprehensively characterize the complex personality traits exhibited by individuals in multimodal interactions.

[0003] Early personality analysis research primarily focused on single-modal data. Textual analysis methods inferred personality traits by analyzing linguistic features (such as word preferences and syntactic patterns), but failed to encompass key indicators of nonverbal behavior. Audio research emphasized tonal features and prosodic patterns, but could not effectively capture subtle changes in facial expressions through the visual channel. While visual analysis methods could identify facial expressions and body movements, they lacked in-depth analysis of semantic connotations. Due to the limited information dimensions, these single-modal methods struggled to effectively address the dynamic presentation of personality traits in real-world situations.

[0004] To overcome the limitations of unimodal analysis, multimodal personality analysis techniques have emerged. This technology improves analytical accuracy by integrating multi-source information such as visual, auditory, and textual data; for example, it uses convolutional neural networks to extract visual features and combines them with recurrent neural networks to process sequential text. However, these methods still face challenges such as insufficient modality fusion efficiency and inadequate depth in feature correlation mining. Furthermore, existing models are mostly built on limited samples, making them difficult to adapt to complex social environments, and they generally rely on manual feature engineering, failing to achieve a full understanding of the deep semantics of personality traits.

[0005] Large Language Models (LLMs) have demonstrated exceptional representational capabilities in natural language understanding tasks, prompting researchers to apply them to the field of personality computation. For example, cue learning techniques are used to drive large models to parse personality indicators from text; however, such methods are limited to the text modality and fail to encompass the rich visual information in videos (such as facial micro-expressions and body language). Although multimodal large models (such as Video-LLaVA) are capable of processing video data, the semantic differences between pre-trained data and personality analysis tasks make it difficult to fully extract personality-related cues from visual features when directly deployed.

[0006] More importantly, current video personality analysis research largely focuses on "first impression" assessments of single video clips, failing to effectively capture the stability of personality traits across different scenarios. Trait activation theory in psychology suggests that individual personality needs to be dynamically presented in diverse scenarios, but existing methods lack mechanisms for integrating multi-scenario data, leading to significant interference with prediction results from specific scenarios. Furthermore, the scarcity of high-quality, multi-scenario video personality datasets further limits the progress of research on stable personality modeling.

[0007] The present invention is compared with the patent CN118228194B entitled "A Multimodal Personality Prediction Method and System Integrating Spatiotemporal Graph Attention Network" as shown below.

[0008]

[0009] The present invention is compared with the patent CN119323002B entitled "Multimodal Personality Perception Method and Device for Attention to Multiple Related Features and Graph Relationships" as shown below.

[0010]

[0011] The present invention is compared with the patent CN119068393B entitled "Personality Prediction Method and Device Integrating CLIP and Adaptive Graph Transformation Network" as shown below.

[0012]

[0013] In summary, the current field of video personality analysis faces three major technical challenges: ① Traditional unimodal and multimodal methods are inefficient in fusing multi-source information from videos; ② Large language models lack dedicated optimization mechanisms for personality-related visual features in video personality analysis; ③ Predictions based on single video segments cannot reflect the stability and cross-scene consistency of personality. This patent addresses these issues by proposing a first-impression personality analysis scheme for video characters based on a large model. Through attention-driven feature adaptation, a two-stage training strategy, and a multi-scene stable personality analysis framework, it achieves accurate modeling and prediction of the personality of characters in videos. Summary of the Invention

[0014] This invention addresses the limitations of existing video personality analysis methods, such as single-modal information constraints, insufficient adaptation of large-scale visual features, and poor stability of single-scene prediction. It provides a method and system for analyzing the first impression of video characters based on a large language model, achieving multimodal information fusion and accurate modeling of personality characteristics.

[0015] This invention discloses a method for analyzing the first impression personality of video characters based on a large model. The method includes the following steps:

[0016] (1) Video preprocessing and multi-level feature extraction

[0017] Frame extraction is performed on the input video to obtain a video frame sequence. A pre-trained VisionTransformer (ViT) model is used to extract visual features from the video frames. The visual encoder first scales the image and divides it into n image blocks. After tiling the pixels of each block from two dimensions to one dimension, the sequence is embedded and positionally encoded. These features are then fed into the modality adapter of the multimodal large model to generate an initial visual semantic representation.

[0018] (2) Adaptive feature correction of the Attn-LoRA module

[0019] The attention-guided low-rank adapter (Attn-LoRA) adaptively adjusts the initial visual semantic representation. The specific steps are as follows: the scaling dot product attention mechanism is used to calculate the attention weights of the visual feature sequence, the low-rank matrix factorization is used to generate feature correction terms, and the correction terms are integrated with the original features to drive the model to focus on personality-related visual cues.

[0020] (3) Optimization of two-stage training strategy

[0021] During the pre-training phase, video character gender recognition and facial expression recognition are used as supervised tasks. These tasks require the model to focus on key visual information of the character, such as facial features, hairstyle, or body contour. By constructing cue templates, the model is encouraged to focus on the main character and facial features, thereby optimizing the Attn-LoRA parameters.

[0022] During the fine-tuning phase, personality analysis prompt templates were designed based on the Big Five personality theory. Video visual features and speech-transcribed text were input into the model, and the cross-entropy loss function was used to enhance personality prediction performance. Minimizing this loss was the training objective of the fine-tuning phase.

[0023] (4) Personality score prediction and assessment

[0024] The processed multimodal features are input into a large language model to generate personality scores for the video character in five dimensions: openness, conscientiousness, extraversion, agreeableness, and neuroticism, with the score range being [0,1].

[0025] (5) System Function Demonstration

[0026] This prototype system, based on the real-time requirements of the task and the need for user interaction, is divided into offline and online components. The offline component performs dataset collection, construction, data preprocessing, and model training, providing basic support for system operation. The online component covers backend business logic processing, frontend interactive interface development, and data visualization implementation, thereby meeting users' needs for rapid interaction. The system integrates functional interfaces such as login and registration, interactive analysis, user data query, experimental result presentation, and dataset visualization. Through various interactive methods, users can perform operations such as video personality analysis, data retrieval, and visualization analysis.

[0027] As one embodiment of the present invention, the step (1) of extracting visual features of video frames using a pre-trained Vision Transformer (ViT) model specifically includes:

[0028] The video frames are scaled and divided into image patches. An input sequence is generated through linear embedding and positional encoding. Visual features are extracted using the ViT Transformer layer, and the output of the penultimate layer is taken as the visual feature representation, as shown in the following formula:

[0029]

[0030]

[0031] in, Let E be the i-th image patch, and E be the embedding matrix. For category embedding vectors, For position encoding.

[0032] As a specific embodiment of the present invention, the video preprocessing and multi-level feature extraction are as follows:

[0033] (1) Video frame extraction and standardization

[0034] The input video is uniformly frame-stripped at a frame rate of 25 frames per second. For videos with an original frame rate lower than 25 fps, linear interpolation is used to supplement the frames, while for videos with a frame rate higher than 25 fps, redundant frames are removed using optical flow to ensure the temporal consistency of the input video sequence.

[0035] (2) Visual feature extraction

[0036] Visual features are extracted using a pre-trained Vision Transformer (ViT-Base / 16) model:

[0037] The video frames were scaled to 224×224 pixels and divided into 14×14 image blocks. Each block was mapped to a 768-dimensional vector through linear embedding.

[0038] The input sequence is formed by introducing a learnable category embedding vector E_cls and a sine / cosine position code E_pos.

[0039]

[0040]

[0041] After passing through 12 layers of Transformer encoder, the output of the 11th layer is taken as the visual feature representation V_feat ∈ℝ^(N+1)×768.

[0042] (3) Multimodal feature fusion

[0043] The visual features V_feat are input into the modality adapter of the Video-LLaVA model, and the visual-language feature space mapping is achieved through two layers of MLP:

[0044]

[0045] in ∈ ℝ^768×4096, ∈ ℝ^4096×4096, output ∈ ℝ^(N+1)×4096

[0046] As one embodiment of the present invention, the adaptive feature correction of the Attn-LoRA module is as follows:

[0047] (1) Attention-driven low-rank correction

[0048] An attention-driven low-rank correction (Attn-LoRA module) is introduced into the linear layer of the modality adapter to dynamically correct feature V':

[0049]

[0050]

[0051]

[0052] Where f is the input feature sequence, / / ∈ ℝ^4096×r is the attention mapping matrix, ∈ℝ^r×4096 is the up-dimensional matrix, b is the parameter of the linear layer, r=128 is the low-rank dimension, and α=512 is the scaling factor.

[0053] (2) Enhanced visual cues related to personality

[0054] Through attention weight visualization analysis, Attn-LoRA focuses on facial regions (eyes / mouth) and limb movement regions. Experiments show that this module improves facial feature attention by 37% and limb movement feature extraction efficiency by 29%.

[0055] As one embodiment of the present invention, the two-stage training strategy is optimized as follows:

[0056] (1) Pre-training stage (focusing on visual features of people)

[0057] Gender recognition task: Use the prompt template "User: Please tell me the gender of the person in the video. Please only output male or female, do not output any other content."

[0058] Assistant: Gender.

[0059] Facial expression recognition task: Use the DeepFace tool to generate 7 types of facial expression labels. The prompt template is: "User: Please pay attention to the facial expressions of the people in the video and choose one of the following words to describe the emotions of the people: happy, surprised, neutral, sad, angry, disgusted, and fearful."

[0060] Assistant: Emotions.

[0061] Training objective: Minimize the cross-entropy loss for gender / facial expression classification, with a learning rate of 5e-5 and training for 2 epochs.

[0062] (2) Fine-tuning stage (personality dimension prediction)

[0063] Based on the Big Five personality theory, the following prompt template is designed: "User: The person in the video said, 'Transcribe text.' Now, please act as a psychology expert and, according to the Big Five personality trait theory, assess the degree to which this person performs on each trait dimension, providing a numerical value between 0 and 1. A smaller value indicates a weaker trait, and a larger value indicates a stronger trait. Please only output the numerical value; do not output any other content."

[0064] Assistant: Score.

[0065] Among them, "transcribed text" refers to text transcribed from speech, "traits" refers to the Big Five personality dimensions, and "trait descriptions" refers to text explaining the dimensions.

[0066] Teacher-mandated training was used to convert real personality scores into string labels (such as "0.6"), and the cross-entropy loss between the generated sequence and the label was calculated.

[0067] We introduced cosine learning rate decay, with a warmup ratio of 0.03, and trained for 4 rounds.

[0068] As one embodiment of the present invention, the personality score prediction and assessment are specifically as follows:

[0069] (1) Multidimensional score generation

[0070] The processed multimodal features are input into the Vicuna-7B model, which generates personality scores for each dimension through autoregression. The output format is as follows:

[0071] ASSISTANT: Openness: 0.65, Conscientiousness: 0.72, ...

[0072] Use regular expressions to extract numerical values ​​and normalize them to the [0,1] interval.

[0073] (2) Performance evaluation indicators

[0074] Average accuracy:

[0075]

[0076] Root Mean Square Error (RMSE):

[0077]

[0078] Pearson correlation coefficient (PCC):

[0079]

[0080] As one embodiment of the present invention, the system functions are specifically demonstrated as follows:

[0081] The system's functionality includes a personality analysis interactive interface, experimental results visualization components, dataset statistical display units, and a model performance comparison module. Specifically:

[0082] The personality analysis interface supports user uploads of MP4 and AVI video files. The system automatically performs frame extraction and generates Big Five personality traits scores (covering openness, conscientiousness, extraversion, agreeableness, and neuroticism) using a model. Analysis results are presented as a radar chart, along with textual explanations for each dimension (e.g., "Extraversion score of 0.72 reflects high activity levels in social situations"). The interface also supports user input of video audio transcribed into text to improve the accuracy of the model's analysis.

[0083] The experimental results visualization component presents the results of comparative experiments based on the First Impression dataset, including bar chart comparisons of accuracy, RMSE, and PCC metrics. Users can switch between different models (including Attn-LoRA, LoRA, and the baseline model) to observe their performance metrics across various personality dimensions, with specific values ​​displayed when the mouse hovers over them.

[0084] Dataset statistics display interface: The distribution of personality scores in the First Impression dataset is displayed in the form of a histogram, reflecting the sample density of each dimension (0-1 points), and the correlation analysis results between demographic variables such as gender and race and personality traits are also presented.

[0085] Model Performance Comparison Unit: This unit presents performance comparison data between the patented solution model and traditional methods (including Histogram, Spectrum, and NJU-LAMDA), allowing users to filter and view data based on indicator type (accuracy / RMSE / PCC) and personality dimensions. The system automatically identifies the improvements made by this model on key indicators, including significant advantages in the conscientiousness dimension (PCC indicator improved by 20%) and the extraversion dimension (RMSE indicator reduced by 15%).

[0086] The video character first impression personality analysis system based on a large model of the present invention includes the following core modules:

[0087] Video preprocessing module: Based on OpenCV, it implements video frame extraction (normalized to 25fps), image scaling (224×224), and abnormal frame filtering.

[0088] Feature extraction and adaptation module: Integrates ViT visual encoder and Attn-LoRA adapter to enhance personality-related cues from visual features.

[0089] The prompting module dynamically generates prompt templates for the pre-training (gender / facial expression recognition) and fine-tuning (personality prediction) stages, and supports user-defined prompt words.

[0090] Results visualization module: Uses ECharts to draw a five-dimensional personality radar chart and a feature attention heatmap (showing the model's attention weight to facial / limb regions).

[0091] This patent also introduces a video-based first impression personality analysis system based on a large model. The system consists of a video preprocessing module, an Attn-LoRA module, a two-stage training module, a personality prediction module, and an interactive interface module. The video preprocessing module performs frame rate normalization and visual feature extraction on the input video; the Attn-LoRA module improves visual features through an adaptive correction mechanism to enhance the model's ability to capture personality-related cues; the two-stage training module sequentially implements pre-training and fine-tuning stages to optimize model performance; the personality prediction module outputs scores for the video person on five personality dimensions based on the processed features; and the interactive interface module provides users with a convenient video upload channel and visualizes the analysis results, realizing a complete workflow from video input to personality analysis result generation.

[0092] Beneficial effects: Compared with the prior art, the present invention has the following significant advantages:

[0093] Improved performance of multimodal feature fusion: The Attn-LoRA module is used to dynamically enhance personality-related visual cues, resulting in an average increase of 14% in PCC on the First Impression dataset, with the responsibility dimension increasing by 20%, which is significantly better than traditional multimodal models (such as the TMIN model).

[0094] The effectiveness of the two-stage training scheme: the pre-training stage drives the model to focus on the subject, which improves the convergence rate of the fine-tuning stage by 50% and reduces the training cycle from 8 rounds to 4 rounds. At the same time, it effectively suppresses overfitting. Ablation experiments show that pre-training leads to an average decrease of 8.7% in RMSE.

[0095] Improved adaptability of large models: The prompt templates developed specifically for video personality analysis tasks overcome the limitations of large models in generating discrete outputs in regression tasks. By using string label mapping technology, the accuracy of numerical prediction is improved to one decimal place, which meets the standard requirements of psychological assessment.

[0096] Engineering deployment advantages: The model parameter size increases by only 0.3% (Attn-LoRA module), the inference time remains at 2.3 seconds per video (for a 15-second clip), and it supports dual deployment schemes of API interface and Web interface. It can be applied to various scenarios such as social platform user profile construction and occupational psychological assessment. Attached Figure Description

[0097] Figure 1This is the overall framework diagram of the present invention;

[0098] Figure 2 This is a flowchart of the invention;

[0099] Figure 3 This is the architecture diagram of the Video-LLaVA model used in this invention;

[0100] Figure 4 This is a diagram of the Vision Transformer model structure used in this invention;

[0101] Figure 5 This is a design diagram of the Attn-LoRA module used in this invention;

[0102] Figure 6 This is a template image for a gender determination task.

[0103] Figure 7 This is a diagram of an algorithm for extracting facial expression tags from video characters;

[0104] Figure 8 This is a template image for a human facial expression recognition task.

[0105] Figure 9 This is a template image for a personality analysis task.

[0106] Figure 10 It is a textual diagram describing personality traits;

[0107] Figure 11 This is the login interface of the prototype system of this invention;

[0108] Figure 12 This is the registration interface of the prototype system of this invention;

[0109] Figure 13 This is the personality analysis page of the prototype system of this invention;

[0110] Figure 14 This is the data query interface of the prototype system of this invention;

[0111] Figure 15 This is the experimental results page for the prototype system of this invention;

[0112] Figure 16 This is the data set analysis visualization interface of the system of the present invention. Detailed Implementation

[0113] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments:

[0114] The following is only one embodiment of the present invention. The present invention has many other embodiments. Without departing from the spirit and essence of the present invention, those skilled in the art can make various corresponding changes and modifications according to the present invention. All such changes and modifications should fall within the protection scope of the appended claims.

[0115] This patent addresses the challenges in large-scale video character first impression personality analysis, including insufficient multimodal information fusion efficiency, weak correlation between visual features and personality, and inadequate model output guidance. It proposes a system framework consisting of a multimodal large-scale model, an Attn-LoRA module, and an instruction prompting module. The multimodal large-scale model extracts features from video frames using a visual encoder and transforms these features into a text semantic space via a modal adapter. These features, along with prompt text, are then input into a large language model to generate personality prediction results. The Attn-LoRA module, integrated into the modal adapter path, uses an attention mechanism to dynamically correct visual features, thereby improving the capture efficiency of personality-related cues such as facial expressions and body language. The instruction prompting module includes gender and facial expression recognition instructions in the pre-training stage and analysis templates based on the Big Five personality theory in the fine-tuning stage. Through structured prompts, it drives the model to focus on key features and output standardized personality scores, ultimately achieving an end-to-end analysis process from video input to the quantification of five personality dimensions.

[0116] like Figure 1 As shown, the present invention provides a method for analyzing the first impression personality of video characters based on a large model, including as follows: Figure 2 The process includes the following steps:

[0117] 1. Video preprocessing and multi-level feature extraction

[0118] (1) The video preprocessing module is responsible for the normalization and feature extraction of the input video in this invention. Its operation process is as follows: In the initial stage, the video is sampled uniformly at a rate of 25 frames per second. For videos with an original frame rate lower than 25fps, the linear interpolation method is used to supplement the frames. For videos with an original frame rate higher than 25fps, the optical flow technology is used to remove redundant frames in order to maintain the temporal consistency of the video sequence. Then, the video frame size is adjusted to 224×224 pixels and divided into 14×14 image blocks. The input sequence is constructed using linear embedding and sine and cosine position coding. Visual features are extracted by a 12-layer VisionTransformer encoder, and the output of the 11th layer is selected as the visual feature representation. Finally, the visual features are input into the modal adapter of the Video-LLaVA model and mapped to the text semantic space through two layers of multilayer perceptron to achieve cross-modal alignment of visual and linguistic features, thereby providing normalized multimodal feature input for subsequent personality analysis.

[0119] This patent uses the existing large-scale multimodal model Video-LLaVA as its core architecture. This model establishes a unified visual representation space, enabling large language models to process image or video data input. Video-LLaVA follows the standard multimodal large model design paradigm, and its structure mainly consists of a visual encoder, a modality adapter, and a large language model. The specific architecture is as follows: Figure 3 As shown.

[0120] (2) Visual encoder

[0121] Video-LLaVA employs the Vision Transformer (ViT) as its visual encoder, treating images as sequences and extracting features through a Transformer structure. For the input video frame sequence, ViT performs the same feature extraction process for each frame. The specific model structure is as follows: Figure 4 As shown, for input video frame images The visual encoder processes images using the following steps: Image segmentation: The image is scaled and divided into N image patches. Each patch's pixels are flattened from two dimensions to one dimension. The i-th image patch is represented as... Where H and W are the original image height and width, C is the number of channels, and P is the block resolution height and width. Sequence embedding and positional encoding: Convert an image x of arbitrary resolution into a sequence of length N with dimensions of... vector sequence Similar to BERT's mechanism for processing natural language, ViT first performs embedding and positional encoding on the sequence, with the specific calculation formula as follows:

[0122]

[0123] For the linear layer parameters of the image embedding module, [;] indicates vector concatenation; The embedded vector sequence; These are learnable embedding vectors, similar to the [CLS] tag in BERT, used to aggregate global sequence information;

[0124] Location encoding. Feature extraction: Deep image features are extracted by modeling long-distance dependencies using Transformer layers.

[0125]

[0126] in, =1,2,…, (L represents the total number of layers in the ViT Transformer). Indicates the first Layer Transformer,

[0127] It takes either the input or the output of the previous layer. Video-LLaVA takes the output of the second-to-last Transformer layer of the ViT. As visual features, for video data, the model extracts M video frames, which are then processed by ViT to obtain M visual features.

[0128] (3) Modal adapter

[0129] Since visual features are incompatible with the text embedding space of large language models, Video-LLaVA projects visual features to the text feature space through a modality adapter. The specific calculation is as follows:

[0130]

[0131]

[0132] in: Let be the visual features of the i-th video frame; It is a spliced ​​sequence of features from M video frames; For two linear layer parameters, , The bias is used; GeLU is used as the activation function. The modal adapter achieves spatial mapping and dimensional transformation through an MLP consisting of two linear layers, and outputs... Visual features in the text feature space.

[0133] (4) Large language model

[0134] Video-LLaVA uses Vicuna-7B v1.5 as the large language model, and the processing flow is as follows:

[0135] Text embedding: Input text is embedded to obtain text features;

[0136] Feature fusion: Text features are concatenated with visual features already mapped to the text space, and after positional encoding, they are input into the Transformer layer;

[0137] Joint processing: The large model processes visual and textual information simultaneously to generate corresponding outputs.

[0138] This invention achieves cross-modal fusion processing of video and text information through the collaborative design of the aforementioned visual encoder, modal adapter, and large language model, and has the technical effect of efficiently extracting visual features and generating semantic understanding output.

[0139] 2. Adaptive Feature Correction of the Attn-LoRA Module

[0140] (1) Technical issues and design motivation

[0141] Video-LLaVA, as a general-purpose multimodal large model, exhibits certain limitations in personality analysis and prediction tasks: although the visual features mapped through the modal adapter contain rich visual content, the key features used for personality judgment are not fully highlighted, making it difficult for the large model to effectively capture this information, thus affecting prediction performance. Given the high cost of retraining the modal adapter, there is an urgent need to develop an efficient fine-tuning scheme.

[0142] Based on the analysis of formula (3), the existing modal adapter is essentially a multilayer perceptron (MLP), which can only perform fixed and indiscriminate transformations on different visual features, lacking dynamic adjustment capabilities, and also unable to identify key information from the context of feature sequences. Therefore, inspired by LoRA fine-tuning technology and attention mechanisms, this invention introduces an attention-driven low-rank adapter (Attn-LoRA), the specific design architecture of which is as follows: Figure 5 As shown.

[0143] (2) Module architecture and working principle

[0144] Attn-LoRA operates on two linear layers of the modal adapter, with the input vector sequence of the linear layers simultaneously fed into the Attn-LoRA module. After introducing Attn-LoRA, the computation flow of the linear layers is as follows:

[0145]

[0146]

[0147]

[0148] in: The input linear layer is a vector sequence (length n, dimension n). ); , represents the original parameters of the linear layer, which are kept frozen during fine-tuning; For the learnable linear layer parameters of Attn-LoRA, respectively... The mapping is a query, key, and value vector, where r is the rank of the low-rank decomposition (r ≪ ); The output after attention calculation. For parameters of the linear layer in higher dimensions, Output for the module;

[0149] α is a scaling factor that adjusts the weight of Attn-LoRA's influence on the original output. This represents the scaling ratio.

[0150] (3) Parameter efficiency advantage

[0151] Attn-LoRA inherits the high parameter efficiency of LoRA. A comparison of parameter counts for different methods is as follows: Direct fine-tuning of linear layers: requires updating... One parameter; attention adapter for non-low-rank decomposition: at least update (3) + )× One parameter; Attn-LoRA: Only needs to be learned (3) + With 1000 × r parameters, and since r is a small constant, the training overhead is significantly reduced.

[0152] (4) Technical differences from traditional LoRA

[0153] Traditional LoRA uses fixed linear transformations in a low-dimensional space to update parameters. Its pre-defined dimensionality compression method limits its adaptability to complex features and cannot handle the global contextual information of serialized data. The Attn-LoRA proposed in this patent integrates an attention mechanism in the dimensionality reduction step, enabling adaptive changes to the mapping relationship based on the input data. This achieves context-based adaptive adjustment of the linear layer output, significantly enhancing the model's ability to capture key visual features in personality analysis tasks. By constructing the Attn-LoRA module, dynamic improvements in visual feature processing are achieved with only a small increase in parameters, avoiding the reconstruction of the original modality adapter, successfully addressing the challenge of key feature extraction in multimodal models for personality analysis tasks.

[0154] 3. Optimization of the two-stage training strategy

[0155] While Attn-LoRA can dynamically respond to input data, in personality analysis tasks, it still needs to be guided to focus on visual information (such as facial features and facial expressions) that plays a crucial role in personality judgment. To address this, this patent proposes a two-stage training framework of "pre-training-fine-tuning," utilizing a prompting module to drive the model to learn task-specific features at different stages. Both training stages rely on prompts to guide the large model to execute tasks and generate specified outputs. The design of the prompting scheme will be detailed below. The prompting module of this patent adopts a hierarchical design, achieving progressive improvement of Attn-LoRA through differentiated prompting strategies in the pre-training and fine-tuning stages.

[0156] (1) Pre-training phase

[0157] The pre-training phase of this invention aims to enable Attn-LoRA to initially acquire the ability to extract visual information about people from videos. To achieve this goal, capability training is accomplished by creating pre-training task prompts based on people's visual attributes. Specifically, a gender recognition task is first constructed, which guides the model to focus on key visual features of people, such as facial features, hairstyles, and body contours. The design is as follows... Figure 6 The gender recognition task prompt template shown uses "User:" and "Assistant:" to identify the roles of the user and the large model assistant during the dialogue, clarifying their identities and scope of speech in the interaction. In the dialogue flow, the user asks questions or assigns tasks to the large model using natural language commands, and the large model generates corresponding responses or performs specified operations based on the user's commands. In this prompt template, the user commands restrict the large model to output only the gender information of the people in the video. The gender variable within the template serves as the gender label generated by the model, and its value is limited to the English words "male" or "female," corresponding to "male" and "female," respectively. Next, an expression recognition task is constructed, requiring the model to focus on changes in facial expressions. Facial expressions in video frames are extracted using the open-source face analysis tool DeepFace, and then... Figure 7 The algorithm shown obtains facial expression tags for people in the video and simultaneously establishes... Figure 8 The template shown is for an expression recognition task. In this template, the user instruction specifies that the large model outputs the facial expressions of people in the video. The emotion variable in the template is the expression label generated by the model, and its value is one of the English words representing emotions, such as "happy," "surprise," "neutral," "sad," "angry," "disgust," "fear," etc. During the pre-training phase, Video-LLaVA achieves the following... <gender>or <emotion>The prediction is achieved through the following formula:

[0158]

[0159] in, Text prompts indicating input during the pre-training phase; This indicates the introduction of the Attn-LoRA Video-LLaVA model into the modal adapter; express The trainable parameters are frozen during training and do not participate in fine-tuning, therefore... Specifically, these are the parameters of Attn-LoRA; This is the prediction of gender or facial expression labels for a person in video V by a large model. The goal is to minimize... The cross-entropy loss with the true value is used as the optimization objective during the pre-training phase.

[0160] (2) Fine-tuning stage

[0161] After the pre-training phase, the Video-LLaVA modal adapter has initially developed the ability to focus on information about people in videos. The fine-tuning phase of this invention aims to further enable it to learn and extract key information related to personality. Based on this, a personality dataset is constructed as follows... Figure 9 The prompt template shown:

[0162] The transcribed text consists of the transcripts of the voices in the video; the trait represents one of the five trait dimensions in the Big Five personality traits (openness, conscientiousness, extraversion, agreeableness, neuroticism); the score represents the score predicted by the large model under each trait dimension; and the trait description is an explanatory text describing the trait, with the description text for each trait as follows: Figure 10 As shown in the table.

[0163] During the fine-tuning phase, Video-LLaVA predicts the score using the following formula:

[0164]

[0165] in, Text prompts entered during the fine-tuning phase The predicted personality trait scores generated for the large model. Since the large model is a generative model, its output predictions are sequences of characters, rather than directly numerical values. Therefore, when calculating the loss, the true values ​​are converted into their corresponding string forms and used together with the character sequences generated by the model to calculate the cross-entropy loss. Minimizing this loss is used as the training objective during the fine-tuning phase.

[0166] 4. Personality Score Prediction and Assessment

[0167] (1) Multidimensional score generation

[0168] The processed multimodal features are input into the Vicuna-7B model, which generates personality scores for each dimension through autoregression. The output format is as follows:

[0169] ASSISTANT: Openness: 0.65, Conscientiousness: 0.72, ...

[0170] Extract numerical values ​​using regular expressions and normalize them to the [0,1] interval.

[0171] (2) Performance evaluation indicators

[0172] The evaluation index system of this invention is applicable to regression tasks for analyzing the first impression personality of video characters. Referring to existing research, it uses mean accuracy, root mean square error (RMSE), and Pearson correlation coefficient (PCC) as the core indicators for quantifying model performance, specifically defined as follows:

[0173] Average accuracy measures the overall closeness of a model's predictions by calculating the mean absolute deviation between predicted and actual values. The formula is:

[0174] Where n is the number of samples in the test set. and and represent the predicted and actual values ​​of the personality traits for the i-th sample, respectively. The closer this index is to 1, the smaller the absolute deviation between the predicted and actual values, and the higher the model's prediction accuracy.

[0175] The root mean square error (RMSE) is calculated by taking the square root of the mean square of the deviations between the predicted and actual values, thus enhancing sensitivity to larger errors. The formula is:

[0176]

[0177] This metric amplifies the prediction error by square, effectively reflecting the stability of the model's predictions. A smaller RMSE value indicates a stronger ability to control prediction errors, especially when handling outliers.

[0178] The Pearson correlation coefficient (PCC) quantifies the degree of linear correlation between predicted and actual values ​​by calculating the ratio of the covariance to the standard deviation. The formula is:

[0179] in, The covariance between the predicted and actual values. and , respectively, are the standard deviations of the two. The PCC value ranges from [−1, 1]. The closer the absolute value is to 1, the stronger the model's ability to capture the overall trend of the data, and it is especially suitable for evaluating the fit of personality traits to the distribution patterns in a group.

[0180] The above indicator system characterizes model performance from different dimensions: Accuracy focuses on single-point prediction accuracy, RMSE emphasizes error stability control, and PCC measures the ability to fit data trends. The combination of the three can comprehensively evaluate the effectiveness of the video personality analysis model in regression tasks.

[0181] 5. System Function Demonstration

[0182] Based on the real-time nature of the task and user interaction requirements, this prototype system is divided into offline and online components. The offline component integrates dataset acquisition, data preprocessing, and model training modules. Given its high computational intensity, long processing time, and low correlation with real-time interaction, it is deployed in an offline environment to provide data and model support for the online component. Specifically, the data acquisition module uses a MySQL database to store the structured information of the video personality dataset; the data preprocessing module uses Python to perform data cleaning tasks; and the model training module relies on Python deep learning libraries such as PyTorch and Transformers to build a large-scale video first impression personality analysis model and a multi-scenario stable personality analysis model, and then trains these models. The online component adopts a browser / server (B / S) architecture and a front-end / back-end separation design. The back-end business logic is implemented using the Django framework, while the front-end uses the Vue.js framework to develop an interactive interface. Combined with tools such as Echarts, it visualizes the data and results, aiming to meet users' needs for rapid interaction. Through the interactive front-end interface, the trained model is invoked for real-time inference, and the data and prediction results are visualized and analyzed. The system includes functional pages for login and registration, interactive analysis, user data query, experimental results display, and dataset visualization: users can... Figure 11 Enter the correct account and password on the login screen shown to access the system. New users can be redirected to... Figure 12 Complete registration on the shown registration page; after entering the system, Figure 13 The left navigation bar supports switching between function pages, while the right main page allows users to interact with the backend model via text input boxes and video upload buttons to perform video personality analysis; the user data query page ( Figure 14 The system provides a dataset video information query interface, supporting filtering queries by video ID; the experimental results page displays model and ablation experiment results in an interactive multi-bar chart format; the dataset visualization page presents the feature distribution of different video personality datasets in chart format, allowing users to obtain detailed data information through mouse interaction. Experimental results are available in... Figure 15 The page displayed presents the ablation study results of a video-based first impression personality analysis model based on a large-scale model, as well as the experimental results of a multi-scenario stable personality analysis model. These results are visualized in multi-bar charts. When the user hovers the mouse over a corresponding data bar, the system automatically activates the highlighting function, displaying the specific value in real time for easy understanding of the data magnitude. Simultaneously, users can selectively show or hide data bars for specific categories by clicking the category labels below the charts. Figure 16 The page provides visualization and analysis functions for the two video personality datasets involved in this study: For the first impression personality dataset, the page displays the distribution of scores for the five personality trait dimensions within the range of 0 to 1; for the self-constructed multi-scene video dataset, the left side of the page displays the overall distribution of the 16 MBTI personality types via a pie chart, while the right side features four pie charts analyzing the distribution of the four dimensions of MBTI respectively. The visualization results show that the distribution of the four dimensions is relatively even. By moving the mouse over the pie chart area, the system instantly provides the specific numerical information corresponding to that area, enabling interactive data query and analysis.< / emotion> < / gender>

Claims

1. A method for analyzing personality of a first impression of a video character based on a large model, characterized in that, The method comprises the following steps: (1) video preprocessing and multi-level feature extraction; Perform frame extraction on the input video to obtain a video frame sequence; extract visual features from the video frames using a pre-trained Vision Transformer model, and send these features to the modality adapter of the multi-modal large model to generate an initial visual semantic representation; (2) adaptive feature correction of the Attn-LoRA module; Adopt an attention mechanism guided low-rank adapter Attn-LoRA to implement adaptive adjustment on the initial visual semantic representation, the specific steps are: use the scaled dot-product attention mechanism to calculate the attention weight of the visual feature sequence, generate a feature correction item by means of low-rank matrix decomposition, and integrate the correction item with the original feature, thereby driving the model to focus on personality-related visual clues; (3) two-stage training strategy optimization; In the pre-training stage, video character gender recognition and facial expression recognition are used as supervised tasks, and a prompt template is constructed to encourage the model to focus on the character subject and facial features, thereby optimizing the Attn-LoRA parameters; In the fine-tuning stage, a personality analysis prompt template is designed based on the Big Five Personality Theory, the video visual features and the speech transcription text are input into the model, and the cross-entropy loss function is used to enhance the personality prediction performance; (4) personality score prediction and evaluation; Input the processed multi-modal features into a large language model to generate the personality scores of the video character in the five dimensions of openness, conscientiousness, extraversion, agreeableness, and neuroticism, with a score range of [0, 1]; (5) system function display; The prototype system is divided into offline and online parts based on task real-time and user interaction needs; the offline part performs data set collection, construction, data preprocessing and model training to provide basic support for system operation; the online part covers backend business logic processing, frontend interface development and data visualization implementation to meet the user's demand for fast interaction; the system integrates login and registration, interactive analysis, user data query, experimental result presentation and data set visualization function interfaces, and uses various interaction methods to enable users to perform video personality analysis, data retrieval and visualization analysis operations.

2. The large model-based video character first impression personality analysis method according to claim 1, characterized in that, In step (1), the visual features of the video frames are extracted using a pre-trained Vision Transformer model, specifically: Scale and divide the video frames into image blocks, generate an input sequence through linear embedding and position encoding, extract visual features through the Transformer layer of Vision Transformer, and take the output of the second-to-last layer as the visual feature representation, as follows: ； ； wherein, is the i-th image block, E is the embedding matrix, is the class embedding vector, is the position encoding.

3. The large model-based video character first impression personality analysis method according to claim 1, characterized in that, The specific implementation of the Attn-LoRA module in step (2) is: Introduce an attention-driven low-rank correction to the linear layer of the modality adapter, as follows: ； ； ； wherein, is an input feature sequence, , , is an attention mapping matrix, is a rank-r matrix, b is a parameter of the linear layer, and r is a low-rank dimension, is a scaling factor.

4. The large model-based video character first impression personality analysis method according to claim 1, characterized in that, In step (3), the prompt template design in the pre-training stage is: Gender recognition prompt template: "User: Please tell me the gender of the person in the video. Please output only male or female, and do not output any other content. Assistant: Gender"; The expression recognition prompt template is: "User: Please pay attention to the facial expression of the person in the video, and choose one from the words "happy, surprised, neutral, sad, angry, disgusted, and scared" to describe the person's emotion;" Assistant: Emotion ".

5. The large model-based video character first impression personality analysis method according to claim 1, characterized in that, The prompt template of the fine-tuning stage in step (3) is: "User: The person in the video said "transcription text", now please evaluate the performance of this person on the trait dimension according to the Big Five Personality Traits Theory, and give a decimal value between 0 and 1, the smaller the value, the weaker the performance of the trait, the larger the value, the stronger the performance; please only output the value, do not output any other content;" Assistant: Score ". Wherein, "transcription text" is the speech transcription text, "trait" is the Big Five Personality Dimension, and "trait description" is the dimension interpretation text.

6. The large model-based video character first impression personality analysis method according to claim 1, characterized in that, The evaluation indicators of the personality score prediction in step (4) include: Average accuracy Accuracy, root mean square error RMSE and Pearson correlation coefficient PCC, the calculation formula is as follows: ； ； ； wherein, is the predicted score, is the true score, cov is the covariance, is the standard deviation.

7. A large model-based video character first impression personality analysis system, characterized in that, The system includes a video preprocessing module, an Attn-LoRA module, a two-stage training module, a personality prediction module, and an interactive interface module; the video preprocessing module performs frame rate standardization and visual feature extraction on the input video; the Attn-LoRA module improves the visual features through an adaptive correction mechanism to enhance the model's ability to capture personality-related clues; the two-stage training module implements the pre-training and fine-tuning stages in sequence to optimize the model performance; the personality prediction module outputs the scores of the video characters on the five personality dimensions based on the processed features; The interactive interface module provides a convenient video upload channel for users and visualizes the analysis results, realizing the whole process function from video input to personality analysis result generation; the system uses the method of any one of claims 1-6 to analyze the first impression personality of the video characters.

Images