Personality feature recognition method and system, electronic equipment and storage medium

Abstract

The invention provides a personality feature recognition method and system, electronic equipment and a storage medium, and the method comprises the following steps: obtaining multi-modal data of a target individual, and extracting a feature sequence of each modal, the multi-modal data comprising at least two of three modals of vision, voice and text; mapping the feature sequence of each modal to a unified hidden space to obtain a modal sequence of a unified dimension; performing hierarchical cross-modal alignment processing on the modal sequence with the unified dimension to establish a corresponding relation between different modal features in a time sequence, and obtaining an aligned fusion sequence; carrying out unified coding on the aligned fusion sequence to obtain a coding sequence containing global representation and frame-level representation; and based on the coding sequence, performing adaptive regression processing to generate a personality trait prediction result. And through layered cross-modal alignment and a dual-adaptive regression strategy, the accuracy of personality traits prediction and the stability of model training are significantly improved.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence and multimodal information processing technology, and in particular to a personality feature recognition method, system, electronic device, and storage medium. Background Technology

[0002] Personality computation, a cutting-edge intersection of artificial intelligence and psychology, aims to automatically infer personality traits from observable signals generated during interactions. In recent years, with the widespread adoption of applications such as online interviews, distance education, and intelligent customer service, automatic personality recognition technology has gradually evolved from single-modality to multi-modal fusion. Visual, speech, and text-based trimodals, due to their complementary information, have become the mainstream technological approach, balancing recognition accuracy and generalization ability.

[0003] However, existing technologies still have the following problems in multimodal personality recognition: First, most methods employ "feature concatenation" or "late-stage fusion," which lacks the ability to handle temporal asynchrony and semantic conflicts in multimodal sequences. In real-world scenarios, visual, speech, and text signals often exhibit temporal alignment discrepancies. Direct concatenation or simple fusion introduces noise, reducing the model's ability to model cross-modal consistency relationships.

[0004] Secondly, the variance distribution of personality tag data across the Big Five personality traits (openness, conscientiousness, extraversion, agreeableness, and neuroticism) is uneven, causing high-variance dimensions to dominate the optimization direction during model training, while low-variance dimensions are under-learned, affecting the overall predictive stability.

[0005] Furthermore, key personality cues are sparsely distributed along the timeline. If uniform time pooling or simple average pooling is used, important segments are easily diluted by a large number of irrelevant frames, thereby reducing the model's sensitivity to sparse key events. Summary of the Invention

[0006] In order to overcome the shortcomings of the prior art, the present invention aims to provide a personality feature recognition method, system, electronic device and storage medium that can effectively solve the problems of multimodal temporal asynchrony, label variance imbalance and sparse key clues.

[0007] To achieve the above objectives, the present invention provides a personality feature recognition method, comprising the following steps: Acquire multimodal data of the target individual and extract feature sequences of each modality. The multimodal data includes at least two of the three modalities: visual, speech, and text. The feature sequences of each modality are mapped to a unified latent space to obtain a modality sequence of a unified dimension; The unified-dimensional modal sequence is subjected to hierarchical cross-modal alignment to establish a temporal correspondence between different modal features, resulting in an aligned fusion sequence. The aligned fused sequence is uniformly encoded to obtain an encoded sequence containing global representation and frame-level representation; Based on the encoded sequence, adaptive regression processing is performed to generate personality trait prediction results.

[0008] Furthermore, after the step of mapping the feature sequences of each modality to a unified latent space, the method further includes adding a learnable classification label to the beginning of each modality sequence and adding position encoding to the sequence.

[0009] Furthermore, the step of performing hierarchical cross-modal alignment processing on the unified-dimensional modal sequence to establish a temporal correspondence between different modal features and obtain the aligned fusion sequence further includes: The unified-dimensional modal sequence is input into the hierarchical cross-modal alignment module. Through cross-modal attention calculation with one modal sequence as the query and the other modal sequences as the key and value, the temporal correspondence between the at least two modalities is established, and the aligned fusion sequence is output.

[0010] Furthermore, the adaptive regression processing includes: Temporal saliency weights are generated for different time steps in the encoded sequence, and the frame-level primary personality tendency predictions are weighted and converged based on the weights to obtain the sentence-level personality prediction values. During the training phase, the weights of the corresponding dimensions in the loss function are adjusted based on the variance of each personality dimension label within the training batch.

[0011] Furthermore, after the step of extracting feature sequences of each modality based on the multimodal data, the method further includes: performing quality assessment on the multimodal data and generating a modality mask based on the assessment results; when performing attention calculation in the hierarchical cross-modal alignment process, adjusting the attention weights or masking specific modalities based on the modality mask.

[0012] To achieve the above objectives, the present invention also provides a personality feature recognition system, comprising: a data and feature preparation module, used to acquire multimodal data of a target individual and extract feature sequences of each modality; The feature mapping module is used to map the feature sequences of each modality to a unified latent space to obtain a modality sequence of a unified dimension. The hierarchical cross-modal alignment module is used to perform hierarchical cross-modal alignment processing on the unified dimension modal sequence to establish the correspondence between different modal features in time sequence and obtain the aligned fusion sequence; A unified encoding module is used to uniformly encode the aligned fused sequence to obtain an encoded sequence containing global representation and frame-level representation; The dual adaptive regression module is used to perform adaptive regression processing based on the encoded sequence to generate personality trait prediction results.

[0013] Furthermore, the dual adaptive regression module includes: a temporal saliency weighting and convergence unit, used to generate temporal saliency weights for different time steps in the encoded sequence, and to perform weighted convergence on the frame-level primary personality tendency prediction based on the weights to obtain the sentence-level personality prediction value; and a dimension adaptive loss calculation unit, used to dynamically adjust the weights of the corresponding personality dimensions in the loss function according to the statistical characteristics of each personality dimension label in the training batch during the model training phase.

[0014] Furthermore, it also includes: The data access module is used to receive or collect the multimodal data; The preprocessing module is used to segment and assess the quality of the multimodal data and generate a modal mask; The hierarchical cross-modal alignment module is configured to perform adaptive processing for low-quality or missing modalities based on the modal mask.

[0015] To achieve the above objectives, the present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor is configured to execute the computer program stored in the memory to implement the personality feature recognition method as described above.

[0016] To achieve the above objectives, the present invention also provides a computer-readable storage medium storing a computer program, which is loaded and executed by a processor to implement the personality feature recognition method described above.

[0017] The personality trait recognition method provided by this invention significantly improves the accuracy of personality trait prediction and the stability of model training through hierarchical cross-modal alignment and dual adaptive regression strategies, and enhances robustness to real-world scenarios such as modality loss.

[0018] Other features and advantages of the invention will be set forth in the description which follows, and will be apparent in part from the description, or may be learned by practicing the invention. Attached Figure Description

[0019] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used together with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings: Figure 1 This is a flowchart of a personality feature recognition method according to an embodiment of the present invention; Figure 2 This is a schematic diagram of the personality feature recognition system according to an embodiment of the present invention; Figure 3 This is a schematic diagram of an electronic device structure according to an embodiment of the present invention. Detailed Implementation

[0020] The preferred embodiments of the present invention will be described below with reference to the accompanying drawings. It should be understood that the preferred embodiments described herein are for illustration and explanation only and are not intended to limit the present invention.

[0021] Embodiments of the present invention will now be described in more detail with reference to the accompanying drawings. While some embodiments of the invention are shown in the drawings, it should be understood that the invention can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of the invention. It should be understood that the accompanying drawings and embodiments are for illustrative purposes only and are not intended to limit the scope of protection of the invention.

[0022] The term "comprising" and its variations as used in this invention are open-ended, meaning "including but not limited to". The term "based on" means "at least partially based on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; and the term "some embodiments" means "at least some embodiments".

[0023] It should be noted that the concepts of "first" and "second" may be mentioned in this invention only to distinguish different devices, components or parts, and are not used to limit the order of the functions performed by these devices, components or parts or their interdependence.

[0024] It should be noted that the terms "a" and "a plurality of" used in this invention are illustrative rather than restrictive, and those skilled in the art should understand that, unless explicitly stated otherwise in the context, they should be understood as "one or more". "A plurality of" should be understood as two or more. Definitions of other terms will be given in the description below.

[0025] Example 1 Figure 1 The flowchart below illustrates a personality feature recognition method according to an embodiment of the present invention. Figure 1 The implementation methods of the present invention will be described in detail.

[0026] Step S101: Acquire and preprocess multimodal data.

[0027] In this step, synchronous or asynchronous multimodal data containing the same individual is acquired, including visual data, speech data, and text data, and each modal data is preprocessed. The visual data is a video sequence, the speech data is the corresponding audio waveform, and the text data can be dialogue transcription, self-statement text, or automatic speech recognition (ASR) results corresponding to the audio.

[0028] Preprocessing of each modal data includes: First, video data is sampled at a fixed frame rate (e.g., 25 frames per second) and the image size is adjusted (e.g., 224×224 pixels). Audio data undergoes pre-emphasis, frame segmentation (25ms frame length, 10ms frame shift), and windowing. Text data is segmented or subdivided. Then, pre-trained deep models are used to extract features. For example, the Visual Transformer (ViT) model is used to extract visual feature vectors from video frames, the Wav2Vec 2.0 model is used to extract acoustic feature vectors from speech frames, and BERT (a pre-trained model) or similar pre-trained language models are used to extract contextual feature vectors for text words or subwords. Finally, for a sample, a visual feature sequence is obtained. Speech feature sequence and text feature sequences ,in, , , These represent the time step (number of frames or number of words) for each modality sequence. , , These represent the dimensions of the corresponding features. To facilitate subsequent processing, all sequences are unified to a fixed maximum time step T by padding with zeros or truncating them.

[0029] In step S102: Map the multimodal features to a unified latent space and add location information.

[0030] This step maps feature vectors from different modalities to the same latent space dimension through three independent linear projection layers. : ; in, The weights of the projection layer are trainable parameters, and their function is to align heterogeneous modal features into a common semantic space. These correspond to visual (v), speech (a), and text (t) sequences, respectively. Subsequently, a sinusoidal positional code is superimposed on each sequence, enabling the model to perceive temporal order. Finally, a learnable class token ([CLS]) is inserted at the beginning of each modality sequence to obtain the final input sequence representation. The [CLS] tag will be used to aggregate global information, serving as the starting point for subsequent personality predictions.

[0031] In step S103: Fine-grained alignment is performed through hierarchical cross-modal attention blocks.

[0032] The three sequences obtained in step S102 , , Parallel input consists of N stacked cross-modal attention blocks (XModalBlocks), performing hierarchical cross-modal interaction and alignment. The core operation of each XModalBlock is cross-modal attention computation. For each target modality... (For example, visual) sequences are used as queries, while sequences concatenated from two other modalities (such as speech and text) are used as keys and values: , , ; in, , It is a learnable projection matrix, where [;] represents concatenation along the feature dimension. This process alternately executes queries on the visual, speech, and text modalities, forming a symmetrical interaction structure. This avoids the bias caused by a fixed master-slave modality and ensures that information from each modality is equally and fully complemented and integrated. In this way, each modality can actively "query" and integrate information from the other two modalities, thereby achieving complementarity and alignment between modalities at the feature level, effectively correcting temporal asynchrony and semantic bias caused by acquisition or transcription.

[0033] In this embodiment, the cross-modal attention output is calculated as follows: ; Each XModalBlock includes a feedforward network (FFN), layer normalization (LayerNorm), and residual connections after cross-modal attention to stabilize training and enhance representation capabilities. After N layers are stacked, the output consists of three fully interactive and aligned sequence representations. .

[0034] in, and This represents the concatenation of the other two modes in the feature dimension (e.g.) , , Scaling factor This is used to suppress the variance amplification caused by the increase of the inner product as the dimension increases, making the gradient of softmax more stable. Intuitively, Go "ask" its modality ,through After scaling and normalization using the softmax function, the weights are obtained, and then... Weighted summation yields the aligned representation ,right By executing in turn, you can get This approach avoids bias in a single "dominant modality" and achieves complementary fusion and temporal synchronization of the three modalities at the token level (using video frames, audio frames, and text words / sub-words as basic units, and implementing fine-grained, time-synchronized semantic alignment and fusion under a cross-modal attention mechanism), providing clean and consistent input for subsequent joint modeling of the unified encoder.

[0035] In step S104: The aligned sequences are modeled in a unified manner.

[0036] The three aligned sequences output from step S103 are concatenated along the feature dimension to form a fused sequence. . fusion sequence The input is fed into a shared encoder module consisting of M layers of standard Transformer encoders. This encoder employs a multi-head self-attention mechanism and a feedforward network to perform unified high-order joint modeling of all modal information, capturing global dependencies across modalities. The parameters of this shared encoder are shared across all time steps and all modal information, which helps learn consistent patterns across modalities and improves generalization ability. The final output of the encoder is... We will use the vector corresponding to the initial [CLS] marker position. Extracted as a global representation of the entire input segment; simultaneously, frame-level vectors at all time steps are retained. (i.e., retain timestamp index).

[0037] Step S105: Personality trait regression prediction based on dual adaptive strategy.

[0038] This step includes two core adaptive mechanisms: time significance weighting and dimension-adaptive loss weighting.

[0039] Temporal saliency weighting: To highlight key time segments, the model learns a temporal saliency weight distribution. Specifically, for each time step t, the frame-level vector... A learnable saliency score is computed using a lightweight attention unit (such as a single-layer feedforward network). Then, the scores of all valid time steps are Softmax normalized to obtain the normalized time weights. : ; In this embodiment, the following is satisfied: ∈(0,1) and , where represents the relative contribution of frame t to the final personality prediction.

[0040] At the same time, for the vector at each time step By applying a shared regression head (linear layer), a preliminary prediction of the five-dimensional personality at that time step is obtained. : ; Final Sentence-Level Five-Dimensional Personality Prediction Score : ; Through the aforementioned temporal saliency learning, highly saliency frames receive greater weight in the weighted convergence, while the influence of redundant or noisy frames is suppressed. This enhances the model's overall sensitivity to sparse key cues and improves predictive stability. The temporal saliency weighting mechanism enables the model to automatically focus on key behavioral segments expressing personality traits (such as specific facial expressions, tone of voice, or wording), suppressing the influence of irrelevant or redundant segments. Dimension-Adaptive Weighted Loss: To balance the differences in learning difficulty across the Big Five personality traits caused by varying label variances, a dimension-adaptive weighted mean squared error (WMSE) loss is used during the training phase. For the current training batch, the variance of the true label for each dimension j (out of 5 dimensions) is calculated. The loss function is defined as: ; in, and These are the true value and the predicted value of the j-th dimension, respectively. Corresponding to the five dimensions of the Big Five personality traits, there are This represents the true personality score vector. This is the model's prediction vector. The adaptive weight for the j-th dimension is inversely scaled according to the label variance of that dimension in the mini-batch. This represents the variance of the j-th dimension label in the current mini-batch, used as a measure of "learning difficulty" (the larger the variance, the smaller the weight, to suppress gradient dominance in that dimension). For numerical stability constants (e.g., 10) -8 ),avoid When the variance is very small, the denominator approaches zero. This design suppresses high-variance dimensions and moderately amplifies low-variance dimensions, thereby balancing the overall learning difficulty of the five dimensions and achieving stable convergence.

[0041] variance This reflects the degree of fluctuation of the label in the current mini-batch. By assigning greater weights to dimensions with smaller variance (relatively stable labels) and less weights to dimensions with larger variance (potentially noisier labels), this loss function effectively balances the gradient contributions of each dimension, guiding the model to converge more stably. The model minimizes [the following text is incomplete and requires further context:] during training... Perform end-to-end optimization for the target.

[0042] In the embodiments of the present invention, temporal saliency weighting and dimensional adaptive loss weighting together constitute a dual adaptive strategy. The former focuses on key information fragments from the time dimension, while the latter balances the learning difficulty of different personality traits from the feature dimension. The two work together to improve the model’s sensitivity to sparse key clues and the stability of overall training.

[0043] Step S106: Output the personality trait recognition results.

[0044] The prediction vector obtained in step S105 Post-processing (e.g., scaling or shifting based on the training data distribution, mapping to commonly used scale ranges) is performed to output the final continuous Big Five personality trait scores, including Openness (O), Conscientiousness (C), Extraversion (E), Agreeableness (A), and Neuroticism (N). Simultaneously, auxiliary information, such as time-significant weight distribution, can be output. This is the key basis for explaining the model's decision-making.

[0045] Furthermore, to handle modality loss situations that may occur in practical applications (such as only audio and text), this invention introduces a modality loss masking mechanism after feature projection in step S102. When a modality is completely missing, its corresponding input sequence is masked. All vectors are set to zero, and in the cross-modal attention calculation of step S103, the key-value pairs corresponding to that modality are removed from... and By excluding certain modalities, attention is focused only on those that are present. This design ensures the system's robustness and usability even when some modalities are missing.

[0046] In this embodiment, to verify the effectiveness of the method of the present invention, a comparative experiment was conducted on the publicly available Chinese Multimodal Personality Dataset (MDPE). This dataset contains synchronized video, audio, and text data of 193 subjects, along with Big Five personality traits labels.

[0047] Experimental setup: The baseline method uses concatenated trimodal features, processed by a shared Transformer encoder, and employs average pooling and mean squared error (MSE) loss for prediction. This invention, based on the same backbone network and features, introduces hierarchical cross-modal alignment (XModalBlock) and dual adaptive regression heads.

[0048] Evaluation metrics: The root mean square error (RMSE) of each dimension and the five-dimensional mean (AVG-RMSE) were used for evaluation. The results are shown in Table 1 below: Table 1

[0049] As shown in Table 1, the method of this invention achieves lower prediction errors across all five personality dimensions. The performance improvement is particularly significant in the conscientiousness (C), extraversion (E), and neuroticism (N) dimensions, with relative error reductions of approximately 8.1%, 8.0%, and 10.7%, respectively. The average error across the five dimensions decreased from 0.184 to 0.173, a relative reduction of approximately 6.0%. These experimental results demonstrate that the hierarchical cross-modal alignment and dual adaptive regression strategy proposed in this invention can effectively improve the overall accuracy of multimodal personality recognition.

[0050] Example 2 This embodiment provides a personality feature recognition system based on hierarchical cross-modal alignment and adaptive regression, used to implement the steps described in Embodiment 1. This system can be deployed on cloud servers, edge computing devices, or local computing platforms for scenarios such as online interview assessments, remote psychological counseling, and intelligent customer service quality inspection.

[0051] Figure 2 This is a schematic diagram of the personality feature recognition system according to an embodiment of the present invention. The following will be combined with... Figure 2 The system of this embodiment will be described in detail.

[0052] like Figure 2 As shown, the personality feature recognition system based on hierarchical cross-modal alignment and adaptive regression in this embodiment includes a data and feature preparation module 201, a feature mapping module 202, a hierarchical cross-modal alignment module 203, a unified encoding module 204, a dual adaptive regression prediction module 205, and an output and interpretation module 206. Each module will be described in detail below.

[0053] The data and feature preparation module 201 is used to receive or acquire the user's visual (video), speech (audio), and text data. This module includes a video capture unit, an audio capture unit, and a text input interface, and is responsible for the initial time-stamp alignment and buffering of the data. It also includes a feature extraction subunit, which embeds or calls pre-trained visual models (such as ViT), speech models (such as Wav2Vec 2.0), and text models (such as BERT) to extract high-level feature sequences from the raw data.

[0054] The feature mapping module 202, connected to the data and feature preparation module 201, receives the extracted multimodal feature sequences. This module contains three independent linear projection layers to map features of different dimensions to a unified latent space. It also includes a positional encoding addition unit and a [CLS] label insertion unit, responsible for adding positional information to each sequence and adding a global classification label to the header, forming a standard Transformer input format. This module is also responsible for generating two types of masks: a temporal length mask (indicating the difference between the actual data and padding in the sequence) and a modality availability mask (indicating which modalities are missing in the input). These masks are passed to downstream modules to support dynamic computation and robust handling of missing modalities.

[0055] The hierarchical cross-modal alignment module 203, connected to the feature mapping module 202, has the core function of establishing soft alignment relationships between modalities at the frame or word level through a learnable cross-modal attention mechanism. This corrects temporal asynchrony and semantic conflicts caused by acquisition or transcription at the source before entering deep joint modeling. This module consists of multiple cross-modal attention blocks (XModalBlocks) connected in series. Each block contains a query, key-value projection layer, scaled dot product attention computation unit, feedforward network, and residual connections and layer normalization components. The core function of this module is to perform bidirectional or tridirectional attention interactions between modalities. Using a sequence from one modality as a query, it dynamically retrieves and fuses relevant information from sequences from other modalities, thereby achieving fine-grained temporal and semantic alignment at the feature level.

[0056] A unified encoding module 204, connected to a hierarchical cross-modal alignment module 203, receives the aligned fused sequence. This module is a multi-layered standard Transformer encoder whose parameters are shared across all modalities and time steps. It performs deep modeling of the fused cross-modal information through a self-attention mechanism, extracting global contextual features and high-order dependencies that imply personality traits. Its output is a global [CLS] vector and a series of frame-level context vectors.

[0057] The dual adaptive regression prediction module 205 is connected to the unified coding module 204. This module further comprises two sub-units: The temporal saliency weighting and convergence unit contains a lightweight saliency scoring network and a Softmax normalization layer for generating weights at each time step. It also includes a shared frame-level regression head (linear layer). Its function is to compute weighted initial values ​​for fragment-level personality predictions.

[0058] The dimension-adaptive loss calculation unit (mainly used in the training phase) calculates the label variance of each personality dimension in the training batch in real time, and dynamically generates loss weights based on the inverse of the variance to construct the weighted mean squared error loss and guide the model parameter update.

[0059] The dual adaptive regression prediction module 205 operates in two modes: system training and inference. In training mode, the two sub-units work together to calculate weighted loss to optimize model parameters. In inference mode, only the temporal significance weighted and convergence unit is used to generate the final prediction result, while the dimension adaptive loss calculation unit does not participate in the work.

[0060] The output and interpretation module 206 is connected to the dual adaptive regression prediction module 205. This module post-processes and formats the regression prediction results, outputting the final Big Five personality dimension scores. Furthermore, this module visualizes or structures the temporal significance weight distribution, providing users with explanations of key time points in the model's decisions, thus enhancing the system's interpretability and credibility.

[0061] Preferably, the system also includes a modality missing handling module. The system integrates modality missing handling logic, typically in the form of software logic integrated into modules such as feature mapping and cross-modal alignment. When a modality input is detected to be missing or of extremely low quality, this module triggers a preset masking mechanism to block the contribution of that modality in the computation graph, guiding the system to automatically degrade to a bimodal or unimodal inference mode, thereby ensuring service continuity and robustness.

[0062] Furthermore, this system also includes an independent training and management platform (for system development and iteration) for managing datasets (using session-level or subject-level isolation partitioning), configuring model hyperparameters (such as the number of cross-modal attention block layers N and the number of unified encoder layers M), performing end-to-end model training (using the aforementioned weighted mean squared error WMSE as the loss function), managing model checkpoints, and recording detailed training logs (including loss curves, dynamic changes in the weights of each dimension, etc.) to support iterative development and performance reproduction of the model.

[0063] Example 3 This embodiment, from the perspective of engineering implementation and system deployment, elaborates in detail the specific solution for transforming the personality feature recognition method described in Embodiment 1 of the present invention into a stable, reliable, and maintainable software system. This embodiment focuses on the engineering details of hardware selection, service-oriented architecture design, end-to-end data processing flow, and operation and maintenance support mechanisms, representing the practical application of the aforementioned method embodiment 1 and system embodiment 2 in a specific production environment. This embodiment mainly includes three parts: Part One: Hardware and Basic Software Platform. Construct a hardware platform consisting of at least one computing node. A typical single-node configuration is as follows: Computing Unit: Equipped with a general-purpose multi-core central processing unit (CPU) and at least one graphics processing unit (GPU) that supports tensor operations for accelerating model training and inference.

[0064] Memory and Storage: The system has a minimum of 64GB of memory to store multimodal sequence data and intermediate activation values ​​from the model. The system disk uses a non-volatile memory (NVMe) solid-state drive to ensure efficient random access to feature cache and training data.

[0065] Acquisition equipment: Configure high-definition cameras, directional microphones, or microphone arrays according to the application scenario. To ensure multimodal data synchronization, an external audio / video acquisition card can be connected, and it is recommended to synchronize all acquisition and computing devices using Network Time Protocol (NTP) or Precision Time Protocol (PTP) to ensure that cross-modal timestamps are aligned at the millisecond level.

[0066] Network: Configure gigabit or 10-gigabit Ethernet switches to meet the high throughput requirements of multi-channel concurrent data acquisition, communication between model services, and online inference requests.

[0067] The basic software platform uses the mainstream Linux operating system and installs the corresponding GPU drivers and general computing acceleration libraries (such as CUDA and cuDNN). The upper-layer applications are deployed in a containerized microservice architecture and managed through container orchestration tools (such as Kubernetes) to achieve resource isolation, elastic scaling, and convenient operation and maintenance.

[0068] Part Two concerns the software service architecture and module functionality. The entire system is decomposed into a series of loosely coupled microservices, each corresponding to... Figure 2 The system architecture shown includes one or more functional modules that communicate with each other via a lightweight message bus or Remote Procedure Call (RPC). Core services include: Data Access Service: Responsible for receiving offline audio and video files or online media streams. Supports subscribing to streaming media using standard protocols (such as RTP / RTSP) from conferencing systems and call centers. Attaches a globally unique session ID and a precise timestamp to each data entry.

[0069] Preprocessing and segmentation service: Segments continuous media streams based on semantic pauses, silence detection, or fixed durations, generating sample triples containing video frame sequences, audio frame sequences, and text sequences (from transcription or input). Built-in quality assessment modules include signal-to-noise ratio estimation, loudness normalization, and image brightness detection, and low-confidence segments are marked.

[0070] Feature extraction service: Deploy pre-trained models for vision (e.g., ViT), speech (e.g., Wav2Vec2), and text (e.g., domain-adapted BERT) to extract features from segmented samples. Output visual feature sequences, speech feature sequences, and text feature sequences.

[0071] Representation Alignment and Encoding Service: This service is the core computational service, integrating steps S102 to S104 of Example 1. It receives trimodal features, performs linear projection, adds positional encoding and [CLS] labeling, hierarchical cross-modal attention alignment (XModalBlock), and unified modeling using a shared Transformer encoder. This service supports dynamically loading model configurations with different numbers of layers (N, M) and enables mixed-precision computation and key-value caching to optimize inference performance.

[0072] Regression Prediction and Convergence Service: Receives global and frame-level vectors output by the encoding service, performs frame-level regression, time significance weight calculation and weighted convergence (corresponding to step S105 in Example 1), and generates the final five-dimensional personality score.

[0073] Training Management Service: Responsible for offline training tasks. Manages datasets (divided by session or subject to avoid information leakage), hyperparameter configuration, loss calculation (including dimensionally adaptive weighted WMSE), model checkpoint saving, and training log recording.

[0074] Inference Gateway Service: Provides a unified application programming interface (API), such as a RESTful API or gRPC interface, to receive identification requests and return structured results (personality scores, saliency distributions, quality metrics, etc.).

[0075] Monitoring and logging services: Collect performance metrics (GPU utilization, memory usage, processing latency, throughput), business metrics (identification confidence level), and abnormal events of all services across the entire chain, and provide real-time dashboards and alarm functions.

[0076] Part Three: End-to-End Data Processing Flow.

[0077] refer to Figure 1 The steps for data flow and processing between microservices are as follows: Step 1: Multimodal Input and Initial Quality Screening. The data access service acquires synchronized video, audio, and text streams. The preprocessing service performs segmentation and quality assessment, marking poor-quality segments (such as long periods of silence, severe clipping, or excessively dark images) as "low confidence" and generating samples of uniform length.

[0078] Step 2: Feature Shaping and Mask Generation. The feature extraction service extracts high-level features for each modality. These are then linearly projected to a unified dimension and layer normalization is applied. Two types of masks are generated simultaneously: a temporal length mask (identifying valid time steps) and a modality availability mask (identifying which modalities are missing).

[0079] Step 3: Construct a unified input sequence. A learnable [CLS] marker is inserted at the beginning of the sequence, and sinusoidal positional codes are superimposed for all time steps. For very long sequences, a pruning strategy that prioritizes preserving semantic boundaries is adopted; for short sequences, padding is performed.

[0080] Step 4: Cross-Attention Alignment. In the alignment and encoding service, cross-modal attention calculation is performed using XModalBlock. This process reads the mask and quality markers generated in Step 2: during low-confidence periods or at locations corresponding to missing modalities, their weight in attention calculation is dynamically reduced; for asynchronous speech-text scenarios, the attention mechanism allows for flexible alignment, either "one-to-many" or "many-to-one".

[0081] Step 5: Unified Encoding Modeling. Input the aligned and fused sequence into a shared M-layer Transformer encoder. To handle long sequences, a length bucketing strategy is enabled, grouping samples of similar length to reduce padding overhead.

[0082] Step Six: Frame-Level Preliminary Regression. For the hidden state of each valid time step output by the encoder, a lightweight shared regression head (linear layer) is used to map the value of the five-dimensional personality tendency at that moment to obtain the preliminary prediction value.

[0083] Step 7: Time-Weighted Convergence. The saliency weights for each time step are learned and calculated. The frame-level predictions are then weighted and summed to obtain the final segment-level prediction. To prevent excessive weight concentration in a few frames, temperature parameters and minimum entropy constraints are applied to the saliency distribution, balancing "focusing on key elements" with "overall stability." When the effective duration of samples is extremely short or the weight learning is unstable, the system automatically reverts to robust convergence methods such as mean pooling.

[0084] Step 8: Dimensional Adaptive Loss Calculation (Training Branch). Activated only during the training phase. The variance of each personality dimension label within the current training batch is calculated periodically, and the dimensional weights for the WMSE loss are dynamically generated accordingly. To avoid inaccurate estimations in the early stages of training, the weights are updated using a moving average, and upper and lower limits are set for pruning.

[0085] Step Nine: Results Output and Additional Context. The regression prediction service outputs the final Big Five personality score (O, C, E, A, N). Simultaneously, the output includes structured metadata such as timestamp, session ID, processing time, a summary of saliency weight distribution, quality markers for each modality, and confidence score, for result traceability, interpretation, and auditing.

[0086] Preferably, integrated management of training and inference is provided, including: sharing the same data preprocessing, feature extraction, and sequence construction code between the training and inference pipelines to ensure consistent input distribution; using a fixed random seed and deterministic operators during the training phase to ensure reproducible experimental results; and supporting request queues and load balancing in the inference gateway. By horizontally scaling the number of instance replicas of the "representation alignment and encoding service" and the "regression prediction and convergence service," system throughput can be linearly improved; hot updates and rolling upgrades of models are supported, and online request processing is not affected during service restarts.

[0087] Preferably, system operation and maintenance, monitoring, and quality assurance are provided, including: Configuration-based management: All core hyperparameters (such as maximum sequence length, number of encoder layers, loss weight smoothing coefficient, and convergence temperature parameter) are managed through a centralized configuration file, and changes require auditing and gray-scale verification.

[0088] Comprehensive monitoring: The monitoring service tracks hardware resource utilization, service throughput, end-to-end latency, error rate, and business metrics (such as the distribution of predicted values ​​for each dimension) in real time. Threshold alerts can be set, and automatic scaling strategies can be implemented.

[0089] Anomaly and Degradation Handling: The system possesses a complete degradation handling chain. For example, when the text modality is completely missing, the modality availability mask will block the text branch, and the model will automatically degenerate into a visual-speech bimodal model; when the video quality is extremely poor for a certain period of time, its temporal saliency weight will be significantly suppressed. The system always provides the most reliable inference possible based on available modalities and information quality, and reflects uncertainty in the output metadata.

[0090] Security and Compliance: Data transmission and storage are encrypted. Access logs and identification result logs are stored separately. Personally identifiable information (PII) that may be included in the results is anonymized before output to meet data privacy regulations.

[0091] Through the engineering deployment scheme described in this embodiment, the present invention is transformed from an algorithm model into an industrial-grade system with high availability, high robustness, scalability and easy operation and maintenance, which can effectively support the large-scale, high-concurrency application needs of real business scenarios such as online interviews and remote assessments.

[0092] Example 4 In embodiments of the present invention, an electronic device is also provided. Figure 3 This is a schematic diagram of the structure of an electronic device according to an embodiment of the present invention, such as... Figure 3 As shown, the electronic device of the present invention includes a processor 301 and a memory 302, wherein, The memory 302 stores a computer program, which, when read and executed by the processor 301, performs the steps described above in the embodiment of the personality feature recognition method.

[0093] Example 5 In embodiments of the present invention, a computer-readable storage medium is also provided, wherein a computer program is stored in the computer program, wherein the computer program is configured to execute the steps in the embodiments of the personality feature recognition method described above when running.

[0094] In this embodiment, the aforementioned computer-readable storage medium may include, but is not limited to, various media capable of storing computer programs, such as USB flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.

[0095] Those skilled in the art will understand that the above description is merely a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. For example, the hierarchical cross-modal alignment module can also be implemented using a soft alignment mechanism based on Dynamic Time Warping (DTW) or Connection Temporal Classification (CTC); the encoder in the unified modeling module can also be replaced by a network structure capable of sequence modeling, such as a Conformer, a long sequence Transformer (e.g., Performer), or a recurrent neural network (e.g., LSTM); the classification label ([CLS]) can also be replaced by a global vector generated by global average pooling or weighted pooling; the temporal saliency weighting can also be equivalent to the normalized temporal weights generated by attention pooling or gated convolution; the weighted loss used to balance dimensions can also be based on a weighting method based on learnable uncertainty; furthermore, the method and system described in the present invention are also applicable to end-to-end scenarios of learning features from raw data. Therefore, any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A personality trait recognition method characterized by comprising: The method comprises the following steps: obtaining multi-modal data of a target individual, and extracting feature sequences of each modality, wherein the multi-modal data comprises at least two of visual, speech and text modalities; mapping the feature sequences of each modality to a unified hidden space to obtain modality sequences of a unified dimension; performing hierarchical cross-modality alignment processing on the modality sequences of the unified dimension to establish a correspondence between different modality features in time sequence, and obtaining a fused sequence after alignment; performing unified encoding on the fused sequence after alignment to obtain an encoded sequence containing global representation and frame-level representation; based on the encoded sequence, performing adaptive regression processing to generate a personality trait prediction result.

2. The personality trait recognition method of claim 1, wherein, After the step of mapping the feature sequences of each modality to a unified hidden space, a step of adding a learnable classification mark at the beginning of each modality sequence and adding position encoding to the sequence is further included.

3. The personality trait recognition method of claim 1, wherein, The step of performing hierarchical cross-modality alignment processing on the modality sequences of the unified dimension to establish a correspondence between different modality features in time sequence, and obtaining a fused sequence after alignment, further comprises: inputting the modality sequences of the unified dimension into a hierarchical cross-modality alignment module, establishing a time sequence correspondence between the at least two modalities through cross-modality attention calculation taking one modality sequence as a query and the remaining modality sequences as keys and values, and outputting a fused sequence after alignment.

4. The personality trait recognition method of claim 1, wherein, The adaptive regression processing comprises: generating time saliency weights for different time steps in the encoded sequence, and performing weighted aggregation on frame-level personality tendency primary predictions based on the weights to obtain a sentence segment-level personality prediction value; in the training phase, adjusting the weight of the corresponding dimension in the loss function based on the variance of each personality dimension label within the training batch.

5. The personality trait recognition method of claim 1, wherein, After the step of extracting feature sequences of each modality based on the multi-modal data, a step of performing quality assessment on the multi-modal data and generating a modality mask according to the assessment result is further included; when the hierarchical cross-modality alignment processing performs attention calculation, the attention weight is adjusted or the specific modality is shielded according to the modality mask.

6. A personality trait recognition system characterized by, The method comprises the following steps: a data and feature preparation module for obtaining multi-modal data of a target individual, and extracting feature sequences of each modality; a feature mapping module for mapping the feature sequences of each modality to a unified hidden space to obtain modality sequences of a unified dimension; a hierarchical cross-modality alignment module for performing hierarchical cross-modality alignment processing on the modality sequences of the unified dimension to establish a correspondence between different modality features in time sequence, and obtaining a fused sequence after alignment; a unified encoding module for performing unified encoding on the fused sequence after alignment to obtain an encoded sequence containing global representation and frame-level representation; a double adaptive regression module for performing adaptive regression processing based on the encoded sequence to generate a personality trait prediction result.

7. The personality trait recognition system according to claim 6, characterized in that, The dual adaptive regression module comprises: a time significance weighting and aggregation unit, configured to generate time significance weights for different time steps in the encoding sequence, and to weight and aggregate frame-level personality tendency primary predictions based on the weights to obtain a sentence segment-level personality prediction value; and a dimension adaptive loss calculation unit, configured to dynamically adjust the weight of a corresponding personality dimension in a loss function according to the statistical characteristics of the personality dimension labels in a training batch in a model training stage.

8. The personality trait recognition system according to claim 6 or 7, characterized in that, Further comprising: a data access module configured to receive or collect the multi-modal data; a preprocessing module configured to perform segmentation and quality assessment on the multi-modal data, and to generate a modal mask; wherein the hierarchical cross-modal alignment module is configured to perform adaptive processing for low-quality or missing modalities according to the modal mask.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, The processor is configured to execute a computer program stored in the memory to implement the personality characteristic recognition method according to any one of claims 1 to 5.

10. A computer-readable storage medium, characterized in that, The storage medium stores a computer program, and the processor loads and executes the computer program to implement the personality characteristic recognition method according to any one of claims 1 to 5.

Images