Personality identification method in asynchronous video and program product

Abstract

The invention discloses a personality recognition method in an asynchronous video and a program product, and the method comprises the steps: forming a plurality of face action unit optimization candidate subsets based on a face action unit original set through employing multiple simulated annealing search; calculating multi-class indexes of each face action unit optimization candidate subset, screening out part of subsets from all the face action unit optimization candidate subsets according to the multi-class indexes to form a Pareto front set, and selecting a face action unit optimal subset from the Pareto front set according to a distance minimum principle; extracting a plurality of key frames from the asynchronous video, and extracting each key frame and face action units in a plurality of frames adjacent to the key frame from the optimal subset of the face action units as an identification window sample; inputting all recognition window samples into the model, and performing semantic fusion to generate a merged global semantic representation; and fusing the global semantic representation and the text answer, inputting into the model, and carrying out personality identification. According to the invention, the stability and accuracy of personality identification are improved.

Description

Technical Field

[0001] This invention relates to large language models, and more particularly to a method and program product for personality recognition in asynchronous video. Background Technology

[0002] With the widespread application of Large Language Models (LLMs) in natural language understanding and personality recognition, text-based large model personality recognition methods have become mainstream. However, in asynchronous video interview (AVI) scenarios, a single text modality often fails to fully reflect the nonverbal behavioral characteristics of candidates, thus limiting the accuracy of personality assessment.

[0003] Existing multimodal methods typically rely on full-face image features or sparsely sampled information, but such processing disrupts the temporal continuity of facial movements, making it difficult to capture subtle dynamic changes related to personality. Furthermore, facial expressions in interview scenarios are usually more restrained, resulting in visual features based on overall facial representations suffering from high noise and insufficient effective discriminative power.

[0004] Therefore, how to effectively preserve the local temporal changes of facial movements, reduce interference from irrelevant visual features, and improve recognition accuracy are key issues that urgently need to be addressed in existing technologies. Summary of the Invention

[0005] To address the problems existing in the prior art, the purpose of this invention is to provide a method and program product for personality recognition in asynchronous videos with higher accuracy.

[0006] To achieve the above-mentioned objectives, the present invention provides the following technical solution:

[0007] A method for personality recognition in asynchronous video includes the following steps:

[0008] (1) Extract several facial motion units from the asynchronous video to form the original set of facial motion units;

[0009] (2) A multi-round simulated annealing search is adopted to form multiple candidate subsets of facial action units based on the original set of facial action units; in each round of simulated annealing search, an initial set of facial action units is formed based on the original set of facial action units, and a single round of simulated annealing is performed with the initial set of facial action units as the search basis and the energy value as the comparison benchmark, so as to finally obtain a candidate subset of facial action units.

[0010] (3) Calculate the prediction effectiveness index, operational stability index and size index of each facial action unit optimization candidate subset;

[0011] (4) Based on the above three indicators, select a subset from all candidate subsets of facial motion units to form a Pareto front set, and select the optimal subset of facial motion units from the Pareto front set according to the principle of minimum distance;

[0012] (5) Extract several key frames from the asynchronous video, and extract the facial action units in each key frame and several adjacent frames from the optimal subset of facial action units as a recognition window sample;

[0013] (6) Input all recognition window samples into the pre-trained large language model and perform semantic fusion to generate a merged global semantic representation;

[0014] (7) Integrate the global semantic representation with the text response, input it into the pre-trained large language model, and perform personality recognition.

[0015] Furthermore, step (2) specifically includes the following steps:

[0016] (2.1) Set the number of annealing wheels i=1;

[0017] (2.2) Randomly select one initialization strategy from multiple initialization strategies, and form an initial set of facial action units based on the original set of facial action units according to the selected initialization strategy;

[0018] (2.3) Randomly change the initial set of facial action units to form a candidate subset of facial action units, and perform a single round of simulated annealing to obtain an optimized candidate subset of facial action units; wherein, during the single round of simulated annealing, the energy value calculated based on the LSTM energy function is used as the comparison benchmark to determine whether to accept the candidate subset of facial action units as the current solution;

[0019] (2.4) Set the number of annealing rounds i = i + 1, and determine whether i is less than or equal to the maximum number of annealing rounds. If it is, return to execute (2.2); otherwise, execute (2.5).

[0020] (2.5) Optimize the candidate subset output of all facial motion units.

[0021] Furthermore, the multiple initialization strategies specifically include three initialization strategies, namely:

[0022] First initialization strategy: Use the original set of facial motion units directly as the initial set of facial motion units;

[0023] Second initialization strategy: Randomly select one from the original set of facial motion units as the initial set of facial motion units;

[0024] The third initialization strategy is to randomly select k facial motion units from the original set of facial motion units as the initial set of facial motion units, where k is generated by a random number generator.

[0025] Furthermore, step (2.3) specifically includes the following steps:

[0026] (2.3.1) Use the initial set of facial action units as the initial value of the current solution;

[0027] (2.3.2) Randomly select a facial action unit from the initial set of facial action units, change its selection state, and form a candidate subset of facial action units;

[0028] (2.3.3) The energy value of the candidate subset of facial action units is calculated using the energy function based on LSTM;

[0029] (2.3.4) When the energy value of the candidate subset of facial action units is less than the energy value of the current solution, the candidate subset of facial action units is accepted as the new current solution; otherwise, the candidate subset of facial action units is accepted as the new current solution with exp(-ΔE / T); ΔE represents the difference between the energy values ​​of the candidate subset of facial action units and the current solution, and T represents the temperature.

[0030] (2.3.5) Update the temperature according to the preset temperature decay coefficient and determine whether the termination condition is met; when the termination condition is met, end the single round of simulated annealing process and output the current solution as the candidate subset for facial action unit optimization; otherwise, return to (2.3.2) to continue execution.

[0031] Furthermore, step (3) specifically includes:

[0032] (3.1) Optimize the candidate subset for each facial action unit and calculate its prediction error in personality recognition as its prediction effectiveness index;

[0033] (3.2) For each candidate subset of facial motion unit optimization, the comprehensive similarity is calculated based on the similarity between it and other candidate subsets of facial motion unit optimization, and is used as its operational stability index.

[0034] (3.3) Optimize the candidate subset for each facial action unit and count the number of facial action units it contains as a scale indicator.

[0035] Furthermore, step (4) specifically includes:

[0036] (4.1) Based on the prediction effectiveness index, operational stability index and scale index, construct the dominance relationship in all facial motion unit optimization candidate subsets, and screen out all facial motion unit optimization candidate subsets that are not dominated by other facial motion unit optimization candidate subsets to form the Pareto front set;

[0037] (4.2) Calculate the mean of the three indices of each candidate subset of facial action units in the Pareto front set, select the candidate subset of facial action units with the smallest mean as the ideal point, calculate the distance from each candidate subset of facial action units to the ideal point, and select the candidate subset of facial action units with the smallest distance as the optimal subset of facial action units.

[0038] Furthermore, step (6) specifically includes:

[0039] (6.1) Input each recognition window sample into the pre-trained large language model, and obtain a semantic natural semantic description of the recognition window sample based on the preset first structured instruction template;

[0040] (6.2) Input the natural semantic description of each recognition window sample into the trained large language model, and iteratively fuse it with the natural semantic descriptions of other recognition window samples based on the preset second structured instruction template to generate a global semantic representation based on iterative semantic fusion.

[0041] Furthermore, step (7) specifically includes:

[0042] (7.1) Obtain the global semantic representation and the text-formatted answers of the person to be identified to the assessment questions in the asynchronous video, and perform joint encoding to obtain a high-dimensional semantic vector;

[0043] (7.2) Input the high-dimensional semantic vector into the pre-trained large language model for personality recognition.

[0044] Furthermore, step (7) is followed by the following steps:

[0045] (8) The LoRA fine-tuning technique is used to perform low-rank incremental updates on the weights of the attention layer of the large language model.

[0046] A computer program product includes a computer program that, when executed by a processor, implements the above-described method.

[0047] Compared with the prior art, the beneficial effects of this invention are as follows: by optimizing the extraction of facial action units, this invention effectively preserves the local temporal changes of facial actions, reduces interference from irrelevant visual features, and improves recognition accuracy. In addition, by semantically describing facial action units and effectively integrating them with textual semantic information, the stability and accuracy of personality recognition are further improved. Attached Figure Description

[0048] Figure 1 This is a flowchart illustrating the personality recognition method in asynchronous video provided in an embodiment of the present invention. Detailed Implementation

[0049] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention.

[0050] Example 1

[0051] This invention discloses a method for personality recognition in asynchronous video, such as... Figure 1 As shown, it includes the following steps:

[0052] (1) Extract several facial motion units from the asynchronous video to form the original set of facial motion units.

[0053] Facial Action Units (AUs) enable fine-grained quantification of facial muscle movements, and different AUs exhibit varying effectiveness in personality trait recognition. Specifically, Openface was used to extract AUs from asynchronous interview videos (each frame corresponds to a set of AU data). The Openface tool provides 17 commonly used AUs: 1, 2, 4, 5, 6, 7, 9, 10, 12, 14, 15, 17, 20, 23, 25, 26, and 45. Therefore, these 17 AUs for each frame serve as the original set of facial action units for that frame.

[0054] However, using all AUs directly would introduce redundant information and increase the computational and storage overhead of subsequent processing. Therefore, it is necessary to construct a compact, stable, and discriminative subset of AUs to support subsequent semantic description generation and personality prediction. Considering that personality recognition is a multi-cue, multi-temporal-dependent task, the synergistic effect of multiple AUs often provides richer behavioral signals than a single AU. Static images are difficult to fully reflect such dynamic cues, so this invention evaluates the effectiveness of the AU subset based on a temporal prediction model, specifically including steps (2) to (4).

[0055] (2) A multi-round simulated annealing search is adopted to form multiple candidate subsets of facial action units based on the original set of facial action units.

[0056] Specifically, to reduce the sensitivity of single-round simulated annealing results to random initialization and random search processes, and to achieve a comprehensive balance between prediction effectiveness, operational stability, and AU subset size, this invention performs the following multi-round simulated annealing search, specifically including:

[0057] (2.1) Set the number of annealing wheels i=1;

[0058] Specifically, multiple sets of random seeds and three initialization strategies are preset. The random seeds are used to control the random behavior involved in the simulated annealing process, and the initialization strategies are used to generate the initial AU subset for a single round of simulated annealing.

[0059] (2.2) Randomly select one initialization strategy from multiple initialization strategies, and form an initial set of facial action units based on the original set of facial action units according to the selected initialization strategy;

[0060] In specific implementation, there can be three initialization strategies:

[0061] First initialization strategy: Use the original set of facial motion units directly as the initial set of facial motion units;

[0062] Second initialization strategy: Randomly select one from the original set of facial motion units as the initial set of facial motion units;

[0063] The third initialization strategy is to randomly select k facial motion units from the original set of facial motion units as the initial set of facial motion units, where k is generated by a random number generator.

[0064] (2.3) Randomly change the initial set of facial action units to form a candidate subset of facial action units. Perform a single round of simulated annealing to obtain an optimized candidate subset of facial action units. During the single round of simulated annealing, the energy value calculated based on the LSTM energy function is used as the comparison benchmark to determine whether to accept the candidate subset of facial action units as the current solution.

[0065] (2.4) Set the number of annealing rounds i = i + 1, and determine whether i is less than or equal to the maximum number of annealing rounds n. If it is, return to execute (2.2); otherwise, execute (2.5).

[0066] (2.5) Optimize the candidate subset output of all facial motion units, denoted as:

[0067]

[0068] in, Indicates the first The candidate subset of facial motion units obtained at the end of the simulated annealing run.

[0069] Specifically, (2.3) includes:

[0070] (2.3.1) Use the initial set of facial action units as the initial value of the current solution;

[0071] (2.3.2) Randomly select a facial action unit from the initial set of facial action units, change its selection state, and form a candidate subset of facial action units; the change of selection state includes: setting the originally selected AU to unselected, or setting the originally unselected AU to selected;

[0072] (2.3.3) The energy value of the candidate subset of facial action units is calculated using the energy function based on LSTM;

[0073] The energy function based on LSTM specifically predicts scores for a subset of candidate facial action units and calculates the mean squared error between the predicted results and the true labels on a validation set. This mean squared error is used as the energy value of the candidate subset of facial action units. The energy value characterizes the error level of the candidate subset of facial action units in the personality prediction task; the smaller the energy value, the better the predictive performance of the corresponding candidate subset of facial action units.

[0074] (2.3.4) When the energy value of the candidate subset of facial action units is less than the energy value of the current solution, the candidate subset of facial action units is accepted as the new current solution; otherwise, the candidate subset of facial action units is accepted as the new current solution with exp(-ΔE / T); ΔE represents the difference between the energy values ​​of the candidate subset of facial action units and the current solution, and T represents the temperature.

[0075] (2.3.5) Update the temperature according to the preset temperature decay coefficient and determine whether the termination condition is met; when the termination condition is met, end the single round of simulated annealing process and output the current solution as the candidate subset for facial action unit optimization; otherwise, return to (2.3.2) to continue execution.

[0076] The termination condition includes at least one of the following: reaching a preset iteration number threshold, or the temperature parameter decreasing to a preset minimum threshold.

[0077] Through the above steps, a single round of simulated annealing search is completed, resulting in an optimized candidate subset of facial motion units.

[0078] (3) Calculate the prediction effectiveness index, operational stability index and scale index of each facial action unit optimization candidate subset.

[0079] This step specifically includes:

[0080] (3.1) For each candidate subset of facial action units, calculate its prediction error in personality recognition as its prediction effectiveness index; specifically, predict the personality recognition score of the candidate subset of facial action units, and calculate the mean square error between the prediction result and the real label on the validation set as its prediction effectiveness index.

[0081] (3.2) For each candidate subset of facial motion units, a comprehensive similarity is calculated based on its similarity to other candidate subsets of facial motion units, which serves as its operational stability index; the specific calculation formula is as follows:

[0082]

[0083]

[0084] In the formula, express The overall similarity express The Jaccard similarity score is used to characterize the consistency of the candidate subsets for facial action unit optimization across multiple annealing runs. The lower the score, the higher the similarity between the corresponding set and other sets, i.e., the more stable it is.

[0085] (3.3) Optimize the candidate subset for each facial action unit and count the number of facial action units it contains as a scale indicator.

[0086] (4) Based on the above three indicators, select some subsets from all candidate subsets of facial action units to form the Pareto front set, and select the optimal subset of facial action units from the Pareto front set according to the principle of minimum distance.

[0087] Step (4) specifically includes:

[0088] (4.1) Based on the prediction effectiveness index, operational stability index and scale index, construct the dominance relationship in all facial motion unit optimization candidate subsets, and screen out all facial motion unit optimization candidate subsets that are not dominated by other facial motion unit optimization candidate subsets to form the Pareto front set;

[0089] Specifically, dominance relationships are constructed across all subsets based on three indicators. These dominance relationships are defined as follows:

[0090] , like:

[0091] in: These represent the candidate subsets of facial motion units obtained in the i-th and j-th rounds of annealing, respectively. express The a-th indicator value, where a,b=1,2,3 represent the effectiveness indicator, operational stability indicator, and scale indicator, respectively.

[0092] The above dominance relationship can be expressed as: if and only if The values ​​on all metrics are not inferior to And at least one indicator's value is better than At that time, the judgment Dominate .

[0093] Based on this, all facial action unit optimization candidate subsets that are not dominated by other facial action unit optimization candidate subsets constitute the Pareto front set. .

[0094] (4.2) Calculate the mean of the three indices of each candidate subset of facial action units in the Pareto front set, select the candidate subset of facial action units with the smallest mean as the ideal point, calculate the distance from each candidate subset of facial action units to the ideal point, and select the candidate subset of facial action units with the smallest distance as the optimal subset of facial action units.

[0095] Specifically, for Pareto front sets any subset of AU In its first The original values ​​for each indicator are denoted as follows: .

[0096] To eliminate dimensional differences between different indicators, the multiple indicators of each subset of AU in the Pareto front set are normalized. The normalization method is as follows:

[0097]

[0098] in: Pareto front solution set (the set of all non-dominated solutions). These represent the first, second, and third positions on the Pareto front. The minimum and maximum values ​​of each indicator. The value is the normalized index value.

[0099] After normalization, the ideal point is defined as the point where all normalized indices reach their minimum value, i.e., the origin in the normalized space. Calculation For each facial action unit, the Euclidean distance d(x) from the candidate subset to the ideal point is optimized, and the optimal subset is selected from the Pareto front set according to the principle of minimum distance. :

[0100]

[0101] Will As the final subset of AUs, they are used in the subsequent AU semantic description generation process.

[0102] Through the above steps, the present invention achieves a comprehensive trade-off between prediction performance, operational stability, and AU subset size among multiple candidate AU subsets obtained from multiple simulated annealing runs.

[0103] (5) Extract several key frames from the asynchronous video, and extract the facial action units in each key frame and several adjacent frames from the optimal subset of facial action units as a recognition window sample.

[0104] Keyframes are determined using an automatic detection method based on inter-frame differences. Specifically, pixel-level differences are calculated between adjacent frames in the video. The absolute values ​​of these differences are then aggregated spatially to obtain a time-varying sequence of inter-frame differences. Local maxima are detected within this sequence, and the corresponding frames are automatically identified as keyframes. This method requires no manual intervention and effectively captures moments of most significant facial motion changes, thus focusing on temporal locations where AU (Aspect Ratio) changes are discriminative.

[0105] For each keyframe, the Active User Arguments (AUs) of the three frames before and after it are extracted to form a time window containing seven consecutive frames, which serves as a recognition window sample. This recognition window sample is used to characterize the dynamic change process of the local AUs around the keyframe and serves as the basic unit for subsequent semantic description generation.

[0106] (6) Input all the recognition window samples into the pre-trained large language model and perform semantic fusion to generate the merged global semantic representation.

[0107] The original facial action unit (AU) sequences provide quantitative information on facial muscle movements, but they are low-dimensional numerical data and cannot directly reveal high-level semantic patterns related to personality traits. To bridge the gap between low-level numerical patterns and high-level semantics, this step proposes a two-step semantic transformation method to convert AU sequences into interpretable natural language descriptions, thereby generating rich and easily understood semantic representations that facilitate subsequent multimodal fusion with textual information. Specifically, this includes:

[0108] (6.1) Input each recognition window sample into the pre-trained large language model, and obtain a semantic natural semantic description of the recognition window sample based on the preset first structured instruction template.

[0109] Among them, the first structured instruction template is a structured Prompt template, which is not dynamically written by humans during the operation, but is an instruction template that is predefined and stored in the system during the system design phase and participates in the automated processing flow.

[0110] During the semantic description generation process, the system automatically fills the input placeholder position in the first structured instruction template with the AU time series in the current recognition window sample, and calls the large language model to generate the corresponding semantic description.

[0111] The specific structured components and Chinese definitions of the first structured instruction template are as follows:

[0112] Role Definition: Used to constrain the functional role played by the large language model in the reasoning process, clarifying its role as a facial action analysis expert to provide professional interpretation of AU temporal changes.

[0113] Task definition: Instructs the large language model to generate a concise and coherent natural language description that covers the overall motion and change trend of all selected AUs within the recognition window samples.

[0114] AU Semantic Definition: A pre-constructed set of AU semantic references used to establish the mapping relationship between AU and its corresponding facial action meaning; at least some of the AU semantic definitions are derived from existing standard documents, and the semantic definitions of the remaining AUs are uniformly standardized according to the facial action coding system.

[0115] Output constraint instructions: These are used to restrict the output format of the model, including but not limited to: not including AU numbers, not including numerical strength information, not segmenting the output, and generating continuous natural language paragraphs.

[0116] Example Description: This provides an example correspondence between the AU time series of the input recognition window samples and the expected semantic output; in one implementation, the example is automatically generated by the system based on randomly selected training samples and stored after consistency verification.

[0117] Input data segment: AU time series used to insert the recognition window samples.

[0118] Output placeholder: Used to indicate the semantic description of the corresponding recognition window sample output by the large language model.

[0119] (6.2) Input the natural semantic description of each recognition window sample into the trained large language model, and iteratively fuse it with the natural semantic descriptions of other recognition window samples based on the preset second structured instruction template to generate a global semantic representation based on iterative semantic fusion.

[0120] Since adjacent recognition windows may have similar or repetitive AU variation patterns, directly concatenating the semantic descriptions of multiple recognition window samples can easily introduce redundant information and increase storage and computational burden. Therefore, this invention proposes a global semantic representation generation method based on iterative semantic fusion, which integrates the semantic descriptions of multiple recognition windows into a compact and coherent global semantic representation.

[0121] The second structured instruction template is a structured Prompt used for generating global semantic representation. It is also an instruction template predefined in the system design phase, and the system automatically fills in the natural semantic description of the corresponding window during the operation, without involving manual interaction.

[0122] It should include at least the following components:

[0123] Role Definition: Used to constrain the large language model as a semantic integration auxiliary module to comprehensively analyze the AU change trends across multiple time periods.

[0124] Task definition: Instructs the large language model to combine two natural semantic descriptions into a unified description to approximate the overall AU change trend over a longer time range.

[0125] Output constraint instructions: These are used to restrict the generated results to a single, coherent natural language description, avoiding list-style output or repetitive statements.

[0126] Example Description: Provides natural semantic descriptions of two recognition window samples and their merged example to guide the model in understanding the semantic fusion method; the example is also automatically generated and stored by the system.

[0127] Input data segment: Used to insert two natural semantic description texts to be merged.

[0128] Output placeholder: Used to generate the merged semantic summary.

[0129] The natural semantic descriptions of each recognition window sample are iteratively fused. Specifically, the global semantic description is initialized to be empty. The system sequentially reads the natural semantic descriptions of each recognition window sample: when the global semantic description is empty, the natural semantic description of the current recognition window sample is directly used as the global semantic description; when the global semantic description is not empty, the current global semantic description and the natural semantic description of the next recognition window sample are combined and filled into the second structured instruction template, and the large language model is called to generate a new global semantic description. This process is repeated until all recognition window samples are integrated, ultimately obtaining a global semantic representation for subsequent personality analysis.

[0130] (7) Integrate the global semantic representation with the text response, input it into the pre-trained large language model, and perform personality recognition.

[0131] While Large Language Models (LLMs) excel in semantic understanding and reasoning, they have limitations in representing continuous numerical values. Because word segmentation mechanisms cannot capture fine-grained differences between floating-point numbers, LLMs cannot accurately reflect personality scores when directly used for text-to-text regression tasks. To address this issue, this invention decouples semantic understanding from numerical prediction, enabling modular processing. Specifically, it includes:

[0132] (7.1) Obtain the global semantic representation and the text-formatted answers of the person to be identified to the assessment questions in the asynchronous video, and perform joint encoding to obtain a high-dimensional semantic vector;

[0133] Global semantic representations reflect the behavioral patterns of the subjects' facial action units in the video. Text-formatted responses are the subjects' answers to pre-set questions during video recordings or structured personality assessments, obtained through audio extraction and automatic transcription. Text responses and AU descriptions are aligned at the individual level to provide complementary information on behavior and language.

[0134] LLM, guided by structured prompts, jointly encodes the aforementioned multimodal inputs to generate high-dimensional semantic embeddings. This embedding tightly integrates behavioral patterns with linguistic semantics and anchors the target personality sub-dimension on a psychological basis, ensuring that the semantic representation contains both contextual information and reflects the individual characteristics of the subject.

[0135] (7.2) Input the high-dimensional semantic vector into the pre-trained large language model for personality recognition.

[0136] The regression head of a large language model consists of a single-layer linear transformation:

[0137]

[0138] in High-dimensional semantic vectors generated for LLM, and These are trainable parameters. Output It is a continuous personality score, which can be directly mapped to the original personality quantification scale (e.g., 1-5 points).

[0139] To improve training efficiency and adapt to limited data, the present invention can also perform step (8) after step (7) to perform low-rank incremental updates of the weights of the attention layer of the large language model using LoRA fine-tuning technology.

[0140] For the attention layer weight matrix LoRA introduces two trainable low-rank matrices. their product Used for incremental updates of the original weights. Matrix It is learned automatically through gradient descent during training.

[0141] Parameter definition: and These are the row and column dimensions of the attention layer weight matrix, determined by the LLM architecture. The rank of the low-rank matrix is ​​set manually (a hyperparameter) and satisfies...

[0142] Weight update formula: ;

[0143] in, Keeping the original pre-trained weights unchanged, only through and The training and updates enable model fine-tuning.

[0144] Training strategy: Regression head parameters and LoRA update matrix and Joint training enables end-to-end continuous personality rating prediction. In practice, the target attention layer and the rank of the low-rank matrix can be specified using existing LoRA libraries (such as HuggingFace peft). Automatically generate and train matrices and .

[0145] Example 2

[0146] This invention also provides a computer program product, such as an app on a mobile phone or tablet, or an installer on a computer. This product includes a computer program / instructions that, when executed by a processor, implement the method described in Embodiment 1. The code for the computer-executable program used to perform the operations of this invention can be written in one or more programming languages ​​or a combination thereof. Programming languages ​​include object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as "C" or similar languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0147] It should be understood that the embodiments and descriptions above are only the principles, main features and advantages of the present invention. Various changes and modifications can be made to the present invention without departing from the spirit and scope of the invention, and all such changes and modifications fall within the protection scope of the present invention.

Claims

1. A method for personality recognition in asynchronous video, characterized in that, Includes the following steps: (1) Extract several facial motion units from the asynchronous video to form the original set of facial motion units; (2) A multi-round simulated annealing search is adopted to form multiple candidate subsets of facial action units based on the original set of facial action units; in each round of simulated annealing search, an initial set of facial action units is formed based on the original set of facial action units, and a single round of simulated annealing is performed with the initial set of facial action units as the search basis and the energy value as the comparison benchmark, so as to finally obtain a candidate subset of facial action units. (3) Calculate the prediction effectiveness index, operational stability index and size index of each facial action unit optimization candidate subset; (4) Based on the above three indicators, select a subset from all candidate subsets of facial motion units to form a Pareto front set, and select the optimal subset of facial motion units from the Pareto front set according to the principle of minimum distance; (5) Extract several key frames from the asynchronous video, and extract the facial action units in each key frame and several adjacent frames from the optimal subset of facial action units as a recognition window sample; (6) Input all recognition window samples into the pre-trained large language model and perform semantic fusion to generate a merged global semantic representation; (7) Integrate the global semantic representation with the text response, input it into the pre-trained large language model, and perform personality recognition.

2. The personality recognition method in asynchronous video according to claim 1, characterized in that, Step (2) specifically includes the following steps: (2.1) Set the number of annealing wheels i=1; (2.2) Randomly select one initialization strategy from multiple initialization strategies, and form an initial set of facial action units based on the original set of facial action units according to the selected initialization strategy; (2.3) Randomly change the initial set of facial action units to form a candidate subset of facial action units, and perform a single round of simulated annealing to obtain an optimized candidate subset of facial action units; wherein, during the single round of simulated annealing, the energy value calculated based on the LSTM energy function is used as the comparison benchmark to determine whether to accept the candidate subset of facial action units as the current solution; (2.4) Set the number of annealing rounds i = i + 1, and determine whether i is less than or equal to the maximum number of annealing rounds. If it is, return to execute (2.2); otherwise, execute (2.5). (2.5) Optimize the candidate subset output of all facial motion units.

3. The personality recognition method in asynchronous video according to claim 2, characterized in that, The multiple initialization strategies specifically include three initialization strategies, namely: First initialization strategy: Use the original set of facial motion units directly as the initial set of facial motion units; Second initialization strategy: Randomly select one from the original set of facial motion units as the initial set of facial motion units; The third initialization strategy is to randomly select k facial motion units from the original set of facial motion units as the initial set of facial motion units, where k is generated by a random number generator.

4. The personality recognition method in asynchronous video according to claim 2, characterized in that, Step (2.3) specifically includes the following steps: (2.3.1) Use the initial set of facial action units as the initial value of the current solution; (2.3.2) Randomly select a facial action unit from the initial set of facial action units, change its selection state, and form a candidate subset of facial action units; (2.3.3) The energy value of the candidate subset of facial action units is calculated using the energy function based on LSTM; (2.3.4) When the energy value of the candidate subset of facial action units is less than the energy value of the current solution, the candidate subset of facial action units is accepted as the new current solution; otherwise, the candidate subset of facial action units is accepted as the new current solution with exp(-ΔE / T); ΔE represents the difference between the energy values ​​of the candidate subset of facial action units and the current solution, and T represents the temperature. (2.3.5) Update the temperature according to the preset temperature decay coefficient and determine whether the termination condition is met; when the termination condition is met, end the single round of simulated annealing process and output the current solution as the candidate subset for facial action unit optimization; otherwise, return to (2.3.2) to continue execution.

5. The personality recognition method in asynchronous video according to claim 1, characterized in that, Step (3) specifically includes: (3.1) Optimize the candidate subset for each facial action unit and calculate its prediction error in personality recognition as its prediction effectiveness index; (3.2) For each candidate subset of facial motion unit optimization, the comprehensive similarity is calculated based on the similarity between it and other candidate subsets of facial motion unit optimization, and is used as its operational stability index. (3.3) Optimize the candidate subset for each facial action unit and count the number of facial action units it contains as a scale indicator.

6. The personality recognition method in asynchronous video according to claim 1, characterized in that, Step (4) specifically includes: (4.1) Based on the prediction effectiveness index, operational stability index and scale index, construct the dominance relationship in all facial motion unit optimization candidate subsets, and screen out all facial motion unit optimization candidate subsets that are not dominated by other facial motion unit optimization candidate subsets to form the Pareto front set; (4.2) Calculate the mean of the three indices of each candidate subset of facial action units in the Pareto front set, select the candidate subset of facial action units with the smallest mean as the ideal point, calculate the distance from each candidate subset of facial action units to the ideal point, and select the candidate subset of facial action units with the smallest distance as the optimal subset of facial action units.

7. The personality recognition method in asynchronous video according to claim 1, characterized in that, Step (6) specifically includes: (6.1) Input each recognition window sample into the pre-trained large language model, and obtain a semantic natural semantic description of the recognition window sample based on the preset first structured instruction template; (6.2) Input the natural semantic description of each recognition window sample into the trained large language model, and iteratively fuse it with the natural semantic descriptions of other recognition window samples based on the preset second structured instruction template to generate a global semantic representation based on iterative semantic fusion.

8. The personality recognition method in asynchronous video according to claim 1, characterized in that, Step (7) specifically includes: (7.1) Obtain the global semantic representation and the text-formatted answers of the person to be identified to the assessment questions in the asynchronous video, and perform joint encoding to obtain a high-dimensional semantic vector; (7.2) Input the high-dimensional semantic vector into the pre-trained large language model for personality recognition.

9. The personality recognition method in asynchronous video according to claim 1, characterized in that, Step (7) is followed by the following steps: (8) The LoRA fine-tuning technique is used to perform low-rank incremental updates on the weights of the attention layer of the large language model.

10. A computer program product, comprising a computer program, characterized in that: When the computer program is executed by a processor, it implements the method of any one of claims 1-9.

Images