Large five-personality prediction method based on quantum multi-modal fusion and space-time diagram network

Abstract

The invention belongs to the technical field of personality prediction, and relates to a large five personality prediction method based on quantum multi-modal fusion and a space-time diagram network. The method comprises the following steps: acquiring an image video, and extracting an image sequence, an audio signal and a text after audio transcription from the image video; extracting facial expression data, limb movement data, voice signal data and text semantic data; extracting feature vectors of time-space relevance from the data respectively; establishing a cross-modal dynamic mapping relationship between the feature vectors, and generating a weight of each piece of data; optimizing the weight of each piece of data based on a quantum optimization algorithm, and establishing a big five-personality trait prediction model according to the optimized weight of each piece of data; collecting data in real time and inputting the data into a big five-personality trait prediction model; and outputting a five-dimensional personality traits vector, and generating each data contribution degree thermodynamic diagram and anti-fact explanation. According to the method, efficient multi-modal feature fusion and accurate spatial-temporal feature modeling are realized, and the method has good interpretability.

Description

Technical Field

[0001] The present invention belongs to the technical field of personality prediction, and relates to a Big Five personality prediction method based on quantum multimodal fusion and spatio-temporal graph network. Background Art

[0002] Personality prediction technology has important application value in the cross-field of psychology and artificial intelligence. Among them, the Big Five personality model is widely used in fields such as behavior analysis, talent assessment, and mental health monitoring due to its universality. Traditional personality assessment mainly relies on subjective questionnaire surveys, which have inherent defects such as low efficiency and susceptibility to social desirability bias. With the development of computer vision, speech processing, and natural language processing technologies, personality prediction methods based on multimodal data have gradually become a research hotspot.

[0003] Existing technologies usually extract features from multi-source data such as videos, voices, and texts, and realize personality trait prediction through machine learning models. However, the current methods still have significant deficiencies in multimodal feature fusion, spatio-temporal feature modeling, and computational efficiency. In terms of multimodal fusion, traditional weighted averaging or simple attention mechanisms are difficult to dynamically adapt to the different demands of each personality dimension for modal features, resulting in limited model prediction accuracy. In terms of temporal feature modeling, existing methods based on RNN or LSTM are difficult to effectively capture long sequence dependencies, while directly applying the Transformer architecture faces the problem of high computational complexity. In terms of spatial feature extraction, unstructured data such as body language is usually modeled with a fixed graph structure, and it is impossible to adaptively learn the dynamic interaction relationships between joint points. In addition, existing methods generally lack interpretability analysis of prediction results and are difficult to meet the requirements of model transparency in practical application scenarios such as psychological counseling and talent selection.

[0004] Therefore, there is an urgent need to develop a new personality prediction method that can achieve multimodal feature fusion, spatio-temporal feature modeling, and interpretability. Summary of the Invention

[0005] To solve the problems in the existing technology, the present invention provides a Big Five personality prediction method based on quantum multimodal fusion and spatio-temporal graph network, which realizes efficient multimodal feature fusion, accurate spatio-temporal feature modeling, and good interpretability.

[0006] To achieve the above object, the present invention adopts the following technical solutions: In the first aspect, the present invention provides a Big Five personality prediction method based on quantum multimodal fusion and spatio-temporal graph network, including the following steps: Obtain image videos, and extract image sequences, audio signals, and texts after audio transcription from the image videos; Extract image data and first text semantic data based on the image sequence; extract speech signal data based on the audio signal; extract second text semantic data based on the text after audio transcription and the image sequence; Extract feature vectors with spatio-temporal correlation from each of the above data respectively; Establish a cross-modal dynamic mapping relationship between the feature vectors and generate weights for each data; Optimize the weights of each data based on the quantum optimization algorithm, and establish a Big Five personality trait prediction model according to the optimized weights of each data; Collect image data, speech signal data and text semantic data in real time, and input them into the Big Five personality trait prediction model; output a five-dimensional personality trait vector, and generate a heat map of the contribution degree of each data and a counterfactual explanation.

[0007] Preferably, before the step of extracting feature vectors with spatio-temporal correlation from each of the above data respectively, preprocess each data.

[0008] Preferably, the specific method for establishing a cross-modal dynamic mapping relationship between the feature vectors and generating weights for each data is as follows: Randomly initialize the cross-modal mapping matrix , and ; where represents the direct mapping matrix from the image feature vector to the text semantic feature vector; represents the direct mapping matrix from the image feature vector to the speech signal feature vector; represents the direct mapping matrix from the text semantic feature vector to the speech signal feature vector; Convert each feature vector through the mapping matrix, and the calculation formula for the mapped feature vector is:

[0009] where, represents the image feature vector; represents the text semantic feature vector; represents the speech signal feature vector; represents the mapped image feature vector; represents the mapped text semantic feature vector; represents the mapped speech signal feature vector; The calculation formula for the similarity matrix between different modal feature vectors is:

[0010] where S is the similarity matrix; The calculation formula for the importance score of each data is:

[0011] Among them, represents the importance score of the th data; represents the gradient of the th data loss function; is the balance parameter of the similarity matrix; is the gradient balance parameter of the loss function; The calculation formula for the initial weight of each data is:

[0012] Among them, is the initial weight value of the th data in the weight matrix; Generate an initial weight matrix according to the initial weight values of each data, and the calculation formula for iterative optimization of the weight matrix is:

[0013] Among them, is the weight matrix after iterations; is the activation function; is the trainable parameter matrix; is the feature vector after iterations; is the bias matrix; Based on generate the weights of each data.

[0014] Preferably, the image data includes facial expression data and limb movement data; the specific method for extracting image data based on the image sequence is: MTCNN detects the face region in the image sequence; based on 3DDFA and according to the face region, a three-dimensional facial model is reconstructed; ResNet50 extracts the facial expression data in the three-dimensional facial model; OpenPose extracts the coordinates of 18 limb joint points in the image sequence, calculates the velocity and acceleration features of each limb joint point coordinate, and obtains the limb movement data.

[0015] Preferably, the specific method for extracting the spatio-temporal correlation feature vectors from each of the data is: using a time series analysis model to extract the spatio-temporal correlation feature vectors of the facial expression data; based on a spatio-temporal graph network to model the spatial topological relationship and temporal motion pattern of the joint points to extract the spatio-temporal correlation feature vectors of the limb movement data; capturing the long-term dependence relationship of the speech signal data through a temporal context encoding model to obtain the spatio-temporal correlation feature vectors of the speech signal data; using a semantic structure analysis model to extract the spatio-temporal correlation feature vectors of the text semantic data.

[0016] Preferably, the specific method for optimizing the weights of each data based on the quantum optimization algorithm is as follows: Express the weights of each data as a quantum bit superposition state, and its calculation formula is:

[0017] where, is expressed as a quantum bit superposition state; corresponds to the th weight combination; is the complex probability amplitude; represents all possible combination numbers corresponding to the weights of N data; Dynamically adjust the weights of each data through the quantum rotation gate, and its calculation formula is:

[0018] where, is the rotation gate matrix acting on the th quantum bit; is the rotation angle; is the quantum learning rate; is the loss function; is the weight of the th data; The formula of the quantum particle swarm optimization algorithm is:

[0019] where, is the quantum state of the th weight at the (t + 1)th iteration; is the individual historical optimal quantum state of the th weight; is the global optimal quantum state; is the quantum state superposition operation; is the quantum entanglement operation; and are both random phase angles, taking ; Optimize the weights of each data through the quantum gradient descent method, and its calculation formula is:

[0020] where, is the update amount of the weight of the th data; is the initial learning rate; is the annealing coefficient. <>

[0021] Preferably, the specific method for generating the heat map of each data contribution degree and the counterfactual explanation is as follows: Based on the five-dimensional personality trait vector, generate a heat map of the contribution degree of each data to the personality dimension; by intervening in the data input into the Big Five personality trait prediction model, quantify the change in the prediction result and provide an interpretable explanation.

[0022] In a second aspect, the present invention provides a Big Five personality prediction system based on quantum multi-modal fusion and spatio-temporal graph network, including: An image acquisition module: used to acquire image videos, and extract image sequences, audio signals, and text after audio transcription from the image videos; A data extraction module: used to extract facial expression data, limb movement data, and first text semantic data based on the image sequence; extract voice signal data based on the audio signal; extract second text semantic data based on the text after audio transcription; A feature extraction module: used to extract feature vectors with spatio-temporal correlation from each of the above data respectively; A weight analysis module: used to establish a cross-modal dynamic mapping relationship between the feature vectors and generate weights for each data; A quantum optimization module: used to optimize the weights of each data based on a quantum optimization algorithm, and establish a Big Five personality trait prediction model according to the optimized weights of each data; A result output module: used to collect facial expression data, limb movement data, voice signal data, and text semantic data in real time, input them into the Big Five personality trait prediction model; output a five-dimensional personality trait vector, and generate a heat map of the contribution degree of each data and a counterfactual explanation.

[0023] In a third aspect, the present invention provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the Big Five personality prediction method based on quantum multi-modal fusion and spatio-temporal graph network when executing the computer program.

[0024] In a fourth aspect, the present invention provides a computer-readable storage medium, the computer-readable storage medium stores a computer program, wherein the computer program implements the steps of the Big Five personality prediction method based on quantum multi-modal fusion and spatio-temporal graph network when being executed by a processor.

[0025] Compared with the prior art, the present invention has the following beneficial effects: Through the spatio-temporal feature extraction and dynamic weight allocation of multi-modal data (facial expressions, actions, speech, text), comprehensively capture the complex correlations of personality representations; optimize the modal weights by introducing a quantum optimization algorithm to enhance the sensitivity of the model to personality features; provide a visual decision-making basis for psychological research through contribution heatmaps and counterfactual explanation mechanisms. The present invention realizes efficient multi-modal feature fusion, accurate spatio-temporal feature modeling, and has good interpretability. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] To more clearly illustrate the technical solutions of the embodiments of the present invention, the following will briefly introduce the drawings required in the embodiments. It should be understood that the following drawings only show some embodiments of the present invention, and therefore should not be regarded as limiting the scope. For those of ordinary skill in the art, without creative efforts, other related drawings can also be obtained based on these drawings.

[0027] Figure 1 It is a flowchart of a Big Five personality prediction method based on quantum multi-modal fusion and spatio-temporal graph network of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0028] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the following will clearly and completely describe the technical solutions in the embodiments of the present invention with reference to the drawings in the embodiments of the present invention. Obviously, the described embodiments are some, but not all, of the embodiments of the present invention. Usually, the components of the embodiments of the present invention described and shown in the drawings here can be arranged and designed in various different configurations.

[0029] Therefore, the following detailed description of the embodiments of the present invention provided in the drawings is not intended to limit the scope of the claimed present invention, but merely represents the selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts fall within the scope of protection of the present invention.

[0030] ]>It should be noted that similar reference numerals and letters indicate similar items in the following drawings. Therefore, once an item is defined in one drawing, it does not need to be further defined and explained in subsequent drawings.

[0031] In the description of the embodiments of the present invention, it should be noted that if terms such as "upper", "lower", "horizontal", "inner", etc. are used to indicate the orientation or positional relationship, it is based on the orientation or positional relationship shown in the drawings, or the orientation or positional relationship in which the invention product is usually placed during use. This is only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation. Therefore, it should not be construed as a limitation to the present invention. In addition, terms such as "first", "second", etc. are only used for distinguishing descriptions and cannot be understood as indicating or implying relative importance.

[0032] In addition, if the term "horizontal" appears, it does not mean that the component is required to be absolutely horizontal, but it can be slightly inclined. For example, "horizontal" only means that its direction is more horizontal relative to "vertical", and does not mean that the structure must be completely horizontal, but it can be slightly inclined.

[0033] In the description of the embodiments of the present invention, it should also be noted that unless otherwise clearly specified and limited, if terms such as "set", "installed", "connected", "coupled" are used, they should be understood in a broad sense. For example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be directly connected or indirectly connected through an intermediate medium, and it can be the communication inside two elements. For those of ordinary skill in the art, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0034] The following further describes the present invention in detail with reference to the drawings: The first object of the present invention is to provide a Big Five personality prediction method based on quantum multimodal fusion and spatio-temporal graph network, as Figure 1 shown, including the following steps: Obtain image videos, and extract image sequences, audio signals, and texts after audio transcription from the image videos; Extract image data (facial expression data, limb movement data) and first text semantic data based on the image sequence; extract voice signal data based on the audio signal; extract second text semantic data based on the text after audio transcription; Extract feature vectors with spatio-temporal correlation from each of the above data respectively; Establish a cross-modal dynamic mapping relationship between the feature vectors and generate weights for each data; Optimize the weights of each data based on the quantum optimization algorithm, and establish a Big Five personality trait prediction model according to the optimized weights of each data; Real-time collect facial expression data, body movement data, voice signal data, and text semantic data, and input them into the Big Five personality trait prediction model; output five-dimensional personality trait vectors, and generate a heat map of the contribution of each data and counterfactual explanations.

[0035] Through the extraction of spatio-temporal features and dynamic weight allocation of multi-modal data (expressions, movements, voices, texts), the present invention comprehensively captures the complex correlations of personality representations. Among them, the facial expression data and body movement data obtained by the present invention have been consented to by the collectors. Secondly, by introducing a quantum optimization algorithm to optimize the modal weights, the limitations of traditional methods in non-linear feature fusion are overcome, and the sensitivity of the model to personality features is improved; finally, through the contribution heat map and counterfactual explanation mechanism, the "black box" limitation of traditional personality prediction models is broken through, providing a visual decision-making basis for psychological research. The method of the present invention has a higher prediction accuracy than single-modal models, and shows stronger robustness in cross-sample tests, providing an analytical method with both scientificity and practicality for fields such as talent assessment and mental health screening.

[0036] Among them, before the step of extracting feature vectors with spatio-temporal correlations from each of the above data, preprocessing is performed on each data. Specifically, face detection and alignment are performed on each frame of the image data, a standardized facial region is cropped, and illumination normalization and grayscale correction are performed to reduce environmental interference. Subsequently, a 3D convolutional network is used to extract spatio-temporal features of consecutive frames, capturing dynamic expression changes and micro-expression details, which not only retains key facial movement information but also eliminates background noise, providing an input with a high signal-to-noise ratio for subsequent expression and body movement analysis. For text semantic data, pre-trained language models (such as BERT or RoBERTa) are used to extract context-related word vectors. For captions or descriptive texts in the video, visual context information is additionally fused to enhance semantic understanding. The processed text data contains both the semantic features of the language itself and is associated with visual content, avoiding the problem of losing context in traditional bag-of-words models. Noise reduction, framing, and windowing processing are performed on the voice signal data. Silent segments are removed through voice activity detection, and dynamic time warping (DTW) is used to align speech sequences of different lengths. The processed speech features can clearly reflect paralinguistic information such as the intonation and rhythm of the speaker, providing a robust acoustic representation for subsequent personality trait analysis.

[0037] The specific method for establishing the cross-modal dynamic mapping relationship between the above feature vectors and generating the weights of each data is as follows: Randomly initialize the cross-modal mapping matrix , and ; where represents the direct mapping matrix from the image feature vector to the text semantic feature vector; Represents the direct mapping matrix from the image feature vector to the speech signal feature vector; Represents the direct mapping matrix from the text semantic feature vector to the speech signal feature vector; Convert each feature vector through the mapping matrix. The calculation formula for the mapped feature vector is:

[0038] Where, Represents the image feature vector; Represents the text semantic feature vector; Represents the speech signal feature vector; Represents the mapped image feature vector; Represents the mapped text semantic feature vector; Represents the mapped speech signal feature vector; The calculation formula for the similarity matrix between different modality feature vectors is:

[0039] Where, S is the similarity matrix; The calculation formula for the importance score of each data is:

[0040] Where, Represents the importance score of the th data; Represents the gradient of the loss function of the th data; is the balance parameter of the similarity matrix; is the gradient balance parameter of the loss function;

[0041] Where, is the initial weight value of the th data in the weight matrix; Generate the initial weight matrix according to the initial weight values of each data. The calculation formula for iterative optimization of the weight matrix is:

[0042] Where, is the weight matrix after iterations; is the activation function; is the trainable parameter matrix; is the feature vector after iterations; is the bias matrix; Based on Generate the weights of each data.

[0043] The present invention explicitly establishes a feature conversion channel among the image, text semantics, and speech signal modalities through a randomly initialized cross-modal mapping matrix, effectively solving the semantic alignment problem between heterogeneous modal data. Based on the dynamic calculation of the similarity matrix S, it can adaptively capture the complementary and collaborative relationships among multi-modal data, and the importance scoring mechanism combined with the loss gradient information ensures the sensitivity of the model to key modal features. By introducing a trainable iterative optimization process, the weight matrix can continuously adjust the contribution ratio of each modality, enhancing the robustness of the model to noisy modalities.

[0044] The present invention solves the time asynchrony problem of heterogeneous signals such as videos, audios, and texts by dynamically time warping to align multi-modal time-series data, laying a foundation for cross-modal fusion; models the spatio-temporal dependencies within each modality through a spatio-temporal graph network (such as the temporal evolution of facial expressions and the spatial topology of limb joints), and then combines the cross-attention mechanism to dynamically calculate the correlations between modalities (such as the collaborative relationship between speech intonation and expression changes), significantly improving the fine-grainedness of feature interaction; and the modality importance scoring based on trainable parameters can not only quantify the contribution differences of different modalities to personality traits (such as text being more sensitive to the openness dimension), but also retain interpretability.

[0045] The specific method for extracting image data based on the image sequence in the present invention is as follows: a multi-task convolutional neural network (MTCNN) detects the face region in the image sequence; a three-dimensional deformation model fitting algorithm (3DDFA) is used to reconstruct a three-dimensional facial model based on the face region; a 50-layer residual network (ResNet50) extracts the facial expression data in the three-dimensional facial model; an open-source pose estimation model (OpenPose) extracts the coordinates of 18 limb joint points in the image sequence, calculates the velocity and acceleration features of each limb joint point coordinate, and obtains the limb movement data. The use of multi-modal hybrid technology to accurately extract facial expression and limb movement features significantly improves the fine-grained analysis ability of personality trait prediction.

[0046] Secondly, the present invention uses the BERT model to perform semantic parsing on the image sequence (such as mapping a smiling face to the text description "smile"), generating the first text semantic data, which forms a dual text representation with the second text semantic data after audio transcription, not only expanding the source of text semantic data, but also achieving a deep semantic interpretation of non-verbal behaviors (such as emoji symbolization) through visual-language cross-modal alignment.

[0047] This method uses multi-modal hybrid technology to accurately extract facial expression and limb movement features, significantly enhancing the fine-grained analysis ability of personality trait prediction. Specifically, MTCNN efficiently locates the face region to ensure the accuracy of subsequent analysis; 3DDFA reconstructs the three-dimensional facial model, overcoming the interference of illumination and angle in two-dimensional images and improving the capture sensitivity of micro-expressions (such as brief frowning); ResNet50 extracts deep expression features from the three-dimensional model to effectively identify complex emotions (such as a forced smile); the dynamic features (speed / acceleration) of 18 joints extracted by OpenPose quantify the spatio-temporal patterns of body language (such as gesture frequency). The whole method solves the defect of insufficient representation of non-verbal behavior in traditional methods by integrating three-dimensional facial dynamics and limb kinematics features.

[0048] The specific method for extracting the spatio-temporal correlation feature vectors from the respective data in the present invention is as follows: using a time series analysis model to extract the spatio-temporal correlation feature vectors of facial expression data; based on a spatio-temporal graph network to model the spatial topological relationship and temporal motion pattern of joint points to extract the spatio-temporal correlation feature vectors of limb movement data; capturing the long-term dependence relationship of speech signal data through a temporal context encoding model to obtain the spatio-temporal correlation feature vectors of speech signal data; using a semantic structure analysis model to extract the spatio-temporal correlation feature vectors of text semantic data.

[0049] The specific method for using a time series analysis model to extract the spatio-temporal correlation feature vectors of facial expression data is as follows: first, extract the spatial information of the facial expression data as the data input into the time series analysis model, and then obtain the spatio-temporal correlation feature vectors of the facial expression data, realizing the deep fusion and collaborative analysis of spatial and temporal information.

[0050] The specific method for optimizing the weights of the respective data based on the quantum optimization algorithm is as follows: representing the weights of the respective data as a superposition state of quantum bits, and its calculation formula is:

[0051] Among them, is represented as a superposition state of quantum bits; corresponds to the th weight combination; is the complex probability amplitude; represents the total number of all possible combinations corresponding to the weights of N data.

[0052] Dynamically adjust the weights of the respective data through a quantum rotation gate, and its calculation formula is:

[0053] Among them, is the rotation gate matrix acting on the th quantum bit; is the rotation angle; is the quantum learning rate; is the loss function; is the -th data weight.

[0054] The formula of the quantum particle swarm optimization algorithm is:

[0055] where, is the quantum state of the -th weight at the (t + 1)-th iteration; is the individual historical optimal quantum state of the -th weight; is the global optimal quantum state; is the quantum state superposition operation; is the quantum entanglement operation; and are both random phase angles, usually taking .

[0056] The weights of each data are optimized by the quantum gradient descent method, and its calculation formula is:

[0057] where, is the update amount of the -th weight; is the initial learning rate; is the annealing coefficient.

[0058] In the present invention, by encoding the weights as the superposition state of quantum bits, the parallel search of the high-dimensional weight space is realized, and the search efficiency is improved compared with the traditional method, effectively avoiding the local optimal problem; the quantum rotation gate dynamically adjusts the rotation angle according to the prediction error, making the weight optimization process have the adaptive ability and the model accuracy is higher; combined with the dynamic inertia weight mechanism of the quantum particle swarm optimization, the global exploration ability is enhanced in the early stage and the local fine tuning is focused in the later stage, and cooperating with the fast convergence characteristic of the quantum gradient descent, the model shows stronger robustness in the cross-dataset test. The quantum optimization framework of the present invention not only improves the prediction accuracy of personality traits, but also its quantum state collapse characteristic provides a verifiable quantitative basis for the contribution degree of each modality, enhancing the interpretability of the model.

[0059] The specific method for generating the heatmap of data contribution degrees and counterfactual explanations is as follows: Based on the five-dimensional personality trait vector, a heatmap of the contribution degrees of each data to the personality dimensions is generated; by intervening in the data input into the Big Five personality trait prediction model, the change in the prediction result is quantified and an interpretable explanation is provided. The heatmap of contribution degrees can intuitively show the differential contributions of each modality data (such as "Extraversion: Voice 45%, Body 30%") to different personality dimensions, enabling psychology experts to quickly identify key behavioral characteristics; the counterfactual explanation quantifies the change in the prediction result and generates a natural language explanation (such as "The Conscientiousness score drops by 12%") by systematically intervening in the input data (such as simulating the scenario of "turning off the body language modality").

[0060] In a second aspect, the present invention provides a Big Five personality prediction system based on quantum multi-modal fusion and spatio-temporal graph network, including: Image acquisition module: used to acquire image videos, and extract image sequences, audio signals, and texts after audio transcription from the image videos; Data extraction module: used to extract facial expression data, body movement data, and first text semantic data based on the image sequence; extract voice signal data based on the audio signal; extract second text semantic data based on the text after audio transcription; Feature extraction module: used to extract feature vectors with spatio-temporal correlation from each of the above data respectively; Weight analysis module: used to establish a cross-modal dynamic mapping relationship between the feature vectors and generate the weights of each data; Quantum optimization module: used to optimize the weights of each data based on the quantum optimization algorithm, and establish a Big Five personality trait prediction model according to the optimized weights of each data; Result output module: used to collect facial expression data, body movement data, voice signal data, and text semantic data in real time, and input them into the Big Five personality trait prediction model; output a five-dimensional personality trait vector, and generate a heatmap of data contribution degrees and counterfactual explanations.

[0061] The system of the present invention realizes high-precision prediction and interpretable analysis of Big Five personality traits by integrating quantum multi-modal and spatio-temporal graph network technologies.

[0062] In one embodiment of the present invention, a computer device is provided, which includes a processor and a memory. The memory is used to store a computer program, and the computer program includes program instructions. The processor is used to execute the program instructions stored in the computer storage medium. The processor may be a Central Processing Unit (CPU), or may also be other general-purpose processors, Digital Signal Processors (DSPs), Application Specific Integrated Circuits (ASICs), Field-Programmable Gate Arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. It is the computing core and control core of the terminal, and is suitable for implementing one or more instructions. Specifically, it is suitable for loading and executing one or more instructions in the computer storage medium to implement the corresponding method flow or corresponding function. The processor described in the embodiment of the present invention can be used for the operation of the Big Five personality prediction method based on quantum multimodal fusion and spatio-temporal graph network.

[0063] The present invention also provides a storage medium, specifically a computer-readable storage medium (Memory). The computer-readable storage medium is a memory device in a computer device and is used to store programs and data. It can be understood that the computer-readable storage medium here can include both the built-in storage medium in the computer device and, of course, the extended storage medium supported by the computer device. The computer-readable storage medium provides a storage space, which stores the operating system of the terminal. And in this storage space, there are also stored one or more instructions suitable for being loaded and executed by the processor. These instructions can be one or more computer programs (including program codes). It should be noted that the computer-readable storage medium here can be a high-speed RAM memory or a non-volatile memory, such as at least one disk memory. One or more instructions stored in the computer-readable storage medium can be loaded and executed by the processor to implement the corresponding steps of the Big Five personality prediction method based on quantum multimodal fusion and spatio-temporal graph network in the above embodiments.

[0064] Those skilled in the art should understand that the embodiments of the present invention can be provided as a method, a system, or a computer program product. Therefore, the present invention can take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present invention can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk memory, CD-ROM, optical memory, etc.) that contain computer-usable program code.

[0065] The present invention is described with reference to the flowcharts and / or block diagrams of methods, apparatuses (systems), and computer program products according to embodiments of the present invention. It should be understood that each flow and / or block in the flowchart and / or block diagram, as well as the combination of flows and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing devices to generate a machine, such that the instructions executed by the processor of the computer or other programmable data processing devices generate means for implementing the functions specified in Figure 1 one flow or multiple flows and / or blocks Figure 1 one block or multiple blocks.

[0066] These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to work in a specific manner, such that the instructions stored in the computer-readable memory generate a manufactured article including instruction means that implement the functions specified in Figure 1 one flow or multiple flows and / or blocks Figure 1 one block or multiple blocks.

[0067] These computer program instructions can also be loaded onto a computer or other programmable data processing device, such that a series of operation steps are executed on the computer or other programmable device to generate a computer-implemented process, and thus the instructions executed on the computer or other programmable device provide steps for implementing the functions specified in Figure 1 one flow or multiple flows and / or blocks Figure 1 one block or multiple blocks.

[0068] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit them. Although the present invention has been described in detail with reference to the above embodiments, those of ordinary skill in the art should understand that: still can modify the specific implementation manners of the present invention or make equivalent replacements, and any modification or equivalent replacement without departing from the spirit and scope of the present invention should be covered by the protection scope of the claims of the present invention.

Claims

1. A Big Five personality prediction method based on quantum multimodal fusion and spatio-temporal graph network, characterized in that, Including the following steps: Obtain an image video, and extract an image sequence, an audio signal, and text after audio transcription from the image video; Extract image data and first text semantic data based on the image sequence; extract speech signal data based on the audio signal; extract second text semantic data based on the text after audio transcription and the image sequence; Extract feature vectors with spatio-temporal correlation from each of the data respectively; Establish a cross-modal dynamic mapping relationship between the feature vectors, and generate weights for each data; Optimize the weights of each data based on a quantum optimization algorithm, and establish a Big Five personality trait prediction model according to the optimized weights of each data; Collect image data, speech signal data, and text semantic data in real time, and input them into the Big Five personality trait prediction model; output a five-dimensional personality trait vector, and generate a heat map of the contribution degree of each data and a counterfactual explanation.

2. The big five personality prediction method based on quantum multimodal fusion and spatio-temporal graph network according to claim 1, characterized in that, Before the step of extracting feature vectors with spatio-temporal correlation from each of the data respectively, preprocess each data.

3. A Big Five personality prediction method based on quantum multimodal fusion and spatio-temporal graph network according to claim 1, characterized in that The specific method for establishing a cross-modal dynamic mapping relationship between the feature vectors and generating weights for each data is: Randomly initialize the cross-modal mapping matrix , and ; where represents the direct mapping matrix from the image feature vector to the text semantic feature vector; represents the direct mapping matrix from the image feature vector to the speech signal feature vector; represents the direct mapping matrix from the text semantic feature vector to the speech signal feature vector; Convert each eigenvector through a mapping matrix. The calculation formula for the eigenvector after mapping is as follows: Among them, represents the image feature vector; represents the text semantic feature vector; represents the voice signal feature vector; represents the mapped image feature vector; represents the mapped text semantic feature vector; represents the mapped voice signal feature vector; The calculation formula for the similarity matrix between feature vectors of different modalities is: where S is the similarity matrix; The calculation formula for the importance score of each data is: Among them, represents the importance score of the th data; represents the gradient of the th data loss function; is the balance parameter of the similarity matrix; is the gradient balance parameter of the loss function; The calculation formula for the initial weight of each data is: Among them, is the initial weight value of the th data in the weight matrix; Generate an initial weight matrix according to the initial weight values of each data, and the calculation formula for iterative optimization of the weight matrix is: Among them, is the weight matrix after iterations; is the activation function; is the trainable parameter matrix; is the -th iteration feature vector; is the bias matrix; Based on Generate the weights of each data.

4. A Big Five personality prediction method based on quantum multimodal fusion and spatio-temporal graph network according to claim 1, characterized in that, The image data includes facial expression data and limb movement data; the specific method for extracting image data based on the image sequence is: MTCNN detects the face area in the image sequence; based on 3DDFA and according to the face area, reconstructs a three-dimensional facial model; ResNet50 extracts facial expression data from the three-dimensional facial model; OpenPose extracts the coordinates of 18 limb joint points in the image sequence, calculates the velocity and acceleration features of each limb joint point coordinate, and obtains limb movement data.

5. A method for predicting the Big Five personality based on quantum multimodal fusion and spatio-temporal graph network according to claim 4, characterized in that, The specific method for extracting feature vectors with spatio-temporal correlation from each of the data respectively is: use a time series analysis model to extract feature vectors with spatio-temporal correlation of facial expression data; based on a spatio-temporal graph network, model the spatial topological relationship and temporal motion pattern of joint points to extract feature vectors with spatio-temporal correlation of limb movement data; capture the long-term dependence relationship of speech signal data through a temporal context encoding model to obtain feature vectors with spatio-temporal correlation of speech signal data; use a semantic structure analysis model to extract feature vectors with spatio-temporal correlation of text semantic data.

6. The big five personality prediction method based on quantum multimodal fusion and spatio-temporal graph network according to claim 1, characterized in that The specific method for optimizing the weights of each data based on the quantum optimization algorithm is: Express the weights of each data as a quantum bit superposition state, and its calculation formula is: Among them, is expressed as a superposition state of qubits; corresponds to the th weight combination; is a complex probability amplitude; represents the total number of all possible combinations corresponding to N data weights; Dynamically adjust the weights of each data through a quantum rotation gate, and its calculation formula is: Among them, is the rotation gate matrix acting on the th qubit; is the rotation angle; is the quantum learning rate; is the loss function; is the th data weight; The formula for the quantum particle swarm optimization algorithm is: Among them, is the quantum state of the -th weight at the (t + 1)-th iteration; is the individual historical optimal quantum state of the -th weight; is the global optimal quantum state; is the quantum state superposition operation; is the quantum entanglement operation; and are both random phase angles, taking ; Optimize the weights of each data through the quantum gradient descent method, and its calculation formula is: Among them, The update amount of the weight of the nth data; is the initial learning rate; is the annealing coefficient.

7. A Big Five personality prediction method based on quantum multi-modal fusion and spatio-temporal graph network according to claim 1, characterized in that, The specific method for generating the heat map of the contribution degree of each data and the counterfactual explanation is as follows: Based on the five-dimensional personality trait vector, a heat map of the contribution degree of each data to the personality dimension is generated; by intervening in the data input into the Big Five personality trait prediction model, the change in the prediction result is quantified and an interpretable explanation is provided.

8. A Big Five personality prediction system based on quantum multimodal fusion and spatio-temporal graph network, characterized in that, Including: Image acquisition module: used to acquire image videos, and extract image sequences, audio signals, and text after audio transcription from the image videos; Data extraction module: used to extract image data and first text semantic data based on the image sequence; extract voice signal data based on the audio signal; extract second text semantic data based on the text after audio transcription; Feature extraction module: used to extract feature vectors with spatio-temporal correlation from each of the data; Weight analysis module: used to establish a cross-modal dynamic mapping relationship between the feature vectors and generate the weights of each data; Quantum optimization module: used to optimize the weights of each data based on the quantum optimization algorithm, and establish a Big Five personality trait prediction model according to the optimized weights of each data; Result output module: used to collect image data, voice signal data, and text semantic data in real time, input them into the Big Five personality trait prediction model; output the five-dimensional personality trait vector, and generate a heat map of the contribution degree of each data and a counterfactual explanation.

9. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, the steps of the method according to any one of claims 1-7 are implemented.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, the steps of the method according to any one of claims 1-7 are implemented.

Images