Method and device for evaluating recognition degree of ai generated virtual avatar based on multi-modal data fusion

Abstract

The application discloses an AI-generated virtual avatar recognition degree evaluation method and device based on multi-modal data fusion, relates to the technical field of artificial intelligence, solves the problem that the virtual avatar is difficult to be objectively evaluated due to single dimension and subjective cognitive interference when the virtual avatar is evaluated in the prior art, and the method comprises the following steps: key feature extraction is performed on eye movement data, fMRI data and behavior data of a subject after preprocessing, a late fusion strategy and a cross-modal attention mechanism are adopted to realize dynamic weighted fusion of the key features to obtain multi-modal features, a virtual avatar recognition degree prediction model is constructed based on the multi-modal features and is applied, complementary information between modalities can be fully mined, the prediction accuracy and objectivity of the virtual avatar recognition degree are improved, the virtual avatar recognition degree can be objectively and accurately evaluated, and the prediction effect is significantly improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, and in particular to an AI-generated virtual avatar identification degree evaluation method and device based on multi-modal data fusion. BACKGROUND

[0002] With the rapid development of artificial intelligence, virtual reality and neuroimaging technology, AI-generated virtual avatars play an increasingly important role in social media, online entertainment and commercial promotion. Virtual avatar identification degree refers to the identification and affinity of consumers to virtual avatars generated by artificial intelligence in terms of appearance, behavior and personality, etc. This indicator not only reflects the emotional investment of consumers in virtual avatars, but also embodies their subjective evaluation of the degree of identity between virtual images and their own identity. Existing virtual avatar evaluation methods mainly rely on questionnaire surveys or single data sources. Such methods not only have single data dimensions, but are also easily affected by subjective cognition, making it difficult to objectively reflect the real psychological reactions of consumers. SUMMARY

[0003] The present application aims to overcome the problem of single dimension and subjective cognition interference in the prior art, which makes it difficult to objectively evaluate virtual avatars. The present application provides an AI-generated virtual avatar identification degree evaluation method and device based on multi-modal data fusion.

[0004] In a first aspect, an AI-generated virtual avatar identification degree evaluation method based on multi-modal data fusion is provided, comprising:

[0005] obtaining subject data, wherein the subject data includes eye movement data, fMRI data and behavior data of subjects watching static images of test virtual avatars generated by AI software;

[0006] preprocessing the subject data;

[0007] extracting key features from the preprocessed subject data;

[0008] adopting a late fusion strategy and a cross-modal attention mechanism to realize dynamic weighted fusion of the key features to obtain multi-modal features;

[0009] constructing an avatar identification degree prediction model based on the multi-modal features;

[0010] using the avatar identification degree prediction model to evaluate the identification degree of AI-generated virtual avatars to output evaluation results.

[0011] In some possible implementation manners, the evaluation result is output through an interactive interface, wherein the evaluation result includes a virtual avatar identification degree prediction score and a confidence interval thereof, a distribution diagram of contribution weights of each modality in the fusion process, and dynamic trends of contribution of each modality feature and change over time are exhibited by using a radar chart, a heat map and a line chart, and the interactive interface is used to realize data export and interactive data filtering.

[0012] In some possible implementation manners, the fMRI data is BOLD signal collected by a functional magnetic resonance imaging device, and the behavior data includes subjective scores and reaction time.

[0013] In some possible implementation manners, the pre-processing of the eye movement data includes data format conversion, removal of noise during blinking by using a low-pass filter and the like, detection of stable fixation points, and filtering of abnormal data; the pre-processing of the fMRI data includes head motion correction by using rigid transformation, time layer correction, spatial standardization, Gaussian smoothing and frequency domain band-pass filtering processing; and the pre-processing of the behavior data includes cleaning, processing of missing values and abnormal values, data standardization, and design of a special pre-processing procedure for different characteristics of the eye movement data, the fMRI data and the behavior data, so as to ensure data quality.

[0014] In some possible implementation manners, the key features of the pre-processed eye movement data include fixation duration, saccade speed and fixation heat map distribution; the key features of the pre-processed fMRI data include a neural score, and a calculation formula of the neural score is: Score = (∑ voxel (BOLD_ voxel × Weight_ voxel )) / (∑ voxel Weight_ voxel ), wherein BOLD_voxel is a blood oxygen level dependent (BOLD) signal intensity value of each voxel, Weight_ voxel represents importance weight of the voxel in the identification degree prediction task, and a statistical parameter method is used to determine Weight_ voxel ; and the key features of the pre-processed behavior data include a quantitative feature vector constituted by subjective scores of the subjects.

[0015] In some possible implementation manners, a late fusion strategy and a cross-modality attention mechanism are used to realize dynamic weighted fusion of the key features, including:

[0016] encoding the key features to obtain an encoding vector of a fixed dimension;

[0017] generating a query vector Q through a fully connected layer, and generating a key vector K for each modalityi and value vector V i ;

[0018] Calculate the matching score S i i of each modality i , where f(·) includes scaled dot product;

[0019] Normalize the matching score by using the Softmax function to obtain the weight:

[0020] ;

[0021] where, represents the attention weight of each modality, i represents the modality whose attention weight is currently being calculated, and j represents the index of all modalities; S i is the matching score, exp(S i ) represents the exponential form of the matching score S i , represents the sum of the exponential forms of the matching scores of all modalities j;

[0022] Sum the encoding vectors of each modality by weighting according to the corresponding attention weight to obtain the unified multi-modal feature representation F fusion :

[0023] F fusion =∑ i α i ·V i

[0024] where V i is the value vector from the i-th modality.

[0025] In some possible implementations, a virtual identity recognition degree prediction model is constructed based on the multi-modal feature, including:

[0026] Construct a data set by combining the multi-modal feature and the virtual identity recognition degree label;

[0027] Divide the data set into a training set and a test set according to a preset proportion;

[0028] Construct a regression prediction model based on a multi-layer perception or a convolutional neural network;

[0029] Train the regression prediction model using the training set and the test set;

[0030] Perform cross-validation and performance evaluation on the regression prediction model, and control the false positive rate through multiple comparison correction to obtain the virtual identity recognition degree prediction model.

[0031] ​In a second aspect, an AI-generated virtual avatar identification degree evaluation device based on multi-modal data fusion is provided, comprising:

[0032] a data acquisition module configured to acquire subject data, wherein the subject data comprises eye movement data, fMRI data and behavior data of a subject watching a virtual avatar static image generated by AI software;

[0033] a preprocessing module configured to preprocess the subject data;

[0034] a feature extraction module configured to extract key features from the preprocessed subject data;

[0035] a weighted fusion module configured to realize dynamic weighted fusion of the key features by using a late fusion strategy and a cross-modal attention mechanism to obtain multi-modal features;

[0036] a model construction module configured to construct an avatar identification degree prediction model based on the multi-modal features;

[0037] a prediction module configured to perform identification degree evaluation of an AI-generated virtual avatar by using the avatar identification degree prediction model to output an evaluation result.

[0038] In a third aspect, a computer readable storage medium is provided, the computer readable medium stores program codes for execution by a device, the program codes comprising steps for executing the method in any one of the implementation manners of the first aspect.

[0039] In a fourth aspect, an electronic device is provided, the electronic device comprising a processor, a memory, and a program or instructions stored on the memory and executable on the processor, the program or instructions being executed by the processor to implement the method in any one of the implementation manners of the first aspect.

[0040] The present application has the following beneficial effects: by using late fusion and cross-modal attention mechanism to fuse key features in eye movement data, fMRI data and behavior data, complementary information between modalities can be fully mined, which has important theoretical significance and practical value for improving the prediction accuracy and objectivity of virtual avatar identification degree, and the avatar identification degree prediction model constructed based on the fused features can objectively and accurately evaluate the virtual avatar identification degree, significantly improving the prediction effect. BRIEF DESCRIPTION OF DRAWINGS

[0041] The accompanying drawings, which form a part of this application, are included to provide a further understanding of the application and are incorporated in and constitute a part of this application. The embodiments illustrated in the drawings are provided to explain the present application and are not intended to limit the present application.

[0042] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed for use in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0043] Figure 1 is a flowchart of the AI-generated virtual avatar identification degree evaluation method based on multi-modal data fusion of the embodiment 1 of the present application;

[0044] Figure 2 is a schematic diagram of the avatar identification degree prediction model in the AI-generated virtual avatar identification degree evaluation method based on multi-modal data fusion of the embodiment 1 of the present application;

[0045] Figure 3 is a structural block diagram of the AI-generated virtual avatar identification degree evaluation device based on multi-modal data fusion of the embodiment 2 of the present application;

[0046] Figure 4 is a schematic diagram of the internal structure of the electronic device of the embodiment 4 of the present application.

[0047] Reference signs:

[0048] 100, data acquisition module; 200, preprocessing module; 300, feature extraction module; 400, weighted fusion module; 500, model construction module; 600, prediction module. DETAILED DESCRIPTION

[0049] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some embodiments of the present application, not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0050] Embodiment 1

[0051] As shown in Figure 1 , the AI-generated virtual avatar identification degree evaluation method based on multi-modal data fusion of the embodiment 1 of the present application includes:

[0052] S100, acquiring subject data, wherein the subject data includes eye movement data, fMRI data and behavior data of a subject watching a static image of a virtual avatar of a subject generated by AI software;

[0053] First, a virtual avatar for the experiment needs to be prepared, for example, ask the subject to take a photo on the day of the experiment, use the photo as the original input to generate a virtual avatar, and process the photo through pre-prepared AI software to generate a corresponding virtual avatar static image.

[0054] Then, data collection is carried out, specifically: while the subject is watching the generated virtual avatar image, both behavior data and fMRI data are collected. The collection of behavior data includes: recording the subject's subjective score (such as identification, preference), reaction time and other related behavior indicators during the watching process; the collection of fMRI data includes: using functional magnetic resonance imaging equipment to collect the subject's brain BOLD signal during the watching process, reflecting the neural activation situation.

[0055] Finally, an independent experimental link is designed to collect eye movement data, using a high-precision eye tracker to collect eye movement data of the subject when watching the virtual avatar image, wherein the eye movement data includes fixation points, saccade trajectories and pupil diameters, etc.

[0056] S200, pre-processing the subject data;

[0057] In order to ensure the quality of the data, special preprocessing procedures are designed for the different characteristics of eye movement data, fMRI data and behavior data, specifically:

[0058] The preprocessing of eye movement data includes:

[0059] Data format conversion: convert the original eye movement record (CSV or proprietary format) to a standard format for subsequent processing.

[0060] Noise filtering: use low-pass filtering or other appropriate filtering algorithms to remove blinking and other high-frequency noise interference, ensuring data smoothing.

[0061] Abnormal data rejection: use statistical methods (outlier detection) to remove obviously abnormal data points to improve data accuracy.

[0062] Feature extraction: extract key indicators from the processed eye movement data, including fixation point position, fixation duration, saccade trajectories and pupil diameter changes, etc., as input features for subsequent cross-modal fusion.

[0063] The preprocessing of fMRI data includes:

[0064] Data format conversion: convert the original fMRI data (DICOM format) to a standard format (NIfTI) for unified processing.

[0065] Head motion correction: Correcting subject head motion using rigid-body transformation to align images at different time points.

[0066] Slice timing correction: Correcting each slice data for time layer, eliminating the influence of time difference during acquisition.

[0067] Spatial normalization: Mapping each subject's image to a standard template (MNI space) through nonlinear transformation, realizing cross-individual data comparison.

[0068] Spatial smoothing: Using Gaussian smoothing to process data, reducing local noise and improving signal-to-noise ratio.

[0069] Frequency band-pass filtering: Using band-pass filter (0.01-0.1 Hz) to filter out low-frequency drift and high-frequency noise, retaining signals related to cognitive activities.

[0070] Preprocessing of behavioral data includes:

[0071] Data cleaning: Checking original behavioral data, removing error records and duplicate data.

[0072] Missing value and outlier processing: Using interpolation or statistical methods to fill in missing values, while identifying and removing or correcting abnormal data to ensure data integrity and accuracy.

[0073] Standardization: Standardizing each dimension data (e.g. Z-score standardization) to make different dimension data comparable, facilitating subsequent multi-modal data fusion and model training.

[0074] S300, extracting key features from preprocessed subject data, specifically including the following steps:

[0075] Extracting fixation duration, saccade speed and fixation heat map distribution from preprocessed eye movement data;

[0076] Mapping fMRI activation images to pre-defined brain region templates related to virtual avatar identification, calculating each subject's neural score using the formula: Score=(∑_voxel(BOLD_voxel×Weight_voxel)) / (∑_voxelWeight_voxel), where BOLD_voxel is the blood oxygen level dependent (BOLD) signal intensity value of each voxel, Weight_ voxel represents the importance weight of the voxel in the identification degree prediction task, and the statistical parameter method is used to determine Weight_ voxel. Specifically, the pre-processed fMRI data are analyzed using a general linear model (GLM) to calculate the degree of activation of each voxel in the task related to virtual avatar identification, which is usually expressed as a β value. The β value reflects the response intensity of each voxel to the experimental conditions and can therefore be used as the weight of the voxel in the calculation of the neural score. In order to ensure the comparability of the weights of each voxel, these β values ​​are normalized, that is, the sum of the β values ​​of all voxels is standardized to 1, so that the contribution of each voxel is reasonably reflected in the overall score. This method is objective and has a broad application basis, and can better reflect the true contribution of each voxel in task-related neural activity.

[0077] Statistical analysis is performed on behavioral data to extract key indicators such as subjective scores and reaction time to form a behavioral feature vector.

[0078] S400, adopting a late fusion strategy and a cross-modal attention mechanism to implement dynamic weighted fusion of the key features to obtain multimodal features, specifically including:

[0079] S401, Feature Encoding: Use convolutional neural networks (CNNs) or fully connected layers to encode the key features extracted from the eye movement data, fMRI data, and behavioral data, respectively, to obtain fixed-dimensional encoding vectors F_eye, F_fMRI, and F_behav.

[0080] Constructing a cross-modal attention module:

[0081] S402: Generate a query vector Q using a fully connected layer.

[0082] S403. Generate a corresponding key vector K for each mode i Sum value vector V i (Generate key vector K through linear transformation i Sum value vector V i ); these value vectors V i It represents the weighted summation of the features of each modality in the subsequent fusion.

[0083] S404, using a unified query vector Q (which can be generated by the encoding vectors of all modalities) and the key vector K of each modality i , calculate the matching score S i : S i =f(Q, K i ), for example, f(·) is the scaled dot product, i.e. , where d is the key vector dimension, K i is the key vector corresponding to each modality. Calculate the matching score or calculate it through MLP nonlinear transformation.

[0084] S405, these matching scores are normalized by Softmax to obtain the attention weight of each modality , that is, the contribution ratio of the i-th modality to the final fusion result, which represents the importance of the modality feature in the current task. The scores are normalized by using the Softmax function: .

[0085] S406, finally, multiply the value vector V i of each modality by the corresponding weight , and sum all modalities to obtain a unified fusion feature representation.

[0086] That is, the weight determines the contribution ratio of eye movement, nuclear magnetic and behavior data in the final fusion representation, thereby dynamically affecting the identification score of the AI generated virtual avatar. The modal encoding vectors are weighted and summed according to the corresponding weights to form a unified fusion feature F fusion :

[0087] F fusion =∑ I α i ·V i

[0088] i∈{eye, fMRI, behav}

[0089] Where V i is the value vector from the i-th modality, eye represents the key features extracted from eye movement data, fMRI represents the key features extracted from fMRI data, and behav represents the key features extracted from behavior data.

[0090] In this embodiment, the late fusion and cross-modal attention mechanism are used, which can fully exploit the complementary information between modalities; it has important theoretical significance and practical value for improving the prediction accuracy and objectivity of virtual avatar identification.

[0091] S500, constructing an avatar identification prediction model based on the multi-modal features;

[0092] As shown in Figure 2 , first, multiply the fusion feature F fusionThe virtual avatar identification degree labels corresponding to each subject are combined to form a complete dataset. This dataset is then randomly divided into a training set and a test set (e.g., 80% for training and 20% for testing) to ensure the model's generalization ability on unknown data. Next, a regression model based on a multi-layer perceptron (MLP) is constructed to map the fused features to the predicted scores of the virtual avatar identification degree. The model structure includes an input layer, multiple hidden layers, and an output layer. The input layer is responsible for receiving preprocessed and normalized F fusion features; while the hidden layers are composed of multiple fully connected layers, each layer uses a ReLU activation function and introduces Dropout as necessary to prevent overfitting, thereby gradually capturing the non-linear relationship between the fused features and the target labels; the output layer consists of a single neuron that uses a linear activation function to output the final predicted score.

[0093] During model training, mean squared error (MSE) is used as the loss function, and the model parameters are iteratively updated by an optimizer (such as Adam) to minimize the error between the predicted scores and the actual labels. At the same time, cross-validation methods (such as K-fold cross-validation) are used to evaluate the stability and generalization performance of the model, and root mean square error (RMSE) and Pearson correlation coefficient are used as evaluation indicators. In addition, to control the statistical error introduced by multiple comparisons, the FDR correction method is used to adjust the results.

[0094] In this embodiment, the prediction model constructed based on the fused features can objectively and accurately evaluate the virtual avatar identification degree, significantly improve the prediction effect, and detailedly describe the algorithm process and key formulas of each module, thereby enhancing the scientificity and protection strength of the technical solution. The avatar identification degree prediction model of the present application is suitable for virtual image design, personalized generation, market marketing, and other fields, and has wide commercial application value.

[0095] S600, using the avatar identification degree prediction model to perform identification degree evaluation on the AI-generated virtual avatar to output an evaluation result.

[0096] To output more intuitive data and realize data export and interactive filtering, the evaluation result is output through an interactive interface, wherein the evaluation result includes virtual avatar identification degree prediction scores and their confidence intervals, distribution graphs of the contribution weights of each modality in the fusion process, and dynamic trends of the contribution of each modality feature and its change over time are displayed using radar charts, heat maps, and line charts, which intuitively display the contribution of each modality feature and its dynamic change over time. Through the interactive interface, users can export, filter, and deeply analyze the data.

[0097] Embodiment 2

[0098] AsFigure 3 The AI-generated virtual avatar recognition degree evaluation device based on multi-modal data fusion related in Embodiment 2 of the present application comprises:

[0099] The data acquisition module 100 is configured to acquire subject data, wherein the subject data comprises eye movement data, fMRI data and behavior data of a subject watching a static image of a virtual avatar of a subject generated by AI software;

[0100] The preprocessing module 200 is configured to preprocess the subject data;

[0101] The feature extraction module 300 is configured to extract key features from the preprocessed subject data;

[0102] The weighted fusion module 400 is configured to realize dynamic weighted fusion of the key features by using a late fusion strategy and a cross-modal attention mechanism to obtain multi-modal features;

[0103] The model construction module 500 is configured to construct an avatar recognition degree prediction model based on the multi-modal features;

[0104] The prediction module 600 is configured to use the avatar recognition degree prediction model to evaluate the recognition degree of an AI-generated virtual avatar to output an evaluation result.

[0105] It should be noted that other specific embodiments of the AI-generated virtual avatar recognition degree evaluation device based on multi-modal data fusion in the present embodiment can refer to the specific embodiments of the AI-generated virtual avatar recognition degree evaluation method based on multi-modal data fusion described above, and to avoid redundancy, details are not repeated here.

[0106] Embodiment 3

[0107] The computer-readable storage medium related in Embodiment 3 of the present application stores program code for execution by a device, and the program code comprises steps for executing the method in any one of the implementation manners in Embodiment 1 of the present application.

[0108] The computer-readable storage medium can be a read only memory (ROM), a static storage device, a dynamic storage device or a random access memory (RAM). The computer-readable storage medium can store program code, and when the program stored in the computer-readable storage medium is executed by a processor, the processor is configured to execute the steps of the method in any one of the implementation manners in Embodiment 1 of the present application.

[0109] Embodiment 4

[0110] As Figure 4As shown, the electronic device related to the embodiment 4 of the present application comprises a processor, a memory, and a program or instruction stored in the memory and executable on the processor, and the program or instruction is executed by the processor to implement the method in any one of the implementation manners in the embodiment 1 of the present application.

[0111] The processor can be a general central processing unit (CPU), a microprocessor, an application specific integrated circuit (ASIC), a graphics processing unit (GPU), or one or more integrated circuits, for executing related programs to implement the method in any one of the implementation manners in the embodiment 1 of the present application.

[0112] The processor can also be an integrated circuit electronic device with signal processing capability. In the implementation process, each step of the method in any one of the implementation manners in the embodiment 1 of the present application can be completed by the integrated logic circuit of hardware in the processor or the instruction in the form of software.

[0113] The processor can also be a general processor, a digital signal processor, an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component. The disclosed methods, steps and logic block diagrams in the embodiments of the present application can be implemented or executed. The general processor can be a microprocessor or the processor can also be any conventional processor. The steps of the method disclosed in the embodiments of the present application can be directly embodied as hardware code processor execution, or executed by a combination of hardware and software modules in the code processor. The software module can be located in a random access memory, a flash memory, a read only memory, a programmable read only memory or an electrically erasable programmable memory, a register or other mature storage medium in the art. The storage medium is located in the memory, and the processor reads the information in the memory, and combines the hardware to complete the functions required by the units included in the data processing device of the embodiments of the present application, or executes the method in any one of the implementation manners in the embodiment 1 of the present application.

[0114] The above is only the preferred specific embodiment of the present application; however, the protection scope of the present application is not limited to this. Any skilled person in the art can make equivalent replacement or change according to the technical solution and the improvement concept of the present application within the technical range disclosed by the present application, which should be covered in the protection scope of the present application.

Claims

1. A method for evaluating the recognition of AI-generated virtual avatars based on multimodal data fusion, characterized in that: include: Acquiring subject data, wherein the subject data includes eye movement data, fMRI data, and behavioral data of a subject viewing a static image of a virtual avatar of the subject generated by the AI ​​software; Preprocessing the subject data; Extract key features from preprocessed subject data; Adopting late fusion strategy and cross-modal attention mechanism to achieve dynamic weighted fusion of the key features to obtain multimodal features; Constructing an avatar identification prediction model based on the multimodal features; Using the avatar recognition prediction model to evaluate the recognition of the AI-generated virtual avatar to output an evaluation result; Among them, the late fusion strategy and cross-modal attention mechanism are used to achieve dynamic weighted fusion of the key features, including: Encoding the key features to obtain an encoding vector of fixed dimension; Generate query vector Q through the fully connected layer and generate key vector K for each modality i Sum value vector V i ; Calculate the matching score S for each modality i :S i =f(Q,K i ), where f(·) includes the scaled dot product; Use the Softmax function to normalize the matching score and get the weight: ； in, represents the attention weight of each modality, i represents the modality for which the attention weight is currently being calculated, and j represents the index of all modalities; S i For each modality matching score, exp(S i ) represents the matching score S i The exponential form of represents the exponential sum of the matching scores of all modal j; The encoding vectors of each modality are weighted and summed according to the corresponding attention weights to obtain a unified multimodal feature representation F fusion : ； Among them, V i is the vector of values ​​from the ith mode; Constructing a doppelganger identification prediction model based on the multimodal features, including: The multimodal features and virtual avatar identification labels form a dataset; Divide the data set into training set and test set according to the preset ratio; Build a regression prediction model based on multi-layer perceptron or convolutional neural network; Training the regression prediction model using a training set and a test set; The regression prediction model was cross-validated and evaluated for performance, and the false positive rate was controlled through multiple comparison correction to obtain a doppelganger identification prediction model.

2. The AI-generated virtual avatar identity evaluation method based on multimodal data fusion according to claim 1 is characterized in that: The evaluation results are output through an interactive interface, where the evaluation results include the predicted score of virtual avatar identification and its confidence interval, the distribution diagram of the contribution weight of each modality in the fusion process, and the use of radar charts, heat maps and line charts to display the contribution of each modality feature and its dynamic trend over time. The interactive interface is used to realize data export and interactive data screening.

3. The AI-generated virtual avatar identity evaluation method based on multimodal data fusion according to claim 1 is characterized in that: The fMRI data is a BOLD signal collected by a functional magnetic resonance imaging device, and the behavioral data includes subjective scores and reaction time.

4. The AI-generated virtual avatar identity evaluation method based on multimodal data fusion according to claim 1 is characterized in that: Preprocessing the eye movement data includes: data format conversion, using low-pass filtering and other methods to remove noise during blinking, detecting stable gaze points, and filtering out abnormal data; preprocessing the fMRI data includes: using rigid transformation to correct head movement, performing time layer correction, spatial normalization, using Gaussian smoothing and frequency domain bandpass filtering; preprocessing the behavioral data includes: cleaning, processing missing values ​​and outliers, and data standardization.

5. The AI-generated virtual avatar identity evaluation method based on multimodal data fusion according to claim 1 is characterized in that: The key features of the pre-processed eye movement data include fixation duration, saccade speed and fixation heat map distribution; the key features of the pre-processed fMRI data include neural score, which is calculated as follows: Score=(∑_ voxel (BOLD_ voxel ×Weight_ voxel )) / (∑_ voxel Weight_ voxel ), where BOLD_ voxel is the blood oxygen level dependent signal intensity value for each voxel, Weight_ voxel It represents the importance weight of the voxel in the identification prediction task; the key features of the preprocessed behavioral data include the quantitative feature vector formed by statistically analyzing the subjective scores of the subjects.

6. An AI-generated virtual avatar identity assessment device based on multimodal data fusion, characterized in that: include: a data acquisition module for acquiring subject data, wherein the subject data includes eye movement data, fMRI data, and behavioral data of a subject viewing a static image of a virtual avatar of the subject generated by the AI ​​software; A preprocessing module, used for preprocessing the subject data; Feature extraction module, used to extract key features from the preprocessed subject data; A weighted fusion module, configured to implement dynamic weighted fusion of the key features using a late fusion strategy and a cross-modal attention mechanism to obtain multimodal features; A model building module, configured to build an avatar identification prediction model based on the multimodal features; A prediction module, configured to use the avatar recognition prediction model to evaluate the recognition of the AI-generated virtual avatar and output an evaluation result; Among them, the late fusion strategy and cross-modal attention mechanism are used to achieve dynamic weighted fusion of the key features, including: Encoding the key features to obtain an encoding vector of fixed dimension; Generate query vector Q through the fully connected layer and generate key vector K for each modality i Sum value vector V i ; Calculate the matching score S for each modality i :S i =f(Q,K i ), where f(·) includes the scaled dot product; Use the Softmax function to normalize the matching score and get the weight: ； in, represents the attention weight of each modality, i represents the modality for which the attention weight is currently being calculated, and j represents the index of all modalities; S i For each modality matching score, exp(S i ) represents the matching score S i The exponential form of represents the exponential sum of the matching scores of all modal j; The encoding vectors of each modality are weighted and summed according to the corresponding attention weights to obtain a unified multimodal feature representation F fusion : ； Among them, V i is the vector of values ​​from the ith mode; Constructing a doppelganger identification prediction model based on the multimodal features, including: The multimodal features and virtual avatar identification labels form a dataset; Divide the data set into training set and test set according to the preset ratio; Build a regression prediction model based on multi-layer perceptron or convolutional neural network; Training the regression prediction model using a training set and a test set; The regression prediction model was cross-validated and evaluated for performance, and the false positive rate was controlled through multiple comparison correction to obtain a doppelganger identification prediction model.

7. A computer-readable storage medium, characterized in that The computer-readable storage medium stores program codes for execution by a device, wherein the program codes include steps for executing the method according to any one of claims 1 to 5.

8. An electronic device, characterized in that: The electronic device includes a processor, a memory, and a program or instruction stored in the memory and executable on the processor, wherein the program or instruction, when executed by the processor, implements the method according to any one of claims 1 to 5.

Images