Multi-modal personality analysis method, analysis model, readable storage medium and device
By using a multimodal personality analysis method, combined with psychological data and image and audio data to train a large model, the bias problem of single-dimensional personality recognition is solved, and more accurate and efficient personality analysis is achieved. It is applicable to scenarios such as criminal behavior analysis, user profiling, and anthropomorphic interaction.
Patent Information
- Application Number
- CN202410958384.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-17
- Publication Date
- 2024-11-08
AI Technical Summary
Existing personality recognition technologies are mainly based on traditional small models or single-dimensional information collection, which leads to biases in understanding users, limited understanding of relevant psychological knowledge, defects in the alignment and fusion of data from different modalities, difficulty in including the understanding of subtle movements, and limited resolution of image encoders.
By using multimodal personality analysis methods, psychological data is collected and preprocessed, combined with image and audio data, and a basic model is formed through large-scale model training. Users are then guided to shoot videos and obtain feature values. Multimodal feature matching is then performed to generate personality analysis conclusions.
It improves the accuracy and efficiency of personality analysis, enhances the understanding of psychological knowledge, and improves the high-resolution image and video understanding capabilities of large models, making it suitable for batch processing analysis scenarios.
Smart Images

Figure BDA0004949558510000101
Abstract
Description
Technical Field
[0001] The present invention belongs to the field of artificial intelligence algorithm-assisted technology, and specifically relates to a multimodal personality analysis method, an analysis model, a readable storage medium and a device. Background Art
[0002] Existing personality recognition technologies are mainly based on traditional small models, or single-dimensional information collection for personality recognition analysis, such as voice, text or video. In terms of personality analysis, analyzing the user's personality from a single modality of data will be missing and there will be deviations in the understanding of the user. The personality analysis model is usually used to analyze the user's behavior, emotions, and actions, and then generate corresponding personality recognition results based on the analysis results. In addition, the existing technology has limitations in understanding psychological knowledge related to psychology when using small models for prediction. Some large model solutions have limitations in the ability to understand long videos and have defects in the alignment and fusion of different modal data. They are limited by the acceptable feature length of the large model, the resolution of the image encoder is limited, and the ability to understand subtle movements is insufficient. For example, in the personality analysis scenario, it is difficult for existing technologies to include knowledge related to psychology, and some thresholds can only be pre-defined from certain dimensions to obtain personality analysis results. Summary of the invention
[0003] In order to solve the problems existing in the prior art, the present invention provides a multimodal personality analysis method, analysis model, readable storage medium and device, which can autonomously guide users to shoot videos and accurately match personality analysis conclusions by acquiring certain behavioral analysis feature data to train the model and simultaneously perform feature recognition on data of multiple modalities.
[0004] The technical solution adopted by the present invention is:
[0005] In a first aspect, the present invention discloses a multimodal personality analysis method, which analyzes a user's personality data based on multiple input materials.
[0006] G100. First, collect psychological data on personality analysis, pre-process the psychological data to form a basic control data set corresponding to behavior and personality analysis, and pre-train the LLM large model with the basic control data set to obtain a basic large model;
[0007] G200. Then, a number of image data and audio data with human behaviors are collected as annotation reference templates, and feature values used to feedback human behaviors in the image data and audio data are analyzed and determined, and the determined feature values are annotated with behavior labels to form corresponding behavior recognition training data sets, and the basic large model is trained with the recognition training data set to obtain a personality analysis model;
[0008] G300. Set a collection scheme, the collection scheme includes some guide information, guide the collection object to shoot the behavior collection video through the guide information, and obtain the feature value according to the collection video preprocessing;
[0009] G400. The personality analysis model determines the corresponding behavioral characteristics based on the characteristic values, and then the personality analysis model matches the corresponding personality analysis data based on the behavioral characteristics to generate the corresponding personality analysis conclusion.
[0010] In combination with the first aspect, the present invention provides a first implementation of the first aspect, in step G100, the preprocessing of the psychology data is specifically as follows:
[0011] Organize psychological data in different recording methods and form them into digital format, and then organize and summarize the digital data;
[0012] The data contains personality analysis conclusions of several behaviors. First, the types of behaviors are classified to determine the number of all behaviors, and then personality analysis conclusions of at least not less than a set threshold number are obtained for each behavior;
[0013] All personality analysis data corresponding to the same behavior are processed, and the repetition of the same behavior is determined through a semantic analysis model. According to a preset repetition threshold, the personality analysis conclusions that are greater than the repetition threshold are matched with the behavior to form a corresponding relationship, and a number of behaviors and the corresponding number of matching personality analysis conclusions are used as the basic control data set.
[0014] In combination with the first implementation of the first aspect, the present invention provides a second implementation of the first aspect, wherein in step G100, the psychological data includes all public data with specific behaviors and corresponding personality analysis conclusions.
[0015] In combination with the first aspect, the present invention provides a third implementation of the first aspect, wherein in step G200, the image data includes independent image data and image stream data at continuous time intervals.
[0016] In combination with the first aspect, the present invention provides a fourth implementation of the first aspect, wherein in step G200, the image data includes a number of images and video materials collected from an undirected group of people using a set collection scheme, and also includes public images and video materials with human behavior.
[0017] In combination with the first aspect, the present invention provides a fifth implementation of the first aspect, in step G200:
[0018] The process of determining the characteristic value of the behavior of the feedback person based on the image data first separates the human and environmental regions in the image data, and then captures the pixel features of the human body and facial regions as characteristic values; then annotates the posture presented by each body region to form a first behavior label, annotates the expression of each facial region to form a second behavior label, and then annotates the environmental region to form a third behavior label, and matches the behavior label obtained in each image data with the characteristic value to form a behavior recognition training data set;
[0019] In the process of determining the characteristic value of the behavior of the feedback person based on the audio data, the human voice and background noise are first separated in the audio data, and the content and tone of the human voice are respectively determined as the characteristic values; then the content is annotated to form a first behavior label, the change of tone is annotated to form a second behavior label, and the background noise is annotated to form a third behavior label. The behavior labels obtained in the audio data are matched with the characteristic values to form a behavior recognition training data set.
[0020] In combination with the first aspect, the present invention provides a sixth implementation of the first aspect, wherein the specific steps of preprocessing the captured video to obtain the feature value in step G300 are as follows:
[0021] First, the feature values are processed by using the Clip model to obtain semantic features in low-resolution videos, and the Vitdet model to obtain image features in high-resolution videos. The semantic features and image features are combined through the feature fusion model, and the image features are clustered and assigned probability values according to their importance, and the image features are reduced in dimension.
[0022] Then, the audio features in the captured video are extracted through the multimodal fusion model of Imagebind, and the text content in the audio features is converted into text features through Tokenizer;
[0023] The reduced-dimensional image features, audio features, and text features are used as feature values.
[0024] In a second aspect, the present invention further provides an analysis model, which is applied in any of the multimodal personality analysis methods described above, comprising:
[0025] The acquisition module guides the user to generate the acquisition video;
[0026] An extraction module processes the collected video and obtains feature values; and
[0027] The matching module matches the corresponding personality analysis data according to the acquired feature values.
[0028] In a third aspect, the present invention further provides a readable storage medium having a computer program stored thereon, wherein when the program is executed by a processor, the multimodal personality analysis method as described in any one of the above is implemented.
[0029] In a fourth aspect, the present invention further provides a device, which is loaded with the above-mentioned analysis model and is used to provide an interactive interface to guide users to shoot and collect videos to analyze the user's personality analysis data.
[0030] The beneficial effects of the present invention are:
[0031] (1) The present invention obtains more behavioral personality analysis data from existing data, processes it to obtain conclusions with more recognition as training data, and uses the LLM large model to learn and form a basic model for behavioral personality analysis of people. Compared with the method based on experience judgment in the prior art, it is more efficient, convenient and accurate, and has a certain degree of authority;
[0032] (2) The present invention obtains accurate behavior recognition samples for training by collecting, marking and annotating various human behaviors. The trained model can capture features of input data of multiple modalities and match corresponding personality analysis conclusions according to feature values, which is convenient for application in scenarios requiring batch processing and analysis, further improving efficiency.
[0033] (3) The present invention uses vitdet to extract high-resolution image features, and uses a feature fusion module to fuse the semantic features of low-resolution images with high-resolution image features. The fused image features have both semantic information and high-resolution image information, thereby improving the high-resolution image understanding capability of the large model without increasing image features.
[0034] (4) The present invention clusters the fused image features, retains important information in the image, reduces the dimensionality of the features, and reduces the image feature dimensions, so that the large model has a longer video understanding capability. DETAILED DESCRIPTION
[0035] The present invention will be further explained below in conjunction with specific embodiments.
[0036] To make the purpose, technical solutions and advantages of the embodiments of the present application clearer, the described embodiments are part of the embodiments of the present application, rather than all of the embodiments. The components of the embodiments of the present application generally described and shown herein can be arranged and designed in various different configurations.
[0037] Embodiment 1:
[0038] This embodiment discloses a multimodal personality analysis method, which guides users to shoot and collect videos for behavioral analysis to determine personality types through training models. Personality broadly refers to the internal tendencies and psychological characteristics that an individual presents to himself and external things in social activities. There are many classification theories for personality types in modern society, such as the Myers-Briggs Type Indicator, the common MBTI, which uses a series of test questions to determine each person's personality type, so that a user portrait of a personality can be quickly formed for the user, which is convenient for a simple and quick definition of each person's personality characteristics.
[0039] There is no strict definition of personality type classification in psychology, because personality is an imprecise summary obtained by evaluating a person's comprehensive behavioral performance and cannot be quantified. As a result, the existing technology is mostly based on empirical judgment, that is, summarizing the behavior-personality rules at a macro level.
[0040] This type of experience summary has many samples and theories, and relying on personal experience judgment has the problem of low efficiency and inaccuracy. The multimodal personality analysis method in this embodiment is based on the existing AI large model for sample training, so that it can be classified based on many existing psychological research materials. Through data processing and aggregation, it can provide a more accurate personality classification judgment based on experience judgment.
[0041] It should be noted that the method provided in this embodiment and the model used in the method are all based on existing data recognized by the psychology community for data processing. The theories therein are all based on existing psychology research theories, and no new standards and analysis methods are proposed. The purpose is to improve judgment efficiency and matching accuracy through automated software programs, so that it is suitable for some scenarios that require a large amount of personality analysis and evaluation, such as criminal behavior analysis, user portrait judgment, human-computer interaction, etc., especially the current technology of using AI large models to realize anthropomorphic interaction, and even in the scenarios of simulated humans and metaverses in the future development.
[0042] Specifically, the analysis method of the present invention comprises the following steps:
[0043] First, we collected a large amount of psychological data on personality analysis, including all the currently available printed and electronic data, and mostly used data from authoritative psychology journals, forums, and academic reports, mainly focusing on the content of human behavior analysis. After preprocessing the psychological data, we formed a basic control data set corresponding to behavior and personality analysis, and used the existing LLM large model to pre-train the basic control data to obtain the basic large model.
[0044] Then, a number of image data and audio data with human behaviors are collected as annotation reference templates, and feature values used to reflect human behaviors in the image data and audio data are analyzed and determined, and the determined feature values are annotated with behavior labels to form a corresponding behavior recognition training data set, and the basic large model is trained with the recognition training data set to obtain a personality analysis model;
[0045] Setting a collection plan, wherein the collection plan includes some guiding information, guiding the collection object to shoot the behavior collection video through the guiding information, and obtaining the feature value according to the preprocessing of the collected video;
[0046] The personality analysis model determines the corresponding behavioral characteristics based on the characteristic values, and then the personality analysis model matches the corresponding personality analysis data based on the behavioral characteristics to generate the corresponding personality analysis conclusion.
[0047] Furthermore, this embodiment further optimizes and limits the analysis method.
[0048] First, core academic journals and periodicals in the field of psychology, as well as research articles and forums published by public channels are used as data acquisition channels to obtain data on analyzing people's behavior and personality types as psychological data. Then the psychological data is preprocessed. If it is paper data, it is scanned to form an electronic version of text data. Through the semantic recognition algorithm, the behaviors and personality analysis conclusions with correlation in the electronic version of the psychological data are extracted, and then a pairing relationship is formed. All the formed pairing relationships are organized into a form, and data processing is performed as a basic control data set, including text verification, duplicate content elimination, etc. At the same time, for the situation where different personality analysis conclusions exist for the same behavior, all non-repeated isolated viewpoints are deleted. Then the LLM large model is trained using this data. In this embodiment, it is mainly based on the LLaMa model released by Meta Company for training, so as to obtain a basic large model that can input corresponding behaviors and judge and give personality types or personality characteristics conclusions.
[0049] Then we start acquiring and labeling multimodal data that reflects people's behavioral characteristics.
[0050] It mainly consists of image data and audio data. Image data includes single independent pictures and video, which are several image streams arranged in time. Relying on existing image recognition algorithms or models, it is possible to obtain the characteristics of people in images or video materials and the environmental characteristics that may cause people to behave accordingly.
[0051] Among them, human characteristics include body posture, movement and other characteristics, as well as facial expression changes. Human body posture and movement can reflect the reaction of people to different influences at this time, and facial expression changes can also reflect people's psychological activities from another perspective. For single or multiple independent pictures, most of them have no contextual relationship and are only analyzed and annotated as isolated samples. Then this static body posture and facial expression are mainly annotated with exaggerated expressions as screening materials, such as dancing after hearing positive news, raising the corners of the mouth and eyes, and anxious movements and expressions caused by hearing negative information. All of these can be feature extracted and annotated. That is, for the actions and facial expressions of a person in a picture, the corresponding area is framed after edge calculation by manual outlining or image recognition algorithm, and then the behavior description corresponding to the feature is manually written.
[0052] For video data, the image recognition algorithm is often used to mark the character area frame by frame. It is necessary not only to determine the static posture and facial expression in each frame, but also to determine the changing characteristics of body movements and facial expressions frame by frame. Then, the overall behavioral characteristics presented in the video data are annotated, and the annotation of body posture is used as the first behavioral label, and the facial expression annotation is used as the second behavioral label. At the same time, if there is no corresponding background description for the behavior of the person presented in the video data, it is impossible to confirm the influencing factors of the corresponding behavior, and different annotations may appear for similar behaviors. For example, the changes in facial expressions caused by pain and the changes in facial expressions caused by sadness may be exaggerated gathering of facial features for a period of time. If there is no background description provided for the video data, it can only be annotated by the environmental area other than the person in the video, such as the analysis and annotation of factors such as scenes, objects, and weather, so as to form a third behavioral label, that is, data reflecting the cause of the behavior, which is used as matching data with the behavioral characteristics as a behavior recognition training data set.
[0053] Furthermore, the image data in this embodiment can be divided according to the acquisition method, and can be divided into a preset acquisition scheme to guide the object to shoot the corresponding acquisition video according to a certain process, and a non-standard video data obtained by a non-standard acquisition scheme. The non-standard video data is usually directly collected from existing public video data, with pictures that can reflect the behavioral characteristics of people, but it is not a fixed factor that guides the generation of behavior, and the resolution, behavior type, and the presentation ratio and angle of the object in the picture are not fixed. Therefore, although the number of acquired videos obtained by the acquisition scheme is less than that of non-standard video data, and the authenticity of human behavioral reactions is lower than that of non-standard video data, the behaviors presented are more convenient for analysis and annotation.
[0054] Therefore, the first method is mainly adopted in this embodiment, that is, setting a standard collection plan, specifically setting a question-and-answer survey table, guiding the subject to answer the questions in sequence, and shooting the video at a fixed angle and ratio at the same time, so that the subject's complete body posture and facial expression can be obtained in the video screen, thereby facilitating overall feature extraction and annotation.
[0055] Specifically, a large amount of video-text paired data was collected and annotated by multiple people. Several fixed personality-related questions were preset and multiple people were asked to annotate the data. The questions were similar. The degree of openness of the person was analyzed through his speech, behavior and expression in the video. The score was assigned with 1 point as the lowest and 3 points as the highest. The results of the responses were similar. In the video, from the dimension of emotion, there were obvious positive emotions, such as happiness and excitement. From the dimension of tone, the tone of speech was higher. From the perspective of body language, some hand movements showed positive emotions, indicating that the subject was highly open and was assigned 3 points.
[0056] Let’s take a brief look at openness among the five basic dimensions of personality:
[0057] The personality trait of openness is characterized by imagination and insight, and people with this trait also tend to have a wide range of interests. They are curious about the world and others, eager to learn new things and enjoy new experiences. People high in this trait tend to be more adventurous and creative. People who lack this trait are often more traditional and may have difficulty with abstract thinking. When annotating videos, use the following content as a reference, mainly divided into judgment points with high openness and judgment points with low openness.
[0058] Highly open:
[0059] (1) Very creative
[0060] (2) Like to try new things
[0061] (3) Focus on meeting new challenges
[0062] (4) Prefer to think about abstract concepts
[0063] Low openness:
[0064] (1) Dislike of change
[0065] (2) Don’t like new things
[0066] (3) Resistance to new ideas
[0067] (4) Not very imaginative
[0068] (5) Dislike of abstract or theoretical concepts
[0069] Then, based on the above points, the scoring rules for video materials are as follows:
[0070]
[0071] The following is one of the labeled cases. High, Medium, and Low correspond to 3 points, 2 points, and 1 point respectively.
[0072] The video content is: Then our family is in the business of collecting gifts, and our parents are very tired. Then by the way, our family focuses on blueberries. Buying blueberries is too cheap, hahaha. Then there is something very practical and proactive, which is to buy a big ice cream for my mother. They," evaluate the people in the video based on five personality traits. The answer format is such as {Openness: high or medium or low} {Conscientiousness: high or medium or low} {Extraversion: high or medium or low} {[Agreeableness: high or medium or low}} {Neuroticism: high or medium or low}, summarize the results of the answers in a similar format, and give labels for the five personality traits."}.
[0073] Annotation result: "value":"Agreeableness: Medium, because the conversation content is the protagonist of the video introducing himself, his eyes are slightly erratic, his emotions fluctuate within a moderate range, and his tone is lively, so the agreeableness is medium.\nOpenness: High, because the conversation content is the protagonist of the video introducing himself, but through the sentence "Then our family also grows some blueberries", while introducing himself, he also promoted his own blueberries. His emotions during the conversation were positive and friendly, and his tone and loudness were moderate, so the openness was high.\nExtraversion: Medium, because the conversation content is the protagonist of the video introducing himself, Friendly, moderate tone and loudness, but occasionally reveal a more reserved expression, sometimes have more nervous hand movements, so the extroversion is medium. \n Conscientiousness: Medium, because the content of the conversation is the protagonist of the video introducing himself, mainly expressing his ambitions, the mood is relatively positive and friendly, the speech is relatively fluent, occasionally there will be emphasis, but the words are too casual, and they don't often look directly at the audience, so the conscientiousness is medium. \n Neuroticism: High, because the blogger uses more interjections during the conversation, has a certain emphasis when speaking, and has a lot of ups and downs in emotions and expressions, so the neuroticism is high. \n"
[0074] Then, clip is used to obtain the semantic features of pictures in low-resolution videos, and vitdet is used to obtain the image features of pictures in high-resolution videos. The high-resolution and low-resolution feature fusion module is used to retain the semantic features of low-resolution pictures and the image features of high-resolution pictures.
[0075] Using the imagebind method to extract speech features from the video, the audio data is converted into a spectrogram with a duration of 10 seconds, and we repeat and fill the spectrogram. For example, a 4-second spectrogram is repeated twice and then padded with zeros for an additional 2 seconds, and the text is converted into text features through the tokenizer.
[0076] For the high-resolution and low-resolution feature fusion modules, the fused image features are clustered, and high probability values are assigned to important parts and low probability values are assigned to unimportant parts. The image features are reduced in dimension to obtain the reduced-dimensional image features, and the reduced-dimensional video features, audio features and text features are combined and passed into the llama large model to generate personality-related response results.
[0077] Furthermore, this embodiment also provides a model, that is, an analysis model obtained by the above analysis method, which includes:
[0078] The acquisition module guides the user to generate the acquisition video;
[0079] An extraction module processes the collected video and obtains feature values; and
[0080] The matching module matches the corresponding personality analysis data according to the acquired feature values.
[0081] This embodiment also provides a readable storage medium on which a computer program is stored. When the program is executed by a processor, the multimodal personality analysis method as described above is implemented.
[0082] This embodiment also provides a device, which is loaded with the above-mentioned analysis model and is used to provide an interactive interface to guide the user to shoot and collect videos to analyze the user's personality analysis data.
[0083] In the several embodiments provided in the present application, it should be understood that the disclosed apparatus and method can be implemented in other ways. The device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as: multiple units or components can be combined, or can be integrated into another system, or some features can be ignored, or not executed. In addition, the coupling, direct coupling, or communication connection between the components shown or discussed can be through some interfaces, and the indirect coupling or communication connection of the devices or units can be electrical, mechanical or other forms.
[0084] The units described above as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units; they may be located in one place or distributed on multiple network units; some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.
[0085] This embodiment also provides a device with the model preset, which is generally a monitoring system with a camera. The target recognition model is set in a control module of the monitoring system, that is, in a computer, and is loaded, controlled and used through a computer program.
[0086] This embodiment further provides a storage medium on which a computer program is stored, wherein the program, when executed by a processor, implements any of the methods described above.
[0087] In addition, all functional units in the embodiments of the present invention may be integrated into one processing unit, or each unit may be separately used as a unit, or two or more units may be integrated into one unit; the above-mentioned integrated units may be implemented in the form of hardware or in the form of hardware plus software functional units.
[0088] Those skilled in the art can understand that all or part of the steps of implementing the above method embodiments can be completed by hardware related to program instructions, and the aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it executes the steps of the above method embodiments; and the aforementioned storage medium includes: mobile storage devices, read-only memories (ROM, Read Only Memory), random access memories (RAM, Random Access Memory), disks or optical disks, and other media that can store program codes.
[0089] Alternatively, if the above-mentioned integrated unit of the present invention is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiment of the present invention can be essentially or partly reflected in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the methods described in each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as mobile storage devices, ROM, RAM, magnetic disks or optical disks.
[0090] The present invention is not limited to the above optional implementations, and anyone can derive other various forms of products under the enlightenment of the present invention. The above specific implementations should not be understood as limiting the scope of protection of the present invention. The scope of protection of the present invention should be based on the definition in the claims, and the description can be used to interpret the claims.
Claims
1. A multimodal personality analysis method, which analyzes a user's personality data based on multiple input materials, characterized in that: G100. First, collect psychological data on personality analysis, pre-process the psychological data to form a basic control data set corresponding to behavior and personality analysis, and pre-train the LLM large model with the basic control data set to obtain a basic large model; G200. Then, a number of image data and audio data with human behaviors are collected as annotation reference templates, and feature values used to feedback human behaviors in the image data and audio data are analyzed and determined, and the determined feature values are annotated with behavior labels to form corresponding behavior recognition training data sets, and the basic large model is trained with the recognition training data set to obtain a personality analysis model; G300. Set a collection scheme, the collection scheme includes some guide information, guide the collection object to shoot the behavior collection video through the guide information, and obtain the feature value according to the collection video preprocessing; G400. The personality analysis model determines the corresponding behavioral characteristics based on the characteristic values, and then the personality analysis model matches the corresponding personality analysis data based on the behavioral characteristics to generate the corresponding personality analysis conclusion.
2. A multimodal personality analysis method according to claim 1, characterized in that: In step G100, the preprocessing of the psychology data is specifically as follows: Organize psychological data in different recording methods and form them into digital format, and then organize and summarize the digital data; The data contains personality analysis conclusions of several behaviors. First, the types of behaviors are classified to determine the number of all behaviors, and then personality analysis conclusions of at least not less than a set threshold number are obtained for each behavior; All personality analysis data corresponding to the same behavior are processed, and the repetition of the same behavior is determined through a semantic analysis model. According to a preset repetition threshold, the personality analysis conclusions that are greater than the repetition threshold are matched with the behavior to form a corresponding relationship, and a number of behaviors and the corresponding number of matching personality analysis conclusions are used as the basic control data set.
3. A multimodal personality analysis method according to claim 2, characterized in that: In step G100, the psychological data includes all public data with specific behaviors and corresponding personality analysis conclusions.
4. A multimodal personality analysis method according to claim 1, characterized in that: In step G200, the image data includes independent image data and image stream data at continuous time intervals.
5. A multimodal personality analysis method according to claim 1, characterized in that: In step G200, the image data includes a number of images and video materials collected from an undirected group of people using a set collection scheme, and also includes public images and video materials with human behavior.
6. A multimodal personality analysis method according to claim 1, characterized in that: In step G200: The process of determining the characteristic value of the behavior of the feedback person based on the image data first separates the human and environmental regions in the image data, and then captures the pixel features of the human body and face regions as characteristic values; Then, the posture presented by each body region is annotated to form a first behavior label, the expression of each facial region is annotated to form a second behavior label, and the environment region is annotated to form a third behavior label. The behavior label obtained in each image data is matched with the feature value to form a behavior recognition training data set; In the process of determining the characteristic value of the behavior of the feedback person based on the audio data, the human voice and background noise are first separated in the audio data, and the content and tone of the human voice are respectively determined as the characteristic values; then the content is annotated to form a first behavior label, the change of tone is annotated to form a second behavior label, and the background noise is annotated to form a third behavior label. The behavior labels obtained in the audio data are matched with the characteristic values to form a behavior recognition training data set.
7. A multimodal personality analysis method according to claim 1, characterized in that: The specific steps of preprocessing the captured video to obtain the characteristic value in step G300 are as follows: First, the feature values are processed by using the Clip model to obtain semantic features in low-resolution videos, and the Vitdet model to obtain image features in high-resolution videos. The semantic features and image features are combined through the feature fusion model, and the image features are clustered and assigned probability values according to their importance, and the image features are reduced in dimension. Then, the audio features in the captured video are extracted through the multimodal fusion model of Imagebind, and the text content in the audio features is converted into text features through Tokenizer; The reduced-dimensional image features, audio features, and text features are used as feature values.
8. An analysis model, characterized in that: The method for multimodal personality analysis as claimed in any one of claims 1 to 7 comprises: The acquisition module guides the user to generate the acquisition video; An extraction module processes the collected video and obtains feature values; and The matching module matches the corresponding personality analysis data according to the acquired feature values.
9. A readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the multimodal personality analysis method as described in any one of claims 1 to 7 is implemented.
10. A device, characterized in that: The analysis model in claim 8 is loaded, and is used to provide an interactive interface to guide users to shoot and collect videos to analyze the user's personality analysis data.
Citation Information
Patent Citations
Psychological test method based on face images and eye movement fixation information
CN106407935A
Multi-modal psychological knowledge graph construction method and device, equipment and storage medium
CN116910270A
Psychological analysis method based on certifying psychology and mapping knowledge domain
CN118121198A
Cited By
Large model training method and system fusing psychological knowledge and multi-modal data
CN120849893A
Method and system for training a large model fusing psychological knowledge and multi-modal data
CN120849893B
