Cognitive analysis method based on multi-modal fusion

Abstract

The invention provides a cognitive analysis method based on multi-modal fusion, and the method comprises the steps: an input layer, which collects audio data, video data, image data and text data related to a user in a social platform, and carries out the vectorization of the data to generate vector information; in the multi-dimensional cognitive space layer, a TransH model containing a plurality of hyperplanes is created, one hyperplane corresponds to one cognitive layer, and vector information of users in the social platform is mapped to each hyperplane so as to generate score functions of which the number is equal to that of the hyperplanes; and analyzing the clustering layer, determining the similarity between the score functions of the users in each hyperplane based on a clustering strategy, clustering the users based on the similarity, and outputting a clustering set. According to the method, multi-cognitive-perspective analysis can be carried out in combination with multi-modal information, and the cognitive condition of the user is efficiently and accurately determined, so that the information is continuously and pertinently delivered to the user, and the probability of flow diffusion of malicious hotspots is reduced.

Description

Technical Field

[0001] The present invention relates to the technical field of cognitive analysis of users in a social platform, and in particular to a cognitive analysis method based on multimodal fusion. Background Art

[0002] With the rapid development of the Internet and the widespread use of social media platforms, people's lifestyles and information exchange patterns have undergone fundamental changes. A large number of people have transferred information sharing and opinion interaction to social platforms, and people express their ideas and wishes in the form of likes, comments, uploads or forwarding data, making the above data contain rich social information and user information.

[0003] When dealing with public events or crises, by tracking the emotional changes of user groups in social media in real time, we can more accurately understand the public's hot spots, make timely adjustments when abnormal hot spots are found, and take measures in advance to push accurate information to users and reduce the risk of blindly following hot spots. Social media sentiment analysis can contribute to the harmony and stability of society. With the advancement of technology and the deepening of application, it will play a more important role in the future.

[0004] However, the knowledge reserves, social emotions, individual emotions, moral concepts, individual experiences, individual psychology, individual preferences, and value orientations of different users on social platforms are not exactly the same. When the same guidance information pushed to users related to hot spots is used to adjust the hot spot orientation, due to user cognitive biases, it is easy for a large number of users to not recognize the guidance information, have no interest in reading the guidance information, or cannot understand the guidance information, resulting in inefficient risk avoidance. Summary of the invention

[0005] In view of this, the problem to be solved by the present invention is to provide a cognitive analysis method based on multimodal fusion, which can combine multimodal information to perform multi-cognitive angle analysis, efficiently and accurately determine the user's cognitive situation, so as to continue to deliver information to the user in a targeted manner and reduce the probability of traffic diffusion of malicious hotspots.

[0006] In order to solve the above technical problems, the technical solution adopted by the present invention is:

[0007] A cognitive analysis method based on multimodal fusion includes an input layer, which collects audio data, video data, image data and text data related to users in a social platform and vectorizes them to generate vector information;

[0008] Multi-dimensional cognitive space layer: Create a TransH model containing several hyperplanes. One hyperplane corresponds to one cognitive level. The vector information of users in the social platform is mapped to each hyperplane to generate a number of scoring functions equal to the number of hyperplanes.

[0009] Analyze the clustering layer, determine the similarity between the score functions of users in each hyperplane based on the clustering strategy, cluster the users based on the similarity and output the cluster set.

[0010] Furthermore, the information delivery layer obtains the clustering sets of users under different hyperplanes to determine the situations of users at different cognitive levels and delivers targeted information to users.

[0011] Furthermore, the content liked, commented, uploaded or forwarded by the user is defined as being related to the user.

[0012] Furthermore, each of the hyperplanes includes a corresponding transformation matrix M r_shqg ∈R k·d , where k and d are the dimensions of the mapped vector, and the vector information includes the head entity v h and tail entity v t , based on the transformation matrix M r_shqg The head entity v h and tail entity v t Converted into the head node mapping feature in the TransH model and the tail node mapping features in the TransH model and The score function of the user's triplet is defined as: where r _shqg It is the relation mapping feature in the TransH model.

[0013] Furthermore, the clustering strategy includes: calculating the distance or similarity of the vectors in the score function of different users, classifying users with close distances or similarities exceeding a threshold into one category and outputting a cognitive combination: in represents the set of similar users in the cognitive space of value orientation, and G represents a graph.

[0014] Furthermore, the similarity calculation method includes: using cosine similarity to calculate the mapping features of the head nodes in the two score functions Tail node mapping features and relation mapping feature r _shqg The similarity.

[0015] Furthermore, the video data is vectorized through the MSAN model.

[0016] Furthermore, the audio data is vectorized through the FunAudio LLM model.

[0017] Furthermore, the image data is vectorized through a clip model.

[0018] Furthermore, the text data is vectorized through an LLM model.

[0019] The advantages and positive effects of the present invention are:

[0020] The text, audio, image and text information related to the user are vectorized to generate vector information. Several hyperplanes are created in the TransH model. Different planes correspond to different cognitive levels. The user's vectorized information is mapped to different hyperplanes. According to the similarity of the user's vectors in different hyperplanes, the users are clustered at different cognitive levels in order to accurately obtain the situation of different users at different cognitive levels. The information delivery layer obtains the clustering results of users at different cognitive levels, and delivers relevant information in a targeted manner to improve the effective dissemination efficiency of relevant information. The clustering results at multiple cognitive levels also provide an interpretable angle and dimension for the analysis results of the information delivery layer, and provide ideas for the customization of downstream tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings:

[0022] Figure 1 It is a schematic diagram of the Trans H algorithm;

[0023] Figure 2 It is an overall principle diagram of a cognitive analysis method based on multimodal fusion of the present invention. DETAILED DESCRIPTION

[0024] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0025] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art of the present invention. The terms used herein in the specification of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. The term "and / or" used herein includes any and all combinations of one or more related listed items.

[0026] The present invention provides a cognitive analysis method based on multimodal fusion, such as Figure 1As shown in the figure, it is a schematic diagram of the Trans H algorithm. In the figure, h and t represent the head entity and tail entity of the user respectively. The head entity and the tail entity are mapped to the hyperplane to generate the mapping vectors hr and tr. Based on hr and tr, the relationship vector dr can be obtained. The user's triple is (hr, dr, tr). By judging the similarity of the triplets of different users, it can be determined whether two users are similar. The core of the TransH model is that for each relationship, comparison and calculation are performed on a specific hyperplane, which has more interpretability in dealing with reflexive relationships, as well as one-to-many, many-to-one, and many-to-many relationships.

[0027] like Figure 2 As shown, the system includes an input layer that collects audio data, video data, image data, and text data related to users in the social platform and vectorizes them to generate vector information.

[0028] The input layer receives text data, image data, audio data, and video data that users like, forward, upload, and comment on social platforms, and uses different feature extraction models to vectorize the above data to output corresponding vector matrices. The vector matrices corresponding to the users together constitute vector information.

[0029] After the text data is input into the clip model and personalized tuning is performed, a vector matrix representing text features is output. After the image data is input into the LLM framework and personalized tuning is performed, a vector matrix representing image features is output. After the audio data is input into the FunAudio LLM model and personalized tuning is performed, a vector matrix representing audio features is output. After the video data is input into the MSAN model and personalized tuning is performed, a vector matrix representing video features is output.

[0030] Multi-dimensional cognitive space layer: Create a TransH model containing several hyperplanes. One hyperplane corresponds to one cognitive level. The vector information of users in the social platform is mapped to each hyperplane to generate a number of scoring functions equal to the number of hyperplanes.

[0031] Complete the training of the TransH model in advance and give a hyperplane W for each cognitive level r During the training of the TransH model, each hyperplane W r There is a transformation matrix M r_shqg ∈R kd , where k and d are the dimensions of the mapped vector.

[0032] After the user's vector information is input into the TransH model, the transformation matrix M r_shqg Map the user's vector information to the corresponding hyperplane W r , to obtain the user's head node mapping features Tail node mapping features and relation mapping feature r _shqg ,and Construct the score function of user triples based on the above mapping features:

[0033]

[0034] One embodiment of the present application is: the cognitive level includes eight aspects: knowledge reserve, social emotion, individual emotion, moral concept, individual experience, individual psychology, individual preference, and value orientation. Therefore, the TransH model includes eight hyperplanes, and each hyperplane corresponds to a transformation matrix. If the social platform contains one million users, each user corresponds to a vector information, and one million vector information are mapped to the corresponding eight hyperplanes through eight transformation matrices, so that each user corresponds to eight score functions.

[0035] Analyze the clustering layer, determine the similarity of the score functions of users on the social platform in each hyperplane based on the clustering strategy, and cluster the users based on the similarity.

[0036] Taking the hyperplane corresponding to the knowledge reserve as an example, the clustering strategy includes: obtaining the score function of the user in the hyperplane, calculating the feature vectors in two different score functions (head node mapping feature Tail node mapping features and relation mapping feature r _shqg ) to determine whether the similarity exceeds the threshold. If yes, the two users are similar; if the similarity does not exceed the threshold, the two users are not similar. Repeat the above method until all users are clustered and several cognitive combinations are output:

[0037]

[0038] in represents a set of similar users in the cognitive space of value orientation, and G represents a graph. Based on the clustering strategy, users are clustered in different hyperplanes to obtain the cognitive combinations of all users in different hyperplanes. An embodiment of the present application is: a method for calculating the similarity of a score function includes: calculating the mapping features of the head nodes in two score functions by cosine similarity Tail node mapping features and relation mapping feature r _shqg The similarity.

[0039] The information delivery layer obtains the clustering sets of users under different hyperplanes to determine the situations of users at different cognitive levels and delivers targeted information to users.

[0040] The information delivery layer receives cognitive combinations of different hyperplanes and can quickly determine the situations of different users at different cognitive levels. Based on the user's cognitive combination and the hyperplane corresponding to the cognitive combination, the information delivery layer can accurately and quickly determine the user's cognitive situation and preferences, so as to accurately push relevant information to the user and improve the effective dissemination efficiency of the pushed information among users. Or locate a group of people with the same cognitive characteristics.

[0041] One embodiment of the present application is: for video data, the feature vector of the video content is usually extracted through the following steps: extracting representative image frames from the video; calculating the feature vectors of the extracted image frames; calculating the eigenvalues ​​of the matrix composed of the feature vectors of the extracted image frames, and these eigenvalues ​​constitute the eigenvectors of the video; performing an orthogonal transformation on the video eigenvectors to obtain the video vectors.

[0042] In order to further improve the fine-grained information in video features, the MSAN model introduces three features, namely frame images, video clips, and optical flow, and their corresponding semantic attributes to represent videos. Frame images provide static semantic information, while optical flow and clips provide dynamic information, which makes the attributes mined from different modes of the video complement each other in sentence generation. Expanding each weight matrix of the traditional RNN in the MSAN model to a set of weight matrices related to the attributes can integrate the multimodal information contained in the video and improve the efficiency of information extraction.

[0043] The embodiments of the present invention are described in detail above, but the contents described are only preferred embodiments of the present invention and cannot be considered to limit the scope of implementation of the present invention. All equivalent changes and improvements made within the scope of the present invention should still fall within the scope of this patent.

Claims

1. A cognitive analysis method based on multimodal fusion, characterized in that: include, The input layer collects audio data, video data, image data, and text data related to users on the social platform and vectorizes them to generate vector information; Multi-dimensional cognitive space layer: Create a TransH model containing several hyperplanes. One hyperplane corresponds to one cognitive level. The vector information of users in the social platform is mapped to each hyperplane to generate a number of scoring functions equal to the number of hyperplanes. Analyze the clustering layer, determine the similarity between the score functions of users in each hyperplane based on the clustering strategy, cluster the users based on the similarity and output the cluster set.

2. The cognitive analysis method based on multimodal fusion according to claim 1, characterized in that: The information delivery layer obtains the clustering sets of users under different hyperplanes to determine the situations of users at different cognitive levels and delivers targeted information to users.

3. The cognitive analysis method based on multimodal fusion according to claim 1, characterized in that: The content liked, commented, uploaded or forwarded by the user is defined as being related to the user.

4. The cognitive analysis method based on multimodal fusion according to claim 1, characterized in that: Each of the hyperplanes includes a corresponding transformation matrix M r_shqg ∈R k·d , where k and d are the dimensions of the mapped vector, and the vector information includes the head entity v h and tail entity v t , based on the transformation matrix M r_shqg The head entity v h and tail entity v t Converted into the head node mapping feature in the TransH model and the tail node mapping features in the TransH model and The score function of the user's triplet is defined as: where r _shqg It is the relation mapping feature in the TransH model.

5. The cognitive analysis method based on multimodal fusion according to claim 1, characterized in that: The clustering strategy includes: calculating the distance or similarity of the vectors in the score function of different users, classifying users with close distances or similarities exceeding a threshold into one category and outputting a cognitive combination: in represents the set of similar users in the cognitive space of value orientation, and G represents a graph.

6. The cognitive analysis method based on multimodal fusion according to claim 5 is characterized in that: The similarity calculation method includes: using cosine similarity to calculate the mapping features of the head nodes in the two score functions Tail node mapping features and relation mapping feature r _shqg The similarity.

7. The cognitive analysis method based on multimodal fusion according to claim 1 is characterized in that: The video data is vectorized through the MSAN model.

8. The cognitive analysis method based on multimodal fusion according to claim 1, characterized in that: The audio data is vectorized by the FunAudio LLM model.

9. The cognitive analysis method based on multimodal fusion according to claim 1, characterized in that: The image data is vectorized through the clip model.

10. The cognitive analysis method based on multimodal fusion according to claim 1, characterized in that: The text data is vectorized through the LLM model.

Images