Group behavior analysis method for multi-modal information fusion and dynamic updating

Abstract

The invention discloses a multi-modal information fusion and dynamic updating group behavior analysis method. The method comprises the steps of information integration and dynamic updating, multi-modal information obtaining and information preprocessing, feature extraction, feature representation conversion, alignment and fusion, and multi-modal information real-time processing and dynamic updating through a stream processing framework. A self-learning mechanism and a self-optimization mechanism are introduced; group portrait generation: extracting population statistical characteristics, geographic position distribution, behavior and emotional psychological characteristics, social relationships, social media and economic characteristics from the multi-modal data to generate a group portrait; analyzing group behaviors; and outputting visualization, and outputting group behavior prediction results in a visualized manner, including a behavior trend graph, a group clustering graph, an association rule graph and an abnormal behavior detection graph. According to the method, multi-modal data from various sources and forms are effectively integrated and analyzed, self-learning and self-optimization mechanisms are introduced, and finally group behavior analysis is performed.

Description

Technical Field

[0001] The present invention belongs to the technical field of group portraits, and specifically relates to a group behavior analysis method of multimodal information fusion and dynamic updating. Background Art

[0002] In the modern information society, group profiling has become an important tool for understanding and analyzing the behavior, characteristics and motivations of specific groups. Group profiling technology is widely used in market research, social media analysis, security intelligence, public policy formulation and other fields. Through group profiling, organizations and institutions can gain a deeper understanding of the internal structure, behavior patterns, demographic characteristics and psychological characteristics of specific groups in order to formulate more effective strategies and decisions.

[0003] Natural language processing (NLP) and social network analysis (SNA) are important technical means to construct group portraits. NLP technology can extract useful information from unstructured text data, such as sentiment tendencies, topic concerns, and opinions; while SNA technology can reveal the relationship network, information dissemination path, key figures, etc. within the group. However, these existing technologies rely on a single data type and may not be able to fully capture the complexity and diversity of group behavior, nor can they fully utilize information from multiple data sources. They lack the ability to conduct comprehensive analysis and find it difficult to fully capture the diverse behavior patterns and emotional expressions of the group. In addition, they show low robustness when faced with data quality issues (such as noise and missing data), and the accuracy of predictions and decisions may be limited. They cannot fully utilize the information in multimodal data to improve the accuracy of analysis.

[0004] The data of a group usually comes from multiple sources and forms, including structured data (such as demographic information), unstructured data (such as text content), semi-structured data (such as log files), and multimedia data (such as images, audio, and video). How to effectively integrate these multimodal data to form a comprehensive and accurate group portrait is the focus of current technological development. Summary of the invention

[0005] The main purpose of the present invention is to overcome the shortcomings and deficiencies of the prior art and to propose a group behavior analysis method with multimodal information fusion and dynamic updating.

[0006] In order to achieve the above object, the present invention adopts the following technical solutions:

[0007] A group behavior analysis method for multimodal information fusion and dynamic updating, comprising:

[0008] Information integration and dynamic update: acquiring multimodal information and preprocessing the information, extracting features, performing representation conversion, alignment and fusion on features, and using a stream processing framework for real-time processing and dynamic update of multimodal information;

[0009] Introduce self-learning and self-optimization mechanisms;

[0010] Group portrait generation: extracting demographic characteristics, geographic location distribution, behavioral and emotional psychological characteristics, social relations, social media, and economic characteristics from multimodal data to generate group portraits;

[0011] Group behavior analysis;

[0012] Output visualization: Output group behavior prediction results in a visual way, including behavior trend graphs, group clustering graphs, association rule graphs, and abnormal behavior detection graphs.

[0013] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0014] 1. The method of the present invention is based on multimodal data integration, self-learning and adaptive optimization mechanism. On the basis of feature extraction, representation and alignment of multiple modal data such as text, image, audio and video, it combines technical means such as feature splicing, weighted summation and PCA dimensionality reduction to deeply integrate multimodal data; through the stream processing framework Apache Flink, real-time processing and dynamic update of multimodal data are realized, and the BERT model is used for sentiment analysis, and K-means clustering and LSTM time series analysis are used for group behavior prediction, so as to provide reliable decision support for marketing, public safety, network security and other fields.

[0015] 2. The present invention provides a solution for feature extraction and unified representation of multimodal data such as text, image, audio and video. The prior art lacks a unified feature representation method, resulting in insufficient data fusion. The present invention uses models such as BERT, CNN, MFCC and LSTM to perform feature extraction and unified normalization to achieve deep fusion, significantly improving the accuracy and depth of data integration.

[0016] 3. The present invention adopts the Apache Flink stream processing framework to realize real-time processing and dynamic updating of multimodal data. The prior art usually adopts a batch processing method, which makes it difficult to reflect the latest data changes in real time. The present invention uses a stream processing framework to quickly respond to data changes and maintain the real-time and accuracy of analysis.

[0017] 4. The present invention has the ability to automatically label new data, self-update models, learn feedback mechanisms, optimize hyperparameters, optimize model architectures, and optimize computing resources; existing technologies are rarely used in self-learning and self-optimization and require manual intervention; the present invention achieves highly intelligent and dynamically adaptive data analysis through technologies such as online gradient descent, genetic algorithms, and neural architecture search. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 It is the overall flow chart of the present invention;

[0019] Figure 2 is the information integration and dynamic update flow chart of the present invention;

[0020] Figure 3 It is a self-learning and self-optimization mechanism analysis flow chart of the present invention;

[0021] Figure 4 It is a flow chart of group portrait generation and group behavior analysis of the present invention. DETAILED DESCRIPTION

[0022] The present invention is further described in detail below in conjunction with embodiments and drawings, but the embodiments of the present invention are not limited thereto.

[0023] Example

[0024] like Figure 1 As shown, the present invention is a group behavior analysis method of multimodal information fusion and dynamic update, comprising:

[0025] S1, information integration and dynamic update, obtain multimodal information and preprocess the information, extract features, perform representation conversion, alignment and fusion on features, and use stream processing framework for real-time processing and dynamic update of multimodal information;

[0026] like Figure 2 As shown, in this embodiment, multimodal information is obtained and preprocessed, specifically:

[0027] Collect multimodal data from multiple data sources, including text, images, audio and video; text data is in T i Input in the form of I i The audio data is input in the form of A i Input in the form of V i Form input;

[0028] Clean and standardize different types of data, and the processing process of text data is as follows:

[0029] The text is segmented and stop words are removed, special tags are added, and words are converted to corresponding IDs. Then, an attention mask is generated, that is, a vector with the same length as the word ID sequence is generated to indicate which positions are valid words and which are fillers. Finally, a segment ID is generated to mark the words belonging to the first sentence and the words belonging to the second sentence. Finally, three vectors are output: T i id =(id1, id2…id sequence_length ), attention mask and segment ID; where T i idThe generated vector has the same length as the word ID sequence, and sequence_length is the length of the word sequence;

[0030] The image data processing process is:

[0031] Resize the image to an output shape of H×W×C, where H and W are the sizes of the target image and C is the number of channels;

[0032] According to the formula:

[0033]

[0034] The adjusted image is normalized, where is the image after resizing the image, μ is the mean of the image dataset, σ is the standard deviation, and the final output is in the following matrix:

[0035]

[0036] The audio data processing process is:

[0037] Use fast Fourier transform to get the audio spectrum according to the following formula:

[0038]

[0039] in, is the audio signal in the frequency domain, f is the frequency, and N is the number of sampling points;

[0040] The spectral subtraction method is used to remove noise using the following formula:

[0041]

[0042] in, is the noise spectrum estimated in the silent part;

[0043] Finally get The form is as follows:

[0044] A i denoised =(a1 denoised , a2 denoised …a n denoised )

[0045] Each element corresponds to the amplitude value of the denoised audio signal at time k, and n represents the total length of the denoised audio signal;

[0046] The video data processing process is as follows:

[0047] Decode the video, decode the video stream into multiple independent frames, and output the frame sequence V i decoded , the shape is N×H×W×C, where N is the number of frames, H and W are the height and width of the frame, and C is the number of color channels;

[0048] Frame sampling, sampling key frames from the frame sequence, and outputting the sampled frame sequence V i sampled ;

[0049] Perform frame preprocessing, resize and normalize each frame to make it suitable for model input;

[0050] The final output is the preprocessed frame sequence V i preprocessed , the output matrix is ​​as follows:

[0051]

[0052] In this embodiment, the text features are extracted as follows:

[0053] Use natural language processing (NLP) to extract semantic features from text data. The processed text is input into the BERT model. The input of BERT includes word ID, attention mask and segment ID. The input word ID sequence T i id First, it is transformed into a fixed-dimensional word vector through the embedding layer, with a shape of E∈R sequence_length×hidden_size , hidden_size is the hidden layer dimension; then each input vector is calculated through the multi-head self-attention mechanism in the model to calculate its dependency with other positions in the sequence. The formula is as follows:

[0054]

[0055] Among them, Q, K, and V are the matrices of query, key, and value, respectively, obtained through different linear transformations. k is the scaling factor, and then concatenates the outputs of multiple self-attentions together:

[0056] MultiHead(Q,K,V)=Concat(head1,…,head h )W O

[0057] Among them, Concat() is the concatenation operation function, head i =Attention(Q i ,K i ,V i ), and finally the vector of each position is passed through a feedforward neural network to further process its semantic information:

[0058] FFN(x)=max(0,xW1+b1)W2+b2

[0059] Among them, W1, W2 are learnable weight matrices, b1, b2 are bias terms, and x is the input of the previous step;

[0060] The output of each layer is residually connected to its input and the layer is normalized. The formula is as follows:

[0061] LayerNorm(x+SubLayer(x))

[0062] The final output is two vectors. The first is the sequence feature matrix, a three-dimensional tensor that provides the vector representation of each word in the sequence. The second is the sentence feature vector, a two-dimensional tensor that provides the overall vector representation of the entire sequence.

[0063] The specific image features are extracted as follows:

[0064] The convolutional neural network (CNN) deep learning method is used to extract visual features from image data. The preprocessed image is input and the convolution operation is performed first:

[0065]

[0066] Among them, F ij is a pixel value of the convolution feature map, W mnc is the weight of the convolution kernel, b is the bias term, H f and W f are the height and width of the convolution kernel, respectively, and C is the number of channels of the input image;

[0067] Output convolution feature map F, shape is Among them, N f is the number of convolution kernels, representing the depth of the output feature map, that is, the number of channels;

[0068] Through the ReLU nonlinear activation function, nonlinear features are introduced:

[0069] A ij =ReLU(F ij )=max(0,F ij )

[0070] Among them, A ij It is a pixel value in the activated feature map. The shape of the activated feature map A is the same as that of the convolution feature map.

[0071] Use maximum pooling to downsample the feature map to reduce the spatial dimension of the feature map while retaining important feature information:

[0072] P ij =max(A(i,j),A(i+1,j),A(i,j+1),A(i+1,j+1))

[0073] The feature map P after pooling has the shape of Where H″×W″ is the height and width after pooling;

[0074] After multiple layers of convolution, activation, and pooling, the extracted feature map is usually flattened into a one-dimensional vector, namely P flat = Flatten(P) and input to the fully connected layer to further combine and extract global features:

[0075] z=W fc ·P flat +b fc

[0076] Among them, W fc is the weight matrix of the fully connected layer, b fc is the bias term;

[0077] The feature vector of the final output image;

[0078] The specific audio features extracted are:

[0079] Mel frequency cepstral coefficients (MFCC) are used to extract sound features from audio data, and the processed audio is subjected to short-time Fourier transform:

[0080]

[0081] Where m is the frame index, f is the frequency, N is the number of sampling points per frame, hop_size is the frame shift, and the output frequency domain signal is The shape is a two-dimensional matrix, containing the spectrum information of multiple frames;

[0082] Apply Mel frequency scaling:

[0083]

[0084] Where f0 is the normal frequency, f mel is the Mel frequency; output Mel spectrum A i mel (m,f mel ), which represents the spectral information of the audio signal on the Mel scale;

[0085] Compute the Mel-frequency cepstral coefficients:

[0086]

[0087] Where c is the coefficient index of MFCC, taking the first 13 coefficients, that is, c = 13;

[0088] The final extracted MFCC features are represented as a two-dimensional array;

[0089] The specific video features are extracted as follows:

[0090] Combining image feature extraction and action recognition technology, key frame features and dynamic information are extracted from video data. The processed video is firstly subjected to static feature extraction using CNN to obtain the static feature vector z of each frame, with the shape z∈R feature _size , and then use LSTM for dynamic feature extraction. For each time step, LSTM processes the input feature z in turn t , combined with the hidden state h of the previous time step t-1 and cell state C t-1 , generate the hidden state h of the current time step t and cell state C t Each time step is processed as follows:

[0091]

[0092] h t =o t tanh(C t )

[0093] Finally, static features and dynamic features are obtained.

[0094] In this embodiment, the features are represented, transformed, aligned and fused, specifically as follows:

[0095] Convert the extracted features into a unified vector representation to facilitate comparison and fusion between different modality data;

[0096] Through timestamps, event markers or other alignment methods, data of different modalities are synchronized in time and space to ensure the relevance between data; in terms of time, the modalities with time dimension, namely video and audio, are aligned according to the minimum time step; in terms of space, alignment is performed based on spatial anchor points; and zero mean unit variance normalization is used:

[0097]

[0098] Among them, μ X is the mean of the eigenvectors, σ X is the standard deviation;

[0099] The final output is the normalized feature vector T norm ,I norm , A norm 、Vnorm ; T norm ,I norm , A norm 、V norm are the eigenvectors corresponding to the four modes output after normalization with zero mean and unit variance;

[0100] The specific feature fusion is:

[0101] The feature vectors of different modes are spliced ​​together to form a comprehensive feature vector F concat ∈R batch _ size×(dimT+dimI+dimA+dimV) , after concatenation, the dimension of the new feature vector is the sum of the dimensions of each modality; batch_size is the number of data samples processed during each training or inference, and the specific value depends on the actual situation; dimT, dimI, dimA, and dimV mean the dimensions of the feature vectors of the four modalities;

[0102] According to the importance of different modes, each eigenvector is weighted and summed:

[0103] F weighted =w T ·T norm +w I I norm +w A ·A norm +w V ·V norm

[0104] The weighted comprehensive feature vector F weighted ∈R batch_size×(dimT+dimI+dimA+dimV) , forming comprehensive features, and the selection of weights is determined based on experience and usage scenarios;

[0105] Finally, we perform dimensionality reduction and use PCA to reduce the dimensionality of high-dimensional features to reduce computational complexity and noise. First, we need to standardize the input data:

[0106]

[0107] Among them, μ is the mean of each feature and σ is the standard deviation; the standardized data is used to calculate the covariance matrix, which describes the linear correlation between the features:

[0108]

[0109] Output the covariance matrix C, perform eigenvalue decomposition on the covariance matrix, and obtain the eigenvalues ​​and corresponding eigenvectors. The eigenvalues ​​represent the variance of the data in the direction of each eigenvector:

[0110] Cv i =λi v i

[0111] Among them, λ i is the ith eigenvalue, v i is the corresponding eigenvector;

[0112] Output eigenvalue vector λ = [λ1,λ2,…,λ dim ] and the eigenvector matrix V∈R dim×dim , sort by the size of the eigenvalues, and select the eigenvectors corresponding to the first n largest eigenvalues ​​as the principal components:

[0113] V pca =[v1,v2,…,v n ]

[0114] Among them, n is the number of principal components selected, which is determined by the cumulative variance contribution rate;

[0115] Project the standardized data onto the selected principal component space to obtain the feature representation after dimensionality reduction:

[0116] F pca =F std V pca

[0117] Finally, a matrix of low-dimensional feature representation is output, which contains the representation of the original high-dimensional data projected onto the principal component direction; each sample data has a projection value in these principal component directions, and these projection values ​​constitute the new feature F after dimensionality reduction. pca ∈R batch _ size×n , where n is the target dimension after dimensionality reduction.

[0118] In this embodiment, a stream processing framework is used to perform real-time processing and dynamic updating of multimodal information, specifically:

[0119] Use the stream processing framework Apache Flink for real-time processing and dynamic updating of multimodal information;

[0120] Flink is used to perform real-time cleaning and feature extraction on the incoming data stream. Assuming that the weight of the model is W(t), the dynamically updated model weight is expressed as:

[0121]

[0122] Flink is used to perform real-time aggregation and analysis of data streams. Flink aggregates data streams within a specific time window through window operations. The aggregation characteristics within the time window [t, t+Δt] are expressed as:

[0123]

[0124] Generate real-time analysis results by classifying, clustering or other analysis operations on the aggregated features;

[0125] Flink iterative operations continuously optimize the model or processing strategy. Based on the feedback of analysis results, the model parameters are updated or the feature extraction method is adjusted to make the system perform better when processing the next batch of data. The iterative update process is expressed as:

[0126]

[0127] in, is the feature after the kth iteration, and α is the update step size.

[0128] S2, introduce self-learning mechanism and self-optimization mechanism;

[0129] like Figure 3 As shown, in this embodiment, introducing the self-learning mechanism specifically includes:

[0130] Automatically label new data, use the trained BERT model, CNN and LSTM to label and infer new data, and automatically generate labels or classification results; output automatically generated labels or classification results;

[0131] Self-update of the model, specifically:

[0132] The model parameters are updated through the online gradient descent method when new data arrives to adapt to the dynamic changes of data; when new text data arrives, BERT's online learning extension is used to update its weights; input new data X new and its true label y new And the current model parameter W, online gradient descent update formula:

[0133]

[0134] Where η is the learning rate, Is the gradient of the loss function with respect to the model parameters. Output the updated model parameters W new ;

[0135] Learning feedback mechanism, specifically:

[0136] Compare the predicted labels with the true labels, adjust the adaptive learning rate, adaptive loss function weight, and finally obtain the adjusted learning rate, loss function or other hyperparameters;

[0137] Input model prediction label and the true label y true , adjust the learning rate and loss function weight according to the prediction error, and the error calculation formula is:

[0138]

[0139] The learning rate adjustment formula is:

[0140] η new =η old ·(1-α·ε 2 )

[0141] Among them, α is the adjustment coefficient;

[0142] Adjust the loss function weight:

[0143] λ new =λ old +β·ε

[0144] Among them, β is the adjustment coefficient;

[0145] Output adjusted learning rate η new and loss function weight λ new .

[0146] In this embodiment, introducing the self-optimization mechanism specifically includes:

[0147] Hyperparameter optimization: The goal of hyperparameter optimization is to find the hyperparameter combination that can achieve the best model performance in a given hyperparameter space;

[0148] Define the hyperparameters to be optimized and their possible value ranges {θ1,θ2,...,θ n}, use genetic algorithms to simulate natural evolution to select optimal parameters, repeat the selection, crossover, and mutation process until convergence conditions are reached or the preset maximum number of generations is reached; the optimal hyperparameter configuration is finally determined for use by the BERT model, CNN, and LSTM;

[0149] Model architecture optimization, specifically:

[0150] Input the initial neural network architecture design and the data set for training and verification; define the structural parameters of the neural network, and select the neural network architecture A randomly or through a strategy from the search space S i To conduct an assessment:

[0151]

[0152] in, Based on architecture A i The model, G(A i ) is the average loss on the validation set; after multiple sampling and evaluation, the architecture A with the best performance on the validation set is selected * As the final optimal architecture;

[0153] Computing resource optimization, specifically:

[0154] First, input the current computing resource status and task requirements, including CPU and GPU utilization, memory usage, and task priority; use task scheduling algorithms and resource monitoring and allocation to adjust resource allocation; monitor resource usage in real time and dynamically adjust resource allocation to respond to system changes and task execution; finally output optimized resource allocation and task scheduling strategies to improve the overall efficiency and response speed of the system.

[0155] S3, group portrait generation, extracting demographic characteristics, geographic location distribution, behavioral and emotional psychological characteristics, social relations, social media and economic characteristics from multimodal data to generate group portraits;

[0156] like Figure 4 As shown, in this embodiment, extracting demographic features is specifically as follows:

[0157] Use the NER model based on conditional random fields to extract information including but not limited to age, gender, and occupation from text data. The formula is as follows:

[0158]

[0159] Among them, y is the label sequence, that is, the extracted demographic features; x is the input sequence, that is, the text data; fk(y,x) is the feature function, λ k is the weight, Z(x) is the normalization factor to ensure the normalization of the probability distribution;

[0160] The extracted demographic features generate a demographic feature vector D i , containing the numerical values ​​of various information;

[0161] Extract geographic location distribution, specifically:

[0162] Use geocoding technology to convert the location described in the text into geographic coordinates, use the K-means clustering method to calculate the minimum intra-cluster sum of squares WCSS, and identify the main activity areas; finally, visualize the geographical distribution through heat maps to show the distribution density of groups in different regions;

[0163] Extract behavioral and emotional psychological features, specifically:

[0164] Use time series analysis and frequent pattern mining techniques to analyze user behavior, and use the autoregressive integrated moving average model ARIMA to analyze and predict user behavior patterns:

[0165] x t =c+φ1x t-1 +φ2x t-2 +...+φp x t-p +∈ t +θ1∈ t-1 +...+θ q ∈ t-q

[0166] Among them, c is a constant term, φ p is the autoregressive coefficient, θ q is the moving average coefficient, ∈ t is the error term;

[0167] Use the Apriori algorithm to mine frequent patterns from behavioral data and identify frequently occurring behavioral combinations; the identified behavioral patterns and frequent behavioral combinations generate a behavioral feature vector B i , indicating the characteristics of users in terms of behavioral patterns;

[0168] Use the sentiment analysis model BERT to analyze the sentiment tendencies in text data; use the LDA topic modeling method to extract the main topics in the text data and analyze the group's concerns and attitudes; based on text analysis, combine the OCEAN model to evaluate the user's psychological characteristics.

[0169] The specific extraction of social relationship features is as follows:

[0170] Construct a graph structure, define nodes and edges, and convert social network data into a graph structure. Nodes represent individuals (such as users and devices), and edges represent relationships between individuals (such as friend relationships and communication records). Based on the actual data, further distinguish the types of edges (such as strong relationships and weak relationships). Based on the actual data, you can also consider weight calculation. If there are differences in the strength of the relationship, you can assign weights to the edges.

[0171] Community detection based on modularity, identifying groups of closely connected nodes in social networks:

[0172]

[0173] Among them, A ij is an element of the adjacency matrix, indicating whether there is a connection between nodes i and j, k i and k j is the degree of nodes i and j, m is the total number of edges in the graph, δ(c i ,c j ) is an indicator function, which is 1 when i and j belong to the same community, otherwise it is 0; this method is used to analyze the social network structure and the relationship within the group, and identify influential individuals or nodes in the network;

[0174] The specific extraction of social media and economic features is as follows:

[0175] Analyze the types and frequency of media users are exposed to; analyze the sources and preferences of users for obtaining information; extract expenditure types and amounts from consumption records and analyze consumption trends;

[0176] After extracting demographic characteristics, geographic location distribution, behavioral and emotional psychological characteristics, social relations, social media, and economic characteristics, bar charts, pie charts, column charts, and network diagrams are used to intuitively display group portraits, analyze group behavioral trends, estimate potential risks and challenges, and provide an entry for downloading analysis reports.

[0177] S4, group behavior analysis; e.g. Figure 4 As shown, in this embodiment, it specifically includes:

[0178] Group clustering, specifically:

[0179] The elbow method is used to determine the number of clusters. By calculating the intra-cluster sum of squares WCSS under different cluster numbers k, the k value that significantly reduces WCSS, that is, the elbow point, is found to determine the appropriate number of clusters:

[0180]

[0181] Among them, C j is the jth cluster, μ j is the centroid of the cluster; then the K-means algorithm is used for clustering, and the obtained results are used to analyze group behavior; the cluster center provides a concise description of the overall characteristics of the cluster; the intra-cluster difference represents the consistency and diversity of the characteristics of the members within the group; the inter-cluster difference is measured by the distance between different cluster centers, and a larger inter-cluster difference means that the characteristics of different groups are significantly different;

[0182] Time series and sentiment feature analysis, specifically:

[0183] Taking time series features as input, the K-means clustering algorithm is used to divide groups into different behavior pattern groups; the groups of different clusters are compared and analyzed to identify the characteristic differences of each group; the Granger causality test method is used to analyze the causal relationship between time series features. The core formula of the Granger causality test is:

[0184]

[0185] Among them, if β j If it is significantly different from zero, then X t Yes t Granger causes; finally, the LSTM time series model is used to predict the future behavior trend of the group, and the BERT model is used for sentiment analysis to obtain the sentiment polarity, sentiment intensity and sentiment trend;

[0186] Behavioral association analysis, specifically:

[0187] Convert the text data into a format where each line represents a transaction and each transaction contains a set of items;

[0188] Input the Apriori algorithm, count the frequency of each item in all transactions, calculate the support, filter out frequent itemsets according to the minimum support threshold, use frequent itemsets to generate new candidate itemsets, calculate their support, and repeat the steps of generating candidate frequent itemsets and calculating support until no new frequent itemsets can be generated;

[0189] For each frequent item set, generate all possible association rules, and for rule A->B, calculate the confidence:

[0190] Confidence(A->B)=Support(A∪B) / Support(A)

[0191] According to the minimum confidence threshold, meaningful association rules are screened out;

[0192] Anomaly detection, specifically:

[0193] The K-nearest neighbor algorithm is used to calculate the average distance between individual behaviors and their nearest neighbors, specifically:

[0194]

[0195] Among them, x ik and x jk The data points X are i and X j The value of the i-th feature, n is the dimension of the feature;

[0196] If the distance exceeds the preset threshold, the individual may be an anomaly; analyze and explain each anomaly point, find out the reason for its deviation from the norm, and visualize it with scatter plots and box plots to clearly highlight abnormal behavior.

[0197] S5. Output visualization: Output the group behavior prediction results in a visual way, including:

[0198] Behavior trend chart: Use a line chart to show the time series trend of group behavior. Each line represents the change of different behavior patterns, which is convenient for observing the dynamic changes of behavior in different time periods.

[0199] Cluster diagram: A two-dimensional or three-dimensional scatter plot is used to display the clustering results of a group. Different colors or shapes represent different group clusters. The marks at the center of the cluster are used to represent the representative characteristics of the cluster, making it easier to identify the similarities and differences between groups.

[0200] Association rule diagram: Use a network diagram to show the association relationship between behaviors. Nodes represent behaviors or events, and the lines between nodes represent their association rules. The thickness and color of the lines represent the strength and confidence of the association.

[0201] Abnormal behavior detection graph: Abnormal behavior in a group is displayed through scatter plots and box plots. Scatter plots are used to intuitively display the difference between abnormal points and normal behavior, and box plots are used to display the distribution of data and the location of abnormal values.

[0202] It should also be noted that in this specification, terms such as "comprises", "includes" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of more restrictions, an element defined by the sentence "comprises a ..." does not exclude the presence of other identical elements in the process, method, article or device including the element.

[0203] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A group behavior analysis method based on multimodal information fusion and dynamic updating, characterized in that: include: Information integration and dynamic updating, acquisition of multimodal information and preprocessing of information, extraction feature, Perform representation conversion, alignment and fusion of features, and use a stream processing framework for real-time processing and dynamic updating of multimodal information; Introduce self-learning and self-optimization mechanisms; Group portrait generation: extracting demographic characteristics, geographic location distribution, behavioral and emotional psychological characteristics, social relations, social media, and economic characteristics from multimodal data to generate group portraits; Group behavior analysis; Output visualization: Output group behavior prediction results in a visual way, including behavior trend graphs, group clustering graphs, association rule graphs, and abnormal behavior detection graphs.

2. The method for analyzing group behavior by multimodal information fusion and dynamic updating according to claim 1, characterized in that: Obtain multimodal information and preprocess the information, specifically: Collect multimodal data from multiple data sources, including text, images, audio and video; text data is in T i Input in the form of I i The audio data is input in the form of A i Input in the form of V i Form input; Clean and standardize different types of data, and the processing process of text data is as follows: The text is segmented and stop words are removed, special tags are added, and words are converted to corresponding IDs. Then, an attention mask is generated, that is, a vector with the same length as the word ID sequence is generated to indicate which positions are valid words and which are fillers. Finally, a segment ID is generated to mark the words belonging to the first sentence and the words belonging to the second sentence. Finally, three vectors are output: T i id =(id1, id2…id sequence_length ), attention mask and segment ID; where T i id The generated vector has the same length as the word ID sequence, and sequence_length is the length of the word sequence; The image data processing process is: Resize the image to an output shape of H×W×C, where H and W are the sizes of the target image and C is the number of channels; According to the formula: The adjusted image is normalized, where is the image after resizing the image, μ is the mean of the image dataset, σ is the standard deviation, and the final output is in the following matrix: The audio data processing process is: Use fast Fourier transform to get the audio spectrum according to the following formula: in, is the audio signal in the frequency domain, f is the frequency, and N is the number of sampling points; The spectral subtraction method is used to remove noise using the following formula: in, is the noise spectrum estimated in the silent part; Finally get The form is as follows: <h2 style=";text-align:left;direction:ltr">A<h2 style=";text-align:left;direction:ltr"> i <h2 style=";text-align:left;direction:ltr"> denoised <h2 style=";text-align:left;direction:ltr"> (a1)<h2 style=";text-align:left;direction:ltr"> denoised <h2 style=";text-align:left;direction:ltr"> a2<h2 style=";text-align:left;direction:ltr"> denoised <h2 style=";text-align:left;direction:ltr"> …a<h2 style=";text-align:left;direction:ltr"> n <h2 style=";text-align:left;direction:ltr"> denoised <h2 style=";text-align:left;direction:ltr"> ) Each element corresponds to the amplitude value of the denoised audio signal at time k, and n represents the total length of the denoised audio signal; The video data processing process is as follows: Decode the video, decode the video stream into multiple independent frames, and output the frame sequence V i decoded , the shape is N×H×W×C, where N is the number of frames, H and W are the height and width of the frame, and C is the number of color channels; Frame sampling, sampling key frames from the frame sequence, and outputting the sampled frame sequence V i sampled ; Perform frame preprocessing, resize and normalize each frame to make it suitable for model input; The final output is the preprocessed frame sequence V i preprocessed , the output matrix is ​​as follows:

3. The method for analyzing group behavior by multimodal information fusion and dynamic updating according to claim 2, characterized in that: The specific features of text extraction are: Use natural language processing (NLP) to extract semantic features from text data. The processed text is input into the BERT model. The input of BERT includes word ID, attention mask and segment ID. The input word ID sequence T i id First, it is transformed into a fixed-dimensional word vector through the embedding layer, with a shape of E∈R sequence_length×hidden_size , hidden_size is the hidden layer dimension; then each input vector is calculated through the multi-head self-attention mechanism in the model to calculate its dependency with other positions in the sequence. The formula is as follows: Among them, Q, K, and V are the matrices of query, key, and value, respectively, obtained through different linear transformations. k is the scaling factor, and then concatenates the outputs of multiple self-attentions together: MultiHead(Q,K,V)=Concat(head1,…,head h )W O Among them, Concat() is the concatenation operation function, head i =Attention(Q i ,K i ,V i ), and finally the vector of each position is passed through a feedforward neural network to further process its semantic information: FFN(x)=max(0,xW1+b1)W2+b2 Among them, W1 and W2 are learnable weight matrices, b1 and b2 are bias terms, and x is the input of the previous step; The output of each layer is residually connected to its input and the layer is normalized. The formula is as follows: LayerNorm(x+SubLayer(x)) The final output is two vectors. The first is the sequence feature matrix, a three-dimensional tensor that provides the vector representation of each word in the sequence. The second is the sentence feature vector, a two-dimensional tensor that provides the overall vector representation of the entire sequence. The specific image features are extracted as follows: The convolutional neural network (CNN) deep learning method is used to extract visual features from image data. The preprocessed image is input and the convolution operation is performed first: Among them, F ij is a pixel value of the convolution feature map, W mnc is the weight of the convolution kernel, b is the bias term, H f and W f are the height and width of the convolution kernel, respectively, and C is the number of channels of the input image; Output convolution feature map F, shape is Among them, N f is the number of convolution kernels, representing the depth of the output feature map, that is, the number of channels; Through the ReLU nonlinear activation function, nonlinear features are introduced: A ij =ReLU(F ij )=max(0,F ij ) Among them, A ij It is a pixel value in the activated feature map. The shape of the activated feature map A is the same as that of the convolution feature map. Use maximum pooling to downsample the feature map to reduce the spatial dimension of the feature map while retaining important feature information: P ij =max(A(i,j),A(i+1,j),A(i,j+1),A(i+1,j+1)) The feature map P after pooling has the shape of Where H″×W″ is the height and width after pooling; After multiple layers of convolution, activation, and pooling, the extracted feature map is usually flattened into a one-dimensional vector, namely P flat = Flatten(P) and input to the fully connected layer to further combine and extract global features: z=W fc ·P flat +b fc Among them, W fc is the weight matrix of the fully connected layer, b fc is the bias term; The feature vector of the final output image; The specific audio features extracted are: Mel frequency cepstral coefficients (MFCC) are used to extract sound features from audio data, and the processed audio is subjected to short-time Fourier transform: Where m is the frame index, f is the frequency, N is the number of sampling points per frame, hop_size is the frame shift, and the output frequency domain signal A i stft , the shape is a two-dimensional matrix, containing the spectrum information of multiple frames; Apply Mel frequency scaling: Where f0 is the normal frequency, f mel is the Mel frequency; output Mel spectrum A i mel (m,f mel ), which represents the spectral information of the audio signal on the Mel scale; Compute the Mel-frequency cepstral coefficients: Where c is the coefficient index of MFCC, taking the first 13 coefficients, that is, c = 13; The final extracted MFCC features are represented as a two-dimensional array; The specific video features are extracted as follows: Combining image feature extraction and action recognition technology, key frame features and dynamic information are extracted from video data. The processed video is firstly subjected to static feature extraction using CNN to obtain the static feature vector z of each frame, with the shape z∈R feature_size , and then use LSTM for dynamic feature extraction. For each time step, LSTM processes the input feature z in turn t , combined with the hidden state h of the previous time step t-1 and cell state C t-1 , generate the hidden state h of the current time step t and cell state C t Each time step is processed as follows: h t =o t ·tanh(C t ) Finally, static features and dynamic features are obtained.

4. The method for analyzing group behavior by multimodal information fusion and dynamic updating according to claim 3 is characterized in that: The features are represented, transformed, aligned and fused, specifically: Convert the extracted features into a unified vector representation to facilitate comparison and fusion between different modality data; Through timestamps, event markers or other alignment methods, data of different modalities are synchronized in time and space to ensure the relevance between data; in terms of time, the modalities with time dimension, i.e. video and audio, are aligned according to the minimum time step; in terms of space, alignment is performed based on spatial anchor points; Normalize using zero mean and unit variance: Among them, μ X is the mean of the eigenvectors, σ X is the standard deviation; The final output is the normalized feature vector T norm ,I norm , A norm 、V norm ; T norm ,I norm , A norm 、V norm are the eigenvectors corresponding to the four modes output after normalization with zero mean and unit variance; The specific feature fusion is: The feature vectors of different modes are spliced ​​together to form a comprehensive feature vector F concat ∈R batch _size×(dimT+dimI+dimA+dimV) , after concatenation, the dimension of the new feature vector is the sum of the dimensions of each modality; where batch_size is the number of data samples processed during each training or inference; dimT, dimI, dimA, and dimV mean the dimensions of the feature vectors of the four modalities; According to the importance of different modes, each eigenvector is weighted and summed: F weighted =w T ·T norm +w I ·I norm +w A ·A norm +w V ·V norm The weighted comprehensive feature vector F weighted ∈R batch_size×(dimT+dimI+dimA+dimV) , forming comprehensive features, and the selection of weights is determined based on experience and usage scenarios; Finally, we perform dimensionality reduction and use PCA to reduce the dimensionality of high-dimensional features to reduce computational complexity and noise. First, we need to standardize the input data: Among them, μ is the mean of each feature and σ is the standard deviation; the standardized data is used to calculate the covariance matrix, which describes the linear correlation between the features: Output the covariance matrix C, perform eigenvalue decomposition on the covariance matrix, and obtain the eigenvalues ​​and corresponding eigenvectors. The eigenvalues ​​represent the variance of the data in the direction of each eigenvector: Cv i = i v i Among them, λ i is the ith eigenvalue, v i is the corresponding eigenvector; Output eigenvalue vector λ = [λ1,λ2,…,λ dim ] and the eigenvector matrix V∈R dim×dim , sort by the size of the eigenvalues, and select the eigenvectors corresponding to the first n largest eigenvalues ​​as the principal components: V pca =[v1,v2,…,v n ] Among them, n is the number of principal components selected, which is determined by the cumulative variance contribution rate; Project the standardized data onto the selected principal component space to obtain the feature representation after dimensionality reduction: F pca =F std V pca Finally, a matrix of low-dimensional feature representation is output, which contains the representation of the original high-dimensional data projected onto the principal component direction; each sample data has a projection value in these principal component directions, and these projection values ​​constitute the new feature F after dimensionality reduction. pca ∈R batch_size×n , where n is the target dimension after dimensionality reduction.

5. The method for analyzing group behavior by multimodal information fusion and dynamic updating according to claim 4, characterized in that: The stream processing framework is used for real-time processing and dynamic updating of multimodal information, specifically: Use the stream processing framework Apache Flink for real-time processing and dynamic updating of multimodal information; Flink is used to perform real-time cleaning and feature extraction on the incoming data stream. Assuming that the weight of the model is W(t), the dynamically updated model weight is expressed as: Flink is used to perform real-time aggregation and analysis of data streams. Flink aggregates data streams within a specific time window through window operations. The aggregation characteristics within the time window [t, t+Δt] are expressed as: Generate real-time analysis results by classifying, clustering or other analysis operations on the aggregated features; Flink iterative operations continuously optimize the model or processing strategy. Based on the feedback of analysis results, the model parameters are updated or the feature extraction method is adjusted to make the system perform better when processing the next batch of data. The iterative update process is expressed as: in, is the feature after the kth iteration, and α is the update step size.

6. The method for analyzing group behavior by multimodal information fusion and dynamic updating according to claim 1, characterized in that: The introduction of self-learning mechanism specifically includes: Automatically label new data, use the trained BERT model, CNN and LSTM to label and infer new data, and automatically generate labels or classification results; output automatically generated labels or classification results; Self-update of the model, specifically: The model parameters are updated through the online gradient descent method when new data arrives to adapt to the dynamic changes of data; when new text data arrives, BERT's online learning extension is used to update its weights; input new data X new and its true label y new And the current model parameter W, online gradient descent update formula: Where η is the learning rate, is the gradient of the loss function with respect to the model parameters; outputs the updated model parameters W new ; Learning feedback mechanism, specifically: Compare the predicted labels with the true labels, adjust the adaptive learning rate, adaptive loss function weight, and finally obtain the adjusted learning rate, loss function or other hyperparameters; Input model prediction label and the true label y true , adjust the learning rate and loss function weight according to the prediction error, and the error calculation formula is: The learning rate adjustment formula is: or new =the old ·(1-a·e 2 ) Among them, α is the adjustment coefficient; Adjust the loss function weight: l new =λ old +b·e Among them, β is the adjustment coefficient; Output adjusted learning rate η new and loss function weight λ new .

7. The method for analyzing group behavior by multimodal information fusion and dynamic updating according to claim 1, characterized in that: The introduction of the self-optimization mechanism specifically includes: Hyperparameter optimization: The goal of hyperparameter optimization is to find the hyperparameter combination that can achieve the best model performance in a given hyperparameter space; Define the hyperparameters to be optimized and their possible value ranges {θ1,θ2,...,θ n }, use genetic algorithms to simulate natural evolution to select optimal parameters, repeat the selection, crossover, and mutation process until convergence conditions are reached or the preset maximum number of generations is reached; the optimal hyperparameter configuration is finally determined for use by the BERT model, CNN, and LSTM; Model architecture optimization, specifically: Input the initial neural network architecture design and the data set for training and verification; define the structural parameters of the neural network, and select the neural network architecture A randomly or through a strategy from the search space S i To conduct an assessment: in, Based on architecture A i The model, G(A i ) is the average loss on the validation set; after multiple sampling and evaluation, the architecture A with the best performance on the validation set is selected * As the final optimal architecture; Computing resource optimization, specifically: First, input the current computing resource status and task requirements, including CPU and GPU utilization, memory usage, and task priority; use task scheduling algorithms and resource monitoring and allocation to adjust resource allocation; monitor resource usage in real time and dynamically adjust resource allocation to respond to system changes and task execution; finally output optimized resource allocation and task scheduling strategies to improve the overall efficiency and response speed of the system.

8. The method for analyzing group behavior by multimodal information fusion and dynamic updating according to claim 1, characterized in that: The demographic features extracted are: Use the NER model based on conditional random fields to extract information including but not limited to age, gender, and occupation from text data. The formula is as follows: Among them, y is the label sequence, that is, the extracted demographic features; x is the input sequence, that is, the text data; f k (y,x) is the characteristic function, λ k is the weight, Z(x) is the normalization factor to ensure the normalization of the probability distribution; The extracted demographic features generate a demographic feature vector D i , containing the numerical values ​​of various information; Extract geographic location distribution, specifically: Use geocoding technology to convert the location described in the text into geographic coordinates, use the K-means clustering method to calculate the minimum intra-cluster sum of squares WCSS, and identify the main activity areas; finally, visualize the geographical distribution through heat maps to show the distribution density of groups in different regions; Extract behavioral and emotional psychological features, specifically: Use time series analysis and frequent pattern mining techniques to analyze user behavior, and use the autoregressive integrated moving average model ARIMA to analyze and predict user behavior patterns: x t =c+φ1x t-1 +φ2x t-2 +...+φ p x t-p +∈ t +θ1∈ t-1 +...+θ q ∈ t-q Among them, c is a constant term, φ p is the autoregressive coefficient, θ q is the moving average coefficient, ∈ t is the error term; Use the Apriori algorithm to mine frequent patterns from behavioral data and identify frequently occurring behavioral combinations; the identified behavioral patterns and frequent behavioral combinations generate a behavioral feature vector B i , indicating the characteristics of users in terms of behavioral patterns; Use the sentiment analysis model BERT to analyze the sentiment tendencies in text data; use the LDA topic modeling method to extract the main topics in the text data and analyze the group's concerns and attitudes; based on text analysis, combine the OCEAN model to evaluate the user's psychological characteristics.

9. The method for analyzing group behavior by multimodal information fusion and dynamic updating according to claim 1, characterized in that: The specific extraction of social relationship features is as follows: Construct a graph structure, define nodes and edges, and convert social network data into a graph structure. Nodes represent individuals, and edges represent relationships between individuals. Based on actual data, further distinguish edge types and calculate weights. If there are differences in the strength of relationships, assign weights to edges. Community detection based on modularity, identifying groups of closely connected nodes in social networks: Among them, A ij is an element of the adjacency matrix, indicating whether there is a connection between nodes i and j, k i and k j is the degree of nodes i and j, m is the total number of edges in the graph, δ(c i ,c j ) is an indicator function, which is 1 when i and j belong to the same community, otherwise it is 0; this method is used to analyze the social network structure and the relationship within the group, and identify influential individuals or nodes in the network; The specific extraction of social media and economic features is as follows: Analyze the types and frequency of media users are exposed to; analyze the sources and preferences of users for obtaining information; extract expenditure types and amounts from consumption records and analyze consumption trends; After extracting demographic characteristics, geographic location distribution, behavioral and emotional psychological characteristics, social relations, social media, and economic characteristics, bar charts, pie charts, column charts, and network diagrams are used to intuitively display group portraits, analyze group behavioral trends, estimate potential risks and challenges, and provide an entry for downloading analysis reports.

10. The method for analyzing group behavior by multimodal information fusion and dynamic updating according to claim 1, characterized in that: Group behavior analysis specifically includes: Group clustering, specifically: The elbow method is used to determine the number of clusters. By calculating the intra-cluster sum of squares WCSS under different cluster numbers k, the k value that significantly reduces WCSS, that is, the elbow point, is found to determine the appropriate number of clusters: Among them, C j is the jth cluster, μ j is the centroid of the cluster; then the K-means algorithm is used for clustering, and the obtained results are used to analyze group behavior; the cluster center provides a concise description of the overall characteristics of the cluster; the intra-cluster difference represents the consistency and diversity of the characteristics of the members within the group; the inter-cluster difference is measured by the distance between different cluster centers, and a larger inter-cluster difference means that the characteristics of different groups are significantly different; Time series and sentiment feature analysis, specifically: Taking time series features as input, the K-means clustering algorithm is used to divide groups into different behavior pattern groups; the groups of different clusters are compared and analyzed to identify the characteristic differences of each group; the Granger causality test method is used to analyze the causal relationship between time series features. The core formula of the Granger causality test is: Among them, if β j If it is significantly different from zero, then X t Yes t Granger causes; finally, the LSTM time series model is used to predict the future behavior trend of the group, and the BERT model is used for sentiment analysis to obtain the sentiment polarity, sentiment intensity and sentiment trend; Behavioral association analysis, specifically: Convert the text data into a format where each line represents a transaction and each transaction contains a set of items; Input the Apriori algorithm, count the frequency of each item in all transactions, calculate the support, filter out frequent itemsets according to the minimum support threshold, use frequent itemsets to generate new candidate itemsets, calculate their support, and repeat the steps of generating candidate frequent itemsets and calculating support until no new frequent itemsets can be generated; For each frequent item set, generate all possible association rules, and for rule A->B, calculate the confidence: Confidence(A->B)=Support(A∪B) / Support(A) According to the minimum confidence threshold, meaningful association rules are screened out; Anomaly detection, specifically: The K-nearest neighbor algorithm is used to calculate the average distance between individual behaviors and their nearest neighbors, specifically: Among them, x ik and x jk The data points X are i and X j The value of the i-th feature, n is the dimension of the feature; If the distance exceeds the preset threshold, the individual may be an anomaly; analyze and explain each anomaly point, find out the reason for its deviation from the norm, and visualize it with scatter plots and box plots to clearly highlight abnormal behavior.

Images