Portrayal generation and feature similarity recommendation method and system based on large language model

Abstract

The invention relates to a portrait generation and feature similarity recommendation method and system based on a large language model, and the system comprises a data collection module, a feature extraction module, a similarity calculation module and a recommendation module, the data collection module collects multi-modal data from a social media platform, and covers audio, video, text and interactive data information; the feature extraction module performs deep feature analysis and processing based on the collected multi-modal data, and extracts features in multiple dimensions; the similarity calculation module generates a training sample based on the feature matrix generated by the multi-dimensional feature extraction module in combination with a large language model, constructs a similarity matrix of audio, video, text and interactive data features, and calculates similarity scores to determine candidate objects; the recommendation module provides personalized recommendation based on the comprehensive similarity score generated by the similarity determination module. Through combination of a large language model and multi-modal features, the system can accurately analyze behaviors and expressions and generate personalized recommendation.

Description

Technical Field

[0001] The present invention belongs to the field of portrait generation and recommendation technology, and in particular to a portrait generation and feature similarity recommendation method and system based on a large language model. Background Art

[0002] In the development of recommendation systems, traditional similarity algorithms have been widely used in multiple industries, such as logistics, medical, chemical, legal, education, and artificial intelligence. They optimize the matching and recommendation process through data-driven technology to achieve personalized services. For example, the vehicle matching recommendation system in the logistics industry uses the analytic hierarchy process and weighted similarity algorithm to improve the matching efficiency of goods and vehicles. The medical and chemical fields use similarity calculations to recommend medical imaging reports and compound matching, and the legal field uses natural language processing technology to recommend similar historical cases. The recommendation systems in these fields are relatively mature, combining big data, machine learning, and natural language processing technologies, which greatly improves the user experience.

[0003] The industry's existing invention patents focus on product quality control, classification, commercial value assessment and merchant identification, and the technology is still insufficient. For example, the product quality control system uses big data analysis and user feedback to evaluate the quality of the products sold, but there is still room for improvement in the in-depth understanding of user interaction data; the classification method relies on pre-trained language models to analyze text and interaction data, providing more accurate classification, but its classification dimension is still limited; the commercial value assessment system evaluates value through performance, but faces challenges when processing multimodal data (such as text, images, and videos); although the merchant identification method can better identify high quality by analyzing user interactions and purchasing behaviors, the reliance on a single data source limits the diversity and accuracy of its recommendation results.

[0004] Although recommendation algorithms have made great progress, current recommendation algorithms still have the following major defects: Single data modality: Many existing systems rely on single-modality data (such as text or interactive data), ignoring the integration of non-text data such as images and videos, resulting in the recommendation system being unable to fully capture multi-dimensional features. Limited accuracy of classification and profiling: Although existing classification systems provide basic classification and profiling, their classification accuracy and detailed dimensions are low, and they are unable to generate comprehensive feature analysis. Insufficient market responsiveness: When faced with rapidly changing markets and user behaviors, existing systems react slowly and lack dynamic adaptability, resulting in insufficient timeliness of recommendation results. Poor interpretability of recommendations: The logic of existing recommendation systems is relatively black-box, making it difficult for users to understand the basis and process of recommendations, resulting in reduced trust.

[0005] Therefore, how to improve the recommendation algorithm in the existing technology to achieve multimodal feature processing, comprehensively capture multi-dimensional features, generate comprehensive feature analysis, and improve the timeliness and credibility of recommendation results is a technical problem that needs to be solved urgently. Summary of the invention

[0006] The purpose of the present invention is to provide a method and system for portrait generation and feature similarity recommendation based on a large language model, so as to improve the recommendation algorithm in the prior art, realize multimodal feature processing, comprehensively capture multidimensional features, generate comprehensive feature analysis, and improve the timeliness and credibility of recommendation results.

[0007] In order to solve the above technical problems, the technical solution adopted by the present invention is as follows:

[0008] In a first aspect, a portrait generation and feature similarity recommendation system based on a large language model includes a data acquisition module, a feature extraction module, a similarity calculation module and a recommendation module, wherein the data acquisition module is connected to the feature extraction module, the feature extraction module is connected to the similarity calculation module, and the similarity calculation module is connected to the recommendation module;

[0009] The data collection module is used to collect multimodal data from social media platforms, including audio, video, text and interactive data information;

[0010] The feature extraction module performs in-depth feature analysis and processing based on the collected multimodal data to extract features in multiple dimensions, including secondary features of capabilities, popularity, behavior patterns, and interaction habits;

[0011] The similarity calculation module generates training samples based on the feature matrix generated by the multidimensional feature extraction module, combines the large language model to construct a similarity matrix of audio, video, text and interactive data features, calculates similarity scores, and determines similar candidate objects;

[0012] The recommendation module, based on the comprehensive similarity score generated by the similarity determination module, selects the results with the highest similarity to the target from the massive amount of data and provides personalized recommendations.

[0013] Preferably, the data acquisition module includes a data interface module and a data preprocessing module, wherein the data interface module obtains the published audio, video, text and interactive multimodal data through a legal API interface or a public data capture method; the data preprocessing module cleans, converts the format and denoises the collected raw data, and the collected data specifically includes posts, comments, likes, reposts, quantity and publishing location.

[0014] Preferably, the feature extraction module includes a text feature extraction submodule, an audio feature extraction submodule, a video feature extraction submodule and an interactive feature extraction submodule; the text feature extraction submodule uses a large language model to perform semantic analysis on the published text content to extract keywords, emotional tendencies and topic preferences; the audio feature extraction submodule analyzes the audio content to extract voice features, emotional expression, intonation and speech speed; the video feature extraction submodule analyzes the visual style, subject display and emotional expression in the video content; the interactive feature extraction submodule extracts the interaction frequency, depth and pattern based on the interactive behavior;

[0015] The process of text feature extraction by the text feature extraction submodule is as follows:

[0016] Count the keywords in the text and the frequencies corresponding to the keywords, and calculate the information entropy of each keyword. The information entropy calculation formula is: H(X)=E[I(X)]=E[-ln(P(X))];

[0017] Where I(X) is the information content of keyword X, E[x] is the expected function of variable x, and P(x) is the probability mass function of variable x;

[0018] The information gain of each keyword is calculated based on the information entropy of each keyword, and the calculation formula of the information gain is: G=H(X)-H(X|a);

[0019] Among them, H(X|a) represents the conditional entropy of keyword X under given condition a;

[0020] Sort by the information gain of each keyword and extract the first n keywords as the features of the text.

[0021] Preferably, the process of audio feature extraction performed by the audio feature extraction submodule is:

[0022] Cut the audio content into several segments of specified length, perform Fourier transform on each audio segment, and obtain the spectrum information of each audio segment. The specific formula for Fourier transform on each audio segment is:

[0023]

[0024] Among them, s(τ) is the audio signal, w(t-τ) is the window function, and f is the audio frequency.

[0025] Preferably, the process of video feature extraction performed by the video feature extraction submodule is:

[0026] The video is decomposed into single-frame images, and features are extracted for the specified frame images based on a convolutional neural network model, wherein the convolutional neural network model is a 3D convolutional neural network model, which includes 3 convolutional layers, 1 fully connected layer and 2 pooling layers, at least two convolution kernels of the three convolutional layers are 3D convolutional layers, and the sizes of the convolution kernels of the two 3D convolutional layers are 7×6×3 and 7×7×3, each node of the fully connected layer is connected to all nodes of its previous layer, and the pooling layer calculates the average value of all values ​​in the window as the representative value of the specified image area.

[0027] Preferably, the feature extraction process of the interactive feature extraction submodule is:

[0028] The interactive data type is identified. If it is identified as text data, it is transmitted to the text feature extraction submodule for text feature extraction; if it is identified as video data, it is transmitted to the video feature extraction submodule for video feature extraction; if it is identified as audio data, it is transmitted to the audio feature extraction submodule for audio feature extraction.

[0029] Preferably, the similarity calculation module calculates the comprehensive similarity score between them and transmits it to the recommendation module; the similarity score will be transmitted to the recommendation module, which specifically includes a training data generation module, a large language model fine-tuning module, and a similarity calculation sub-module. The training data generation module screens the historical data of its published content and extracts keyword data to generate training data; the large language model fine-tuning module uses the large language model to fine-tune the training data to generate a preliminary sample for similarity calculation; the similarity calculation sub-module uses a similarity algorithm to calculate the similarity matrix of each modal feature respectively.

[0030] Preferably, the similarity matrix of each modal feature includes a text similarity matrix, an audio similarity matrix, and a video similarity matrix;

[0031] Text similarity matrix: calculates text similarity by similarity of language style, topic and keywords;

[0032] Audio similarity matrix: calculates the similarity of audio content by analyzing audio emotion, intonation, and speech speed;

[0033] Video similarity matrix: Generate a video similarity matrix through visual performance and style analysis in the video.

[0034] Preferably, the calculation process of the text similarity matrix is ​​as follows: extracting the specified features of the text content based on the text feature extraction submodule, converting the corresponding text specified features into text vectors in the vector space based on the frequency, frequency and weight of the extracted specified features, generating a text similarity matrix from all text specified feature vectors, and calculating the cosine similarity of the text specified feature vectors in the text similarity matrix. The calculation formula of the cosine similarity is as follows:

[0035] Cosine similarity = A·B / (||A||·||B||);

[0036] Where A and B are two text vectors, A·B is the dot product of the two text vectors, ||A|| and ||B|| are the moduli of the two text vectors;

[0037] The cosine similarity of the text similarity matrix is ​​obtained by summing the cosine similarities of the specified feature vectors of all texts.

[0038] In a second aspect, a method for generating a portrait and recommending a feature similarity based on a large language model is provided, wherein any one of the systems for generating a portrait and recommending a feature similarity based on a large language model is applied, and the method comprises the following steps:

[0039] S1: The data collection module legally collects audio, video, text and interactive data information from the social platform, and transmits it to the feature extraction module after pre-processing;

[0040] S2: The feature extraction module performs feature analysis on the collected multimodal data, generates feature extraction including text, audio, video, and interactive data, and transmits the extracted features to the similarity calculation module;

[0041] S3: The similarity calculation module calculates the similarity matrix of each modal feature based on the extracted features, calculates the comprehensive similarity score based on the similarity matrix of each modal feature, and transmits the similarity score result to the recommendation module;

[0042] S4: The recommendation module selects the results with the highest similarity based on the similarity scoring results, provides personalized recommendations, and displays them to the user;

[0043] The process of text feature extraction by the text feature extraction submodule is as follows:

[0044] Count the keywords in the text and the frequencies corresponding to the keywords, and calculate the information entropy of each keyword. The information entropy calculation formula is: H(X)=E[I(X)]=E[-ln(P(X))];

[0045] Where I(X) is the information content of keyword X, E[x] is the expected function of variable x, and P(x) is the probability mass function of variable x;

[0046] The information gain of each keyword is calculated based on the information entropy of each keyword, and the calculation formula of the information gain is: G=H(X)-H(X|a);

[0047] Among them, H(X|a) represents the conditional entropy of keyword X under given condition a;

[0048] Sort by the information gain of each keyword and extract the first n keywords as the features of the text.

[0049] The process of audio feature extraction by the audio feature extraction submodule is as follows:

[0050] Cut the audio content into several segments of specified length, perform Fourier transform on each audio segment, and obtain the spectrum information of each audio segment. The specific formula for Fourier transform on each audio segment is:

[0051]

[0052] Among them, s(τ) is the audio signal, w(t-τ) is the window function, and f is the audio frequency.

[0053] The process of video feature extraction by the video feature extraction submodule is as follows:

[0054] The video is decomposed into single-frame images, and features are extracted for the specified frame images based on a convolutional neural network model, wherein the convolutional neural network model is a 3D convolutional neural network model, which includes 3 convolutional layers, 1 fully connected layer and 2 pooling layers, at least two convolution kernels of the three convolutional layers are 3D convolutional layers, and the sizes of the convolution kernels of the two 3D convolutional layers are 7×6×3 and 7×7×3, each node of the fully connected layer is connected to all nodes of its previous layer, and the pooling layer calculates the average value of all values ​​in the window as the representative value of the specified image area.

[0055] The feature extraction process of the interactive feature extraction submodule is as follows:

[0056] The interactive data type is identified. If it is identified as text data, it is transmitted to the text feature extraction submodule for text feature extraction; if it is identified as video data, it is transmitted to the video feature extraction submodule for video feature extraction; if it is identified as audio data, it is transmitted to the audio feature extraction submodule for audio feature extraction.

[0057] The beneficial effects of the present invention include:

[0058] The large language model-based portrait generation and feature similarity recommendation method and system provided by the present invention are provided with a data collection module, a feature extraction module, a similarity calculation module and a recommendation module. The data collection module collects multimodal data from a social media platform, covering audio, video, text and interactive data information; the feature extraction module performs in-depth feature analysis and processing based on the collected multimodal data, and extracts features in multiple dimensions; the similarity calculation module generates training samples based on the feature matrix generated by the multidimensional feature extraction module and the large language model, constructs a similarity matrix of audio, video, text and interactive data features, and calculates similarity scores to determine candidate objects; the recommendation module provides personalized recommendations based on the comprehensive similarity scores generated by the similarity determination module.

[0059] First, through the collaborative work of the data collection module, feature extraction module, similarity calculation module and recommendation module, the entire process of collecting public data from social media, extracting multi-dimensional features, determining similarities and finally making recommendations is realized. Through the combination of a large language model and multimodal features, the system can accurately analyze behavior and performance, generate personalized recommendations, and has wide adaptability.

[0060] Second, the feature extraction module is set up with text feature extraction sub-module, audio feature extraction sub-module, video feature extraction sub-module and interactive feature extraction sub-module to perform targeted feature extraction on text features, audio features, video features and interactive features respectively, effectively improving the accuracy of feature extraction, and combined with the subsequent similarity calculation module to perform similarity calculation on different features respectively, to further ensure the accuracy of the similarity calculation results, and further ensure the accuracy of the recommendation results of the recommendation module.

[0061] Third, count the keywords in the text and the frequency of each keyword, calculate the information entropy of each keyword, and calculate the information gain of each keyword based on the information entropy of each keyword; set the calculation formula of information entropy and information gain, quantify the keyword features, sort them according to the size of the information gain of each keyword, extract the first n keywords as the features of the text, and effectively ensure the accuracy of text feature extraction.

[0062] Fourth, by cutting the audio content into several segments of specified length, performing Fourier transform on each segment, obtaining the spectrum information of each segment, and quantizing the audio features based on the set Fourier transform formula, the accuracy of audio feature extraction is effectively improved.

[0063] Fifth, decompose the video into single-frame images, and extract features from the specified frame images based on the convolutional neural network model. The convolutional neural network model is a 3D convolutional neural network model, which includes 3 convolutional layers, 1 fully connected layer, and 2 pooling layers. At least two of the 3 convolutional layers have convolutional kernels that are 3D convolutional layers. The sizes of the convolutional kernels of the two 3D convolutional layers are 7×6×3 and 7×7×3. Each node of the fully connected layer is connected to all nodes of the previous layer. The pooling layer calculates the average value of all values ​​in the window as the representative value of the specified image area. The set 3D convolutional neural network model is used to achieve efficient and accurate extraction of video features.

[0064] Sixth, based on the frequency, frequency and weight of the extracted specified features, the corresponding text specified features are converted into text vectors in the vector space, a text similarity matrix is ​​generated from all text specified feature vectors, and the cosine similarity of the text specified feature vectors in the text similarity matrix is ​​calculated. Then, accurate personalized recommendations are achieved based on the cosine similarity calculation results. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] Figure 1 This is a structural diagram of the portrait generation and feature similarity recommendation system based on a large language model of the present invention.

[0066] Figure 2 The figure is a flowchart of the method for generating a portrait and recommending feature similarity based on a large language model according to the present invention.

[0067] Figure 3 Schematic diagram of the process of text feature extraction performed by the text feature extraction submodule of the present invention. DETAILED DESCRIPTION

[0068] The following is combined with Figure 1 to Figure 3 The present invention is further described in detail:

[0069] Example 1

[0070] See also Figure 1 As shown, the portrait generation and feature similarity recommendation system based on the large language model includes a data collection module, a feature extraction module, a similarity calculation module and a recommendation module, wherein the data collection module is connected to the feature extraction module, the feature extraction module is connected to the similarity calculation module, and the similarity calculation module is connected to the recommendation module. The data collection module is used to collect multimodal data from social media platforms, including audio, video, text and interactive data information.

[0071] The feature extraction module performs in-depth feature analysis and processing based on the collected multimodal data, extracting features in multiple dimensions, including secondary features of capabilities, popularity, behavior patterns, and interaction habits. The similarity calculation module generates training samples based on the feature matrix generated by the multidimensional feature extraction module and the large language model, constructs a similarity matrix of audio, video, text, and interactive data features, calculates similarity scores, and determines similar candidate objects. The recommendation module selects the results with the highest similarity to the target from the massive data based on the comprehensive similarity score generated by the similarity determination module, and provides personalized recommendations.

[0072] Through the collaborative work of the data collection module, feature extraction module, similarity calculation module and recommendation module, the entire process of collecting public data from social media, extracting multi-dimensional features, determining similarities and finally making recommendations is realized. Through the combination of large language models and multimodal features, the system can accurately analyze behaviors and performances, generate personalized recommendations, and has wide adaptability.

[0073] The data acquisition module includes a data interface module and a data preprocessing module. The data interface module obtains the published audio, video, text and interactive multimodal data through a legal API interface or public data crawling method; the data preprocessing module cleans, converts the format and denoises the collected raw data. Through the above preprocessing process of the collected raw data, it can avoid the influence of invalid data and abnormal data in the raw data on the subsequent feature extraction process. The collected data specifically includes posts, comments, likes, reposts, quantity and publishing location.

[0074] In this embodiment, the feature extraction module includes a text feature extraction submodule, an audio feature extraction submodule, a video feature extraction submodule and an interactive feature extraction submodule; the text feature extraction submodule uses a large language model to perform semantic analysis on the published text content to extract keywords, emotional tendencies and topic preferences; the audio feature extraction submodule analyzes the audio content to extract voice features, emotional expression, intonation and speaking speed; the video feature extraction submodule analyzes the visual style, subject display and emotional expression in the video content; the interactive feature extraction submodule extracts the interaction frequency, depth and pattern based on the interactive behavior.

[0075] Example 2

[0076] Based on Example 1, see Figure 3 ,The process of text feature extraction by the text feature extraction submodule is:

[0077] The keywords in the text and the frequencies corresponding to the keywords are counted, and the information entropy of each keyword is calculated. The calculation formula of the information entropy is: H(X)=E[I(X)]=E[-ln(P(X))]. Wherein, I(X) is the information content of keyword X, E[x] is the expected function of variable x, and P(x) is the probability mass function of variable x. The information gain of each keyword is calculated based on the information entropy of each keyword. The calculation formula of the information gain is: G=H(X)-H(X|a); where H(X|a) represents the conditional entropy of keyword X under given condition a. Sort by the information gain of each keyword, and extract the first n keywords as the features of the text. The keywords in the text and the frequencies corresponding to the keywords are counted, and the information entropy of each keyword is calculated. The information gain of each keyword is calculated based on the information entropy of each keyword; and the calculation formula of information entropy and information gain is set to quantify the keyword features, sort by the information gain of each keyword, and extract the first n keywords as the features of the text, which effectively ensures the accuracy of text feature extraction.

[0078] The process of audio feature extraction by the audio feature extraction submodule is as follows:

[0079] Cut the audio content into several segments of specified length, perform Fourier transform on each audio segment, and obtain the spectrum information of each audio segment. The specific formula for Fourier transform on each audio segment is:

[0080]

[0081] Among them, s(τ) is the audio signal, w(t-τ) is the window function, and f is the audio frequency.

[0082] By cutting the audio content into several segments of specified length, performing Fourier transform on each segment, obtaining the spectrum information of each segment, and quantizing the audio features based on the set Fourier transform formula, the accuracy of audio feature extraction is effectively improved.

[0083] The process of video feature extraction by the video feature extraction submodule is as follows:

[0084] Decompose the video into single-frame images, and extract features from the specified frame images based on a convolutional neural network model, wherein the convolutional neural network model is a 3D convolutional neural network model, and the 3D convolutional neural network model includes 3 convolutional layers, 1 fully connected layer, and 2 pooling layers, wherein at least two of the 3 convolutional layers have convolutional kernels that are 3D convolutional layers, and the sizes of the convolutional kernels of the two 3D convolutional layers are 7×6×3 and 7×7×3, and each node of the fully connected layer is connected to all nodes of the previous layer, and the pooling layer calculates the average value of all values ​​in the window as the representative value of the specified image area. Decompose the video into single-frame images, and extract features from the specified frame images based on a convolutional neural network model, and the convolutional neural network model is a 3D convolutional neural network model.

[0085] The fully connected layer compresses the features extracted from the input image after convolution and pooling operations, and completes the classification function of the model based on the compressed features. Each neuron receives the output of all neurons in the previous layer, and weights and sums these inputs, performs nonlinear transformation through the activation function, and finally outputs the result. The fully connected layer can transform high-dimensional data into low-dimensional data, or transform low-dimensional data into high-dimensional data, while retaining useful information. The fully connected layer flattens the input data. Since the output of the convolution layer or pooling layer is a two-dimensional feature map, it needs to be converted into one dimension as the input of the fully connected layer. Therefore, the two-dimensional feature map needs to be flattened into a one-dimensional array through the Flatten layer so that the fully connected layer can process the data correctly. Each neuron is connected to all neurons in the previous layer, and each connection has a corresponding weight. These weights are used to adjust the impact of the input data, and finally the nonlinear transformation is performed through the activation function to obtain the output of the neuron. At the end of the fully connected layer, the Softmax activation function is used to output the probability distribution of each category, thereby achieving the classification task.

[0086] By setting up a 3D convolutional neural network model, features can be extracted from the spatial and temporal dimensions, thereby capturing motion information obtained from multiple consecutive frames, and achieving efficient and accurate extraction of video features. It can then divide the features more finely according to the needs of different fields, providing users with more detailed and quantitative analysis.

[0087] The feature extraction process of the interactive feature extraction submodule is as follows:

[0088] The interactive data type is identified. If it is identified as text data, it is transmitted to the text feature extraction submodule for text feature extraction; if it is identified as video data, it is transmitted to the video feature extraction submodule for video feature extraction; if it is identified as audio data, it is transmitted to the audio feature extraction submodule for audio feature extraction.

[0089] The similarity calculation module calculates the comprehensive similarity score between them and transmits it to the recommendation module; the similarity score will be transmitted to the recommendation module, which specifically includes a training data generation module, a large language model fine-tuning module, and a similarity calculation submodule. The training data generation module screens the historical data of its published content and extracts keyword data to generate training data; the large language model fine-tuning module uses the large language model to fine-tune the training data to generate a preliminary sample for similarity calculation; the similarity calculation submodule uses a similarity algorithm to calculate the similarity matrix of each modal feature respectively.

[0090] Example 3

[0091] Based on Example 1 or Example 2, the similarity matrix of each modal feature includes a text similarity matrix, an audio similarity matrix, and a video similarity matrix. Text similarity matrix: calculates text similarity through the similarity of language style, topic, and keywords; Audio similarity matrix: calculates the similarity of audio content through analysis of audio emotion, intonation, and speech speed; Video similarity matrix: generates a video similarity matrix through visual expression and style analysis in the video.

[0092] In this embodiment, the calculation process of the text similarity matrix is ​​as follows: extract the specified features of the text content based on the text feature extraction submodule, convert the corresponding text specified features into text vectors in the vector space based on the frequency, frequency and weight of the extracted specified features, generate a text similarity matrix from all text specified feature vectors, and calculate the cosine similarity of the text specified feature vectors in the text similarity matrix. The calculation formula of the cosine similarity is as follows:

[0093] Cosine similarity = A·B / (||A||·||B||);

[0094] Where A and B are two text vectors, A·B is the dot product of the two text vectors, and ||A|| and ||B|| are the moduli of the two text vectors. The cosine similarity of the specified feature vectors of all texts is summed to obtain the cosine similarity of the text similarity matrix.

[0095] The calculation process of the audio similarity matrix and the video similarity matrix is ​​similar to that of the text similarity matrix, except that the features extracted in the early stage are different, so the constructed feature vectors are different. The principle of calculating the cosine similarity from the feature vectors is the same.

[0096] A method for image generation and feature similarity recommendation based on a large language model, using any of the image generation and feature similarity recommendation systems based on a large language model, see Figure 2 , including the following steps:

[0097] S1: The data collection module legally collects audio, video, text and interactive data information from the social platform, and transmits it to the feature extraction module after pre-processing;

[0098] S2: The feature extraction module performs feature analysis on the collected multimodal data, generates feature extraction including text, audio, video, and interactive data, and transmits the extracted features to the similarity calculation module;

[0099] S3: The similarity calculation module calculates the similarity matrix of each modal feature based on the extracted features, calculates the comprehensive similarity score based on the similarity matrix of each modal feature, and transmits the similarity score result to the recommendation module;

[0100] S4: The recommendation module selects the results with the highest similarity based on the similarity scoring results, provides personalized recommendations, and displays them to the user;

[0101] The process of text feature extraction by the text feature extraction submodule is as follows:

[0102] Count the keywords in the text and the frequencies corresponding to each keyword, and calculate the information entropy of each keyword. The information entropy calculation formula is: H(X)=E[I(X)]=E[-ln(P(X))]; where I(X) is the information content of keyword X, E[x] is the expected function of variable x, and P(x) is the probability mass function of variable x. Calculate the information gain of each keyword based on the information entropy of each keyword. The information gain calculation formula is: G=H(X)-H(X|a); where H(X|a) represents the conditional entropy of keyword X under given condition a. Sort by the information gain of each keyword and extract the first n keywords as the features of the text.

[0103] The process of audio feature extraction by the audio feature extraction submodule is as follows:

[0104] Cut the audio content into several segments of specified length, perform Fourier transform on each audio segment, and obtain the spectrum information of each audio segment. The specific formula for Fourier transform on each audio segment is:

[0105]

[0106] Among them, s(τ) is the audio signal, w(t-τ) is the window function, and f is the audio frequency.

[0107] The process of video feature extraction by the video feature extraction submodule is as follows:

[0108] The video is decomposed into single-frame images, and features are extracted for the specified frame images based on a convolutional neural network model, wherein the convolutional neural network model is a 3D convolutional neural network model, which includes 3 convolutional layers, 1 fully connected layer and 2 pooling layers, at least two convolution kernels of the three convolutional layers are 3D convolutional layers, and the sizes of the convolution kernels of the two 3D convolutional layers are 7×6×3 and 7×7×3, each node of the fully connected layer is connected to all nodes of its previous layer, and the pooling layer calculates the average value of all values ​​in the window as the representative value of the specified image area.

[0109] The feature extraction process of the interactive feature extraction submodule is as follows:

[0110] The interactive data type is identified. If it is identified as text data, it is transmitted to the text feature extraction submodule for text feature extraction; if it is identified as video data, it is transmitted to the video feature extraction submodule for video feature extraction; if it is identified as audio data, it is transmitted to the audio feature extraction submodule for audio feature extraction.

[0111] In summary, the portrait generation and feature similarity recommendation method and system based on the large language model provided by the present invention integrates multimodal information such as audio, video, text and interaction, comprehensively extracts multidimensional features such as popularity, ability, behavioral habits and interaction patterns, generates accurate portraits, and ensures the comprehensiveness and accuracy of the analysis. Through the large language model, deep semantic analysis of text content can be performed efficiently, which can ensure accurate understanding and interpretation of the content and enhance the system's processing capabilities at the semantic level. Multidimensional similarity calculation and personalized recommendation: The system is based on the similarity matrix of multimodal features, comprehensively calculates the similarity scores between them, and provides accurate personalized recommendations to meet diverse needs such as brand promotion and marketing cooperation.

[0112] Through the collaborative work of the data collection module, feature extraction module, similarity calculation module and recommendation module, the entire process of collecting public data from social media, extracting multi-dimensional features, determining similarities and finally recommending is realized. Through the combination of large language models and multimodal features, the system can accurately analyze behaviors and performances, generate personalized recommendations, and has wide adaptability. The feature extraction module is set up with text feature extraction submodules, audio feature extraction submodules, video feature extraction submodules and interactive feature extraction submodules to perform targeted feature extraction on text features, audio features, video features and interactive features, effectively improving the accuracy of feature extraction, and combining with the subsequent similarity calculation module to perform similarity calculations on different features, further ensuring the accuracy of the similarity calculation results, and further ensuring the accuracy of the recommendation results of the recommendation module.

[0113] The key words in the text and the corresponding frequencies of each key word are counted, the information entropy of each key word is calculated, and the information gain of each key word is calculated based on the information entropy of each key word; and the calculation formulas of information entropy and information gain are set to quantify the key word features, sort them according to the information gain of each key word, and extract the first n key words as the features of the text, which effectively ensures the accuracy of text feature extraction. By cutting the audio content into several segments of a specified length, performing Fourier transform on each audio segment, obtaining the spectrum information of each audio segment, and realizing the quantization of audio features based on the set Fourier transform formula, the accuracy of audio feature extraction is effectively improved.

[0114] The video is decomposed into single-frame images, and the features of the specified frame images are extracted based on the convolutional neural network model. The convolutional neural network model is a 3D convolutional neural network model, and the 3D convolutional neural network model includes 3 convolutional layers, 1 fully connected layer and 2 pooling layers. At least two of the 3 convolutional kernels are 3D convolutional layers, and the sizes of the convolution kernels of the two 3D convolutional layers are 7×6×3 and 7×7×3. Each node of the fully connected layer is connected to all nodes of its previous layer, and the pooling layer calculates the average value of all values ​​in the window as the representative value of the specified image area. The set 3D convolutional neural network model is used to realize efficient and accurate extraction of video features. Based on the frequency, frequency and weight of the extracted specified features, the corresponding text specified features are converted into text vectors in the vector space, and a text similarity matrix is ​​generated from all text specified feature vectors, and the cosine similarity of the text specified feature vectors in the text similarity matrix is ​​calculated. Then, accurate personalized recommendations are realized based on the cosine similarity calculation results.

Claims

1. A portrait generation and feature similarity recommendation system based on a large language model, characterized by: It includes a data acquisition module, a feature extraction module, a similarity calculation module and a recommendation module, wherein the data acquisition module is connected to the feature extraction module, the feature extraction module is connected to the similarity calculation module, and the similarity calculation module is connected to the recommendation module; The data collection module is used to collect multimodal data from social media platforms, including audio, video, text and interactive data information; The feature extraction module performs in-depth feature analysis and processing based on the collected multimodal data to extract features in multiple dimensions, including secondary features of capabilities, popularity, behavior patterns, and interaction habits; The similarity calculation module generates training samples based on the feature matrix generated by the multidimensional feature extraction module, combines the large language model to construct a similarity matrix of audio, video, text and interactive data features, calculates similarity scores, and determines similar candidate objects; The recommendation module, based on the comprehensive similarity score generated by the similarity determination module, selects the results with the highest similarity to the target from the massive amount of data and provides personalized recommendations.

2. The portrait generation and feature similarity recommendation system based on a large language model according to claim 1, characterized in that: The data acquisition module includes a data interface module and a data preprocessing module. The data interface module obtains the published audio, video, text and interactive multimodal data through a legal API interface or public data capture method; the data preprocessing module cleans, converts the format and denoises the collected raw data. The collected data specifically includes posts, comments, likes, reposts, quantity and publishing location.

3. The portrait generation and feature similarity recommendation system based on a large language model according to claim 1, characterized in that: The feature extraction module includes a text feature extraction submodule, an audio feature extraction submodule, a video feature extraction submodule and an interactive feature extraction submodule; The text feature extraction submodule uses a large language model to perform semantic analysis on the published text content to extract keywords, sentiment tendencies and topic preferences; the audio feature extraction submodule analyzes the audio content to extract voice features, sentiment expression, intonation and speech speed; the video feature extraction submodule analyzes the visual style, subject display and sentiment expression in the video content; the interactive feature extraction submodule extracts the interaction frequency, depth and pattern based on the interactive behavior; The process of text feature extraction by the text feature extraction submodule is as follows: Count the keywords in the text and the frequencies corresponding to the keywords, and calculate the information entropy of each keyword. The information entropy calculation formula is: H(X)=E[I(X)]=E[-ln(P(X))]; Where I(X) is the information content of keyword X, E[x] is the expected function of variable x, and P(x) is the probability mass function of variable x; The information gain of each keyword is calculated based on the information entropy of each keyword, and the calculation formula of the information gain is: G=H(X)-H(X|a); Among them, H(X|a) represents the conditional entropy of keyword X under given condition a; Sort by the information gain of each keyword and extract the first n keywords as the features of the text.

4. The portrait generation and feature similarity recommendation system based on a large language model according to claim 3, characterized in that: The process of audio feature extraction by the audio feature extraction submodule is as follows: Cut the audio content into several segments of specified length, perform Fourier transform on each audio segment, and obtain the spectrum information of each audio segment. The specific formula for Fourier transform on each audio segment is: Among them, s(τ) is the audio signal, w(t-τ) is the window function, and f is the audio frequency.

5. The portrait generation and feature similarity recommendation system based on a large language model according to claim 3, characterized in that: The process of video feature extraction by the video feature extraction submodule is as follows: The video is decomposed into single-frame images, and features are extracted for the specified frame images based on a convolutional neural network model, wherein the convolutional neural network model is a 3D convolutional neural network model, which includes 3 convolutional layers, 1 fully connected layer and 2 pooling layers, at least two convolution kernels of the three convolutional layers are 3D convolutional layers, and the sizes of the convolution kernels of the two 3D convolutional layers are 7×6×3 and 7×7×3, each node of the fully connected layer is connected to all nodes of its previous layer, and the pooling layer calculates the average value of all values ​​in the window as the representative value of the specified image area.

6. The portrait generation and feature similarity recommendation system based on a large language model according to claim 3, characterized in that: The feature extraction process of the interactive feature extraction submodule is as follows: The interactive data type is identified. If it is identified as text data, it is transmitted to the text feature extraction submodule for text feature extraction; if it is identified as video data, it is transmitted to the video feature extraction submodule for video feature extraction; if it is identified as audio data, it is transmitted to the audio feature extraction submodule for audio feature extraction.

7. The portrait generation and feature similarity recommendation system based on a large language model according to claim 1, characterized in that: The similarity calculation module calculates the comprehensive similarity score between them and transmits it to the recommendation module; the similarity score will be transmitted to the recommendation module, which specifically includes a training data generation module, a large language model fine-tuning module, and a similarity calculation submodule. The training data generation module screens the historical data of its published content and extracts keyword data to generate training data; the large language model fine-tuning module uses the large language model to fine-tune the training data to generate a preliminary sample for similarity calculation; the similarity calculation submodule uses a similarity algorithm to calculate the similarity matrix of each modal feature respectively.

8. The portrait generation and feature similarity recommendation system based on a large language model according to claim 7, characterized in that: The similarity matrix of each modality feature includes a text similarity matrix, an audio similarity matrix, and a video similarity matrix; Text similarity matrix: calculates text similarity by similarity of language style, topic and keywords; Audio similarity matrix: calculates the similarity of audio content by analyzing audio emotion, intonation, and speech speed; Video similarity matrix: Generate a video similarity matrix through visual performance and style analysis in the video.

9. The portrait generation and feature similarity recommendation system based on a large language model according to claim 8, characterized in that: The calculation process of the text similarity matrix is ​​as follows: extract the specified features of the text content based on the text feature extraction submodule, convert the corresponding text specified features into text vectors in the vector space based on the frequency, frequency and weight of the extracted specified features, generate a text similarity matrix from all text specified feature vectors, and calculate the cosine similarity of the text specified feature vectors in the text similarity matrix. The calculation formula of the cosine similarity is as follows: Cosine similarity = A·B / (||A||·||B||); Where A and B are two text vectors, A·B is the dot product of the two text vectors, ||A|| and ||B|| are the moduli of the two text vectors; The cosine similarity of the text similarity matrix is ​​obtained by summing the cosine similarities of the specified feature vectors of all texts.

10. A method for image generation and feature similarity recommendation based on a large language model, using the image generation and feature similarity recommendation system based on a large language model as described in any one of claims 1 to 9, characterized in that: The following steps are involved: S1: The data collection module legally collects audio, video, text and interactive data information from the social platform, and transmits it to the feature extraction module after pre-processing; S2: The feature extraction module performs feature analysis on the collected multimodal data, generates feature extraction including text, audio, video, and interactive data, and transmits the extracted features to the similarity calculation module; S3: The similarity calculation module calculates the similarity matrix of each modal feature based on the extracted features, calculates the comprehensive similarity score based on the similarity matrix of each modal feature, and transmits the similarity score result to the recommendation module; S4: The recommendation module selects the results with the highest similarity based on the similarity scoring results, provides personalized recommendations, and displays them to the user; The process of text feature extraction by the text feature extraction submodule is as follows: Count the keywords in the text and the frequencies corresponding to the keywords, and calculate the information entropy of each keyword. The information entropy calculation formula is: H(X)=E[I(X)]=E[-ln(P(X))]; Where I(X) is the information content of keyword X, E[x] is the expected function of variable x, and P(x) is the probability mass function of variable x; The information gain of each keyword is calculated based on the information entropy of each keyword, and the calculation formula of the information gain is: G=H(X)-H(X|a); Among them, H(X|a) represents the conditional entropy of keyword X under given condition a; Sort by the information gain of each keyword and extract the first n keywords as the features of the text. The process of audio feature extraction by the audio feature extraction submodule is as follows: Cut the audio content into several segments of specified length, perform Fourier transform on each audio segment, and obtain the spectrum information of each audio segment. The specific formula for Fourier transform on each audio segment is: Among them, s(τ) is the audio signal, w(t-τ) is the window function, and f is the audio frequency. The process of video feature extraction by the video feature extraction submodule is as follows: The video is decomposed into single-frame images, and features are extracted for the specified frame images based on a convolutional neural network model, wherein the convolutional neural network model is a 3D convolutional neural network model, which includes 3 convolutional layers, 1 fully connected layer and 2 pooling layers, at least two convolution kernels of the three convolutional layers are 3D convolutional layers, and the sizes of the convolution kernels of the two 3D convolutional layers are 7×6×3 and 7×7×3, each node of the fully connected layer is connected to all nodes of its previous layer, and the pooling layer calculates the average value of all values ​​in the window as the representative value of the specified image area. The feature extraction process of the interactive feature extraction submodule is as follows: The interactive data type is identified. If it is identified as text data, it is transmitted to the text feature extraction submodule for text feature extraction; if it is identified as video data, it is transmitted to the video feature extraction submodule for video feature extraction; if it is identified as audio data, it is transmitted to the audio feature extraction submodule for audio feature extraction.

Images