Star idol multi-mode response system based on emotion recognition

Abstract

The invention relates to the technical field of data analysis, in particular to a star idol multi-mode response system based on emotion recognition. The system comprises a personality analysis module, a digital duplicate generation module, a fan operation management module and an emotion recognition interaction response module, and can obtain corresponding public speech, social media data and interview content and perform personality fluctuation control analysis so as to obtain idol personality characteristics after fluctuation control; acquiring real-time multi-modal data and performing personality digital duplicate generation to generate a star idol digital duplicate model; and obtaining fan behavior data, carrying out fan behavior portrait and fan behavior interaction recommendation, and carrying out emotion recognition interaction response analysis between fans and idols in combination with the star idol digital duplication model so as to generate an emotion interaction response scene state between fan behavior interaction and star idols. According to the invention, the intelligent fan operation process of the star digital avatar can be realized through the star idol technology driven by artificial intelligence.

Description

Technical Field

[0001] The present invention relates to the field of data analysis technology, and in particular to a star idol multimodal response system based on emotion recognition. Background Art

[0002] Currently, the interaction between fans and celebrities (or characters in celebrity IP dramas) is mainly achieved through social media platforms (such as Weibo, Douyin, Instagram), fan clubs or celebrity idol systems. Existing celebrity idol technologies are mostly based on fixed personality templates and indiscriminate push strategies (such as Japan's Hatsune Miku and China's Luo Tianyi), and lack the ability to dynamically map the personality characteristics of real-life celebrities.

[0003] In addition, similar patents such as CN108052250A disclose a star idol performance data processing method and system based on multimodal interaction, wherein the method includes obtaining multimodal input data, inputting the multimodal input data into a pre-established deep learning model for matching, obtaining multimodal output data, outputting the multimodal output data, and performing by the star idol; thereby, when the current star idol's performance skill is turned on, the skill data is parsed by the cloud server, and the multimodal output data is decided, and the multimodal output data is displayed by the star idol through an imaging device, so that the star idol's performance is real-time, and the performance data corresponds to the skill content, and users can also enjoy a personalized and smooth experience, and the human-computer interaction effect is good. Although it can achieve realistic, smooth and anthropomorphic mapping effects through virtual robots, it is relatively rudimentary in terms of emotional maintenance and fan operation, and lacks personalized multimodal interaction capabilities and active contact mechanisms. Summary of the Invention

[0004] Based on this, it is necessary for the present invention to provide a star idol multimodal response system based on emotion recognition to solve at least one of the above technical problems.

[0005] To achieve the above objectives, a multimodal response system for celebrity idols based on emotion recognition is proposed, which includes the following modules:

[0006] The personality analysis module is used to obtain the public statements, social media data and interview content corresponding to the star idols, and conduct personality fluctuation control analysis based on the public statements, social media data and interview content corresponding to the star idols to obtain the personality characteristics of the idols after fluctuation control;

[0007] A digital avatar generation module is used to obtain real-time multimodal data corresponding to celebrity idols through access, and generate personality digital avatars based on the celebrity idols' personality characteristics after fluctuation control, so as to generate a celebrity idol digital avatar model;

[0008] The fan operation management module is used to obtain fan behavior data and create fan behavior profiling based on the fan behavior data to obtain fan user profiles; and to make fan behavior interaction recommendations for fan user profiles to generate fan behavior interaction operation recommendation content;

[0009] The emotion recognition and interactive response module is used to recommend content based on fan behavior interaction operations and combine the star idol digital avatar model to conduct emotion recognition and interactive response analysis between fans and idols, so as to generate the emotional interaction response scenario status between fan behavior interaction and star idols.

[0010] Furthermore, the personality analysis module includes the following functions:

[0011] Obtain public comments, social media data, and interview content corresponding to celebrity idols;

[0012] De-noise, remove duplicates, and fill in missing values on the public statements, social media data, and interview content corresponding to the celebrity idols to obtain the pre-processed public statements, social media data, and interview content of the idols;

[0013] Determine the personality 3D vector based on the pre-processed public statements, social media data, and interview content of the idols to generate the personality 3D vector of the star idols;

[0014] Based on the three-dimensional personality vector of star idols, personality baseline fluctuation control is performed to obtain the idol personality characteristics after fluctuation control.

[0015] Furthermore, the interview content includes live video, audio and expression data.

[0016] Furthermore, determining the three-dimensional personality vector based on the pre-processed idol's public statements, social media data, and interview content includes:

[0017] Perform semantic and sentiment dictionary matching on the corresponding text content in the pre-processed idols' public speeches, social media data, and interviews to obtain the corresponding sentiment word polarity and intensity metadata in the idols' speeches, data, and content;

[0018] Perform facial expression recognition analysis on the pre-processed idols’ public statements, social media data, and corresponding interview videos and expressions in the interview content to obtain the corresponding interview facial expression information in the idols’ statements, data, and content;

[0019] Extract speech intonation and semantic keywords from pre-processed idol public statements, social media data, and corresponding interview, variety show, social media, and live broadcast audio content to obtain the corresponding audio speech intonation and semantic keywords in the idol's speech, data, and content;

[0020] Based on the corresponding facial expression information in the idols' speeches, data, and content, we conduct in-depth behavioral pattern analysis of the corresponding speeches, audio, and video in the idols' public speeches, social media data, and interviews to obtain the corresponding behavioral semantic intention patterns in the idols' speeches, data, and content;

[0021] The personality three-dimensional vector is determined based on the corresponding emotional word polarity, intensity metadata, interview facial expression information, audio voice intonation and semantic keywords, and behavioral semantic intention patterns in the idol's speech, data and content, so as to analyze and calculate the personality dimensions of the star idol corresponding to the degree of publicity, affinity and humor value, and then map the publicity, affinity and humor value into the corresponding three-dimensional personality vector to generate the star idol's three-dimensional personality vector.

[0022] Furthermore, the personality baseline fluctuation control based on the celebrity idol personality three-dimensional vector includes:

[0023] Normalize the three-dimensional personality vector of the star idol at each time point to obtain the three-dimensional standard vector of the idol personality at each time point;

[0024] Draw the personality change baseline for the personality dimension corresponding to the three-dimensional standard vector of the idol personality at each time point to obtain the three-dimensional change baseline of the idol personality;

[0025] Obtaining the standard values corresponding to the three personality dimensions in the three-dimensional standard vector of the idol personality at each time point, and setting the standard values corresponding to the three personality dimensions as the personality fluctuation threshold to obtain the three-dimensional fluctuation threshold of the idol personality;

[0026] Based on the three-dimensional fluctuation threshold of idol personality and combined with the three-dimensional change baseline of idol personality, the personality baseline fluctuation control is performed on the three-dimensional vector of star idol personality at each time point, so as to compare and judge based on the three-dimensional vector of star idol personality at each time point with the corresponding three-dimensional fluctuation threshold of idol personality. If the corresponding personality dimension is less than plus or minus 1.5 times the three-dimensional fluctuation threshold of idol personality, the personality dimension is judged to be stable; otherwise, the personality dimension is judged to be unstable, and the personality baseline trigger correction is performed on the corresponding personality dimension judged to be unstable in combination with the three-dimensional change baseline of idol personality until the fluctuation of the corresponding personality dimension is stabilized and used as a personality characteristic, so as to obtain the personality characteristics of the idol after fluctuation control.

[0027] Furthermore, the digital avatar generation module includes the following functions:

[0028] Access real-time multimodal data on celebrity idols, including interviews, variety shows, social media, and live streaming data.

[0029] The idol personality traits after fluctuation control are deconstructed and coded in time and space, so as to deconstruct the personality traits corresponding to different time and context according to time series and spatial context, and convert them into digital symbol sequences that can be processed by computers to obtain the idol personality traits time and space coding dataset;

[0030] Based on the spatiotemporal coding dataset of idol personality traits, a prototype framework for the corresponding star idol digital avatar is built, including an input layer, an intermediate processing layer, and an output layer. The input layer receives the spatiotemporal coding dataset of the corresponding idol personality traits, the intermediate processing layer performs feature conversion and calculation, and the output layer generates a virtual image and behavior that matches the idol's personality.

[0031] Based on the star idol digital avatar prototype framework and combined with variational autoencoders and generative adversarial networks, the real-time multimodal data corresponding to star idols are used to fuse digital avatars to generate digital avatars. The star idol digital avatar prototype framework is combined with variational autoencoders and generative adversarial networks. According to the real-time multimodal personality characteristics obtained from the analysis of the real-time multimodal data corresponding to the star idols, the digital avatar characteristics corresponding to the virtual image and behavioral performance are generated. The corresponding personality characteristic encoding and decoding process is adjusted in combination with the variational autoencoder to better capture and reconstruct the personality characteristics corresponding to the idols. At the same time, the generator and discriminator of the generative adversarial network are optimized, so that the generator can generate more realistic star idol performances and the discriminator can more accurately judge the authenticity of the generator to generate a star idol digital avatar model.

[0032] Furthermore, the fan operation management module includes the following functions:

[0033] Obtain fan behavior data;

[0034] Perform spatiotemporal granularity analysis and discretization on fan behavior data to analyze the distribution of fan behavior data in time and space dimensions. Time can be divided into different time periods, including hours, days, and weeks. Space can be divided according to the fans' location. Continuous fan behavior data is discretized according to the corresponding spatiotemporal granularity to obtain spatiotemporal discretized fan behavior data.

[0035] Based on the spatiotemporal discretization of fan behavior data, symbolic mapping and association mining are performed on each corresponding fan behavior pattern in the fan behavior data to assign corresponding behavior symbols to each discretized fan behavior data. The behavior symbols corresponding to each fan are converted into symbol sequences, and the association rule mining algorithm is used to analyze the association relationship between the symbol sequences. The corresponding behavior pattern combinations that frequently appear for each fan are found to obtain the fan behavior pattern symbol association dataset;

[0036] Perform fan behavior profiling for each fan behavior pattern corresponding to the fan behavior data according to the fan behavior pattern symbol association dataset to obtain a fan user profile;

[0037] Conduct fan behavior interaction recommendations for fan user portraits to generate fan behavior interaction operation recommendation content.

[0038] Furthermore, the fan behavior interaction recommendation for the fan user portrait includes:

[0039] Conduct fan behavior interaction statistics on fan user portraits to obtain the frequency of fan behavior interaction;

[0040] Based on the frequency of fan behavior interaction, the fan user portrait is mined for behavior interaction interest points to obtain fan behavior interaction interest points;

[0041] Fan behavior interaction recommendations are made based on fan behavior interaction interest points to generate fan behavior interaction operation recommendation content.

[0042] Furthermore, the emotion recognition interactive response module includes the following functions:

[0043] Through the RFM model, including recency, frequency, and engagement, we conduct fan interaction identification and analysis on the recommended content of fan behavior interaction operations to identify the corresponding fan life cycle stage and the corresponding fan interaction data, including visit time, number, and spending power, to obtain the prediction results of fan interaction recommendation behavior;

[0044] Based on the prediction results of fan interaction recommendation behavior and combined with the star idol digital avatar model, star idol contact scenarios are constructed to generate star idol active contact scenarios corresponding to different fan behavior interaction conditions;

[0045] Based on the corresponding star idol active contact scenarios under different fan behavior interaction conditions and combined with the decision tree, the star idol digital avatar model is used to analyze the emotional recognition interaction response between fans and idols, so as to construct the corresponding multi-layer star idol interaction response jump logic according to the decision tree, and generate the corresponding emotional response state according to the corresponding multi-layer star idol interaction response jump logic, so as to generate the emotional interaction response scenario state between fan behavior interaction and star idols.

[0046] Furthermore, the corresponding multi-layer star idol interaction response jump logic includes: for abnormal replies, if the semantic deviation is greater than 0.7 and lasts for 10 seconds, the star idol's corresponding three most recent interactions will be called to perform abnormal repair of the star idol's corresponding personality; for active contact, if the fan has not interacted for 7 days and the sign-in is interrupted, the star idol's corresponding fan-loving mode will be activated and a customized wake-up easter egg will be sent; for sudden interruptions, if the service interruption is greater than 55 seconds and less than 65 seconds, a stage accident recall scenario apology will be generated; for sensitive topics, the star idol will be switched to the agent agent mode In response to fan doubts, if the sum of the negative sentiment analysis value and the increase in speech speed is greater than 40%, the corresponding emotional transfer of the star idol will be triggered; in response to group events, if the star idol repeats the same question 500 times within 10 minutes, the corresponding scene transfer speech will be activated; in response to cultural conflicts, if the foreign language recognition confidence is less than 0.3, the corresponding celebrity will be called to imitate the dialect style; and in response to emotional overload, if the corresponding star idol's continuous interaction exceeds 120 minutes, the corresponding backstage lounge virtual scene will be activated to gradually cool down the interaction.

[0047] Beneficial effects of the present invention:

[0048] The present invention proposes a multimodal response system for celebrity idols based on emotion recognition. The system consists of a personality analysis module, a digital avatar generation module, a fan operation and management module, and an emotion recognition and interactive response module. Compared with existing technologies, the present invention has the advantage of collecting multidimensional data such as celebrity idols' public statements, social media updates, and interviews, enabling a deep understanding of their behavior, emotional expression, values, and personality traits. This data provides a foundation for analyzing the idols' personality traits, helping to capture the changes in their public image and delve into their behavioral patterns and emotional fluctuations in different scenarios. Subsequently, a personality fluctuation control algorithm is applied to fine-tune and smooth this data. Fluctuation control effectively eliminates short-term emotional fluctuations or public misunderstandings that may occur on social media or in interviews, making their personality more stable and consistent with the long-term strategy of idol development. This allows the idols' virtual personalities to remain within an appropriate, continuous fluctuation range, preventing excessive emotional fluctuations caused by external events. This allows for dynamic mapping of the real celebrity's personality traits. This controlled personality not only maintains the idols' unique charm but also avoids excessive emotional fluctuations, thereby ensuring the long-term stability of the idols' image and the continued recognition of fans. Secondly, the real-time performance of star idols is captured and analyzed through multimodal data (such as video, audio, images, social interaction, etc.), which can include live video, social platform content, fan interaction, and even movies or variety shows starring idols. Through efficient data collection and processing technology, various information such as idols' behavior, voice, facial expressions, body language, etc. are converted into data form, and a digital avatar model of the idol is constructed through advanced artificial intelligence algorithms. This digital avatar will form an accurate and highly realistic image of the star idol based on the idol's historical behavior patterns, speech and emotional fluctuations. It can achieve a high degree of unity between the idol's personality and public image, and enable it to maintain consistent emotions and behavior patterns during interaction, making the idol's image more vivid and rich, and the interaction with fans more natural and real. Then, by real-time tracking of fans' behavioral data on social media, fan groups and other interactive platforms (such as likes, comments, shares, viewing time, emotional tendencies, etc.), a detailed fan behavior portrait can be constructed. The fan portrait is a detailed analysis of the fan group's behavioral characteristics, interests, interaction habits, emotional appeals and other dimensions, which helps to accurately understand the fans' preferences, emotional needs and interaction patterns with idols. On this basis, fan behavior interaction recommendations are made. This process uses big data analysis and machine learning models, combined with fans' historical behavior and preferences, to automatically generate personalized content recommendations. Through these recommendations, fans can be exposed to idol content, interactive forms and marketing activities that they are more interested in, thereby increasing fans' activity and participation, and thus better maintaining the emotional and operational management between star idols and fans.Finally, through the emotional interaction between fan behavior and celebrity idols, more humane communication and response can be achieved. Based on the interactive operation recommendation content based on fan behavior and the celebrity idol digital avatar model, it is possible to identify and interpret fans' emotional needs. Through real-time emotion recognition technology, fans' emotional fluctuations, behavioral patterns, and psychological states can be analyzed. This technology can accurately capture the emotional changes of fans during the interaction process and automatically generate corresponding emotional response strategies for celebrity idols based on these changes. These response strategies can be reflected in the celebrity idols' interactive behavior. For example, the idols' responses can be adjusted through tone, expression, and speech content to meet the fans' emotional needs. For example, when fans express joy, the idols' responses will be more positive and friendly. When fans express doubts or dissatisfaction, the idols can provide emotional support through comfort and answers. This makes the relationship between fans and idols closer. Fans not only feel the idols' care and recognition, but also gain a deeper sense of participation and satisfaction through this emotional connection, thus better realizing the active contact mechanism between celebrity idols and fans. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Other features, objects and advantages of the present invention will become more apparent upon reading the detailed description of non-limiting embodiments thereof made with reference to the following drawings:

[0050] Figure 1 Schematic diagram of the modules of the celebrity idol multimodal response system based on emotion recognition of the present invention;

[0051] Figure 2 for Figure 1 Schematic diagram of the functional flow of the personality analysis module;

[0052] Figure 3 for Figure 1 Schematic diagram of the functional flow of the digital avatar generation module;

[0053] Figure 4 This is a schematic diagram of the coding corresponding to the three-dimensional vector of the celebrity idol personality of the present invention. DETAILED DESCRIPTION

[0054] The following is a clear and complete description of the technical system of the present invention in conjunction with the accompanying drawings. It is obvious that the embodiments described are part of the embodiments of the present invention, but not all of them. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without making any creative efforts are within the scope of protection of the present invention.

[0055] In addition, the accompanying drawings are merely schematic illustrations of the present invention and are not necessarily drawn to scale. Identical reference numerals in the figures denote identical or similar parts, and thus repetitive descriptions thereof will be omitted. Some of the block diagrams shown in the accompanying drawings are functional entities that do not necessarily correspond to physically or logically independent entities. These functional entities may be implemented in software, in one or more hardware modules or integrated circuits, or in different networks and / or processor systems and / or microcontroller systems.

[0056] It should be understood that although the terms "first," "second," and the like may be used herein to describe various elements, these elements should not be limited by these terms. These terms are used solely to distinguish one element from another. For example, a first element may be referred to as a second element, and similarly, a second element may be referred to as a first element, without departing from the scope of the exemplary embodiments. The term "and / or" as used herein includes any and all combinations of one or more of the listed associated items.

[0057] To achieve this, please refer to Figures 1 to 4 The present invention provides a star idol multimodal response system based on emotion recognition, which includes the following modules:

[0058] The personality analysis module is used to obtain the public statements, social media data and interview content corresponding to the star idols, and conduct personality fluctuation control analysis based on the public statements, social media data and interview content corresponding to the star idols to obtain the personality characteristics of the idols after fluctuation control;

[0059] A digital avatar generation module is used to obtain real-time multimodal data corresponding to celebrity idols through access, and generate personality digital avatars based on the celebrity idols' personality characteristics after fluctuation control, so as to generate a celebrity idol digital avatar model;

[0060] The fan operation management module is used to obtain fan behavior data and create fan behavior profiling based on the fan behavior data to obtain fan user profiles; and to make fan behavior interaction recommendations for fan user profiles to generate fan behavior interaction operation recommendation content;

[0061] The emotion recognition and interactive response module is used to recommend content based on fan behavior interaction operations and combine the star idol digital avatar model to conduct emotion recognition and interactive response analysis between fans and idols, so as to generate the emotional interaction response scenario status between fan behavior interaction and star idols.

[0062] In the embodiment of the present invention, please refer to Figure 1FIG. 1 is a schematic diagram of a module of a star idol multimodal response system based on emotion recognition according to the present invention. In this example, the star idol multimodal response system based on emotion recognition includes the following modules:

[0063] S1: Personality analysis module, used to obtain the public statements, social media data and interview content corresponding to the star idol, and perform personality fluctuation control analysis based on the public statements, social media data and interview content corresponding to the star idol to obtain the idol's personality characteristics after fluctuation control;

[0064] In an embodiment of the present invention, the public comments, social media data and interview content corresponding to celebrity idols (or celebrity IP character idols in dramas) are obtained by utilizing a specific platform interface. For public comments, Python's Scrapy framework is used to extract data from major news websites and information platforms. For example, the name of the celebrity idol is entered in the Sina News search bar, and search rules are set to traverse the search results page to extract the news text content containing the celebrity's comments. For social media data, if the celebrity is active on Weibo, the Weibo open platform interface is applied for, and Python's Tweepy library is used to extract the Weibo content, comments and likes data posted by the celebrity based on his Weibo account ID. When obtaining interview content, the celebrity name and the keyword "interview" are searched on video websites such as Tencent Video and Youku, and the corresponding interview video link is obtained. The video is downloaded locally with the help of the video download tool you-get. Subsequently, natural language processing technology and computer vision technology are used to process these data. For text data, a pre-trained language is used. Models such as BERT perform semantic analysis and extract emotional features. For interview videos, a facial expression recognition model based on a convolutional neural network is used to identify facial expressions. Through comprehensive analysis of this multimodal data, the three-dimensional personality vector of the celebrity idol is determined, including, for example, the three dimensions of assertiveness, likeability, and humor. Statistical methods are used to control the baseline fluctuation of the personality three-dimensional vector. The three-dimensional personality vector data of the celebrity idol from multiple different periods is collected, and the standard deviation is calculated to obtain the personality baseline vector. The value corresponding to each personality dimension is compared with the range of ±1.5 times the standard deviation. If the value corresponding to the personality dimension is within the threshold range corresponding to ±1.5 times the standard deviation, the current vector is considered to be within a reasonable fluctuation range and is directly used as the personality characteristic of the idol after fluctuation control. If the value corresponding to the personality dimension is not within the threshold range corresponding to ±1.5 times the standard deviation, a new vector is generated from the personality baseline vector and the current vector through linear interpolation to make it closer to the personality baseline, ultimately obtaining the personality characteristic of the idol after fluctuation control.

[0065] S2: A digital avatar generation module, which is used to obtain real-time multimodal data corresponding to celebrity idols through access, and generate personality digital avatars based on the celebrity idols' personality characteristics after fluctuation control, so as to generate a celebrity idol digital avatar model;

[0066] In an embodiment of the present invention, during the data access and model generation phase, a real-time data acquisition interface is used to obtain real-time multimodal data corresponding to celebrity idols. For example, the video, audio, and barrage data of celebrity idols during live broadcasts are obtained in real time through the API interface provided by the live broadcast platform; the latest dynamics and fan interaction data released by celebrities are obtained through the real-time data push interface of the social media platform. In addition, by using Python's data analysis and machine learning library, a personality digital avatar is generated for the real-time multimodal data based on the personality characteristics of the idols after fluctuation control, so as to read the personality characteristic data and integrate it into the generation model as prior knowledge. The generation model is built by using the deep learning framework TensorFlow, and the model input is used. The input is real-time multimodal data, which is processed by a multi-layer neural network to output a virtual image and behavioral performance that matches the idol's personality. For example, when live video data is input, the model optimizes and simulates the expressions and movements in the video according to the idol's personality characteristics to generate a virtual image performance that is more in line with the idol's personality. During the training process, variational autoencoders and generative adversarial networks are used to improve model performance. The variational autoencoder is used to better capture and reconstruct the idol's personality characteristics and optimize the encoding and decoding process of the input data; the generative adversarial network generates star idol performances through the generator, and the discriminator judges its authenticity. The two are continuously optimized to make the generated star idol performance more realistic. After multiple rounds of training, a star idol digital avatar model is finally generated.

[0067] S3: Fan operation management module, used to obtain fan behavior data and create fan behavior profiling based on the fan behavior data to obtain fan user profiles; and to make fan behavior interaction recommendations based on the fan user profiles to generate fan behavior interaction operation recommendation content;

[0068] In an embodiment of the present invention, fan behavior data is obtained by utilizing the background data interface of social media platforms, official websites of celebrity idols and related applications. For social media platforms such as Weibo, the API interface provided by the Weibo open platform is used with the help of Python's Tweepy library to obtain fans' attention time, Weibo content likes, Weibo content comments and Weibo forwarding behavior data based on the fan list of the celebrity idol's official account. For the celebrity idol's official website, the website server's log recording function is used to collect data such as the time fans visit the website, the pages browsed, and the length of stay. In terms of applications, if the celebrity idol has an exclusive APP, the APP's built-in data collection module is used to record fans' operating behaviors in the APP, such as watching celebrity idol live broadcasts, purchasing virtual goods, participating in interactive activities, etc., and these fan behavior data are stored in the database. The Python pandas library is used to perform spatiotemporal granularity analysis and discretization on the fan behavior data, and the time is subdivided into hours, days and weeks. The dt.hour, dt.date, and dt.d are used to store the fan behavior data. Functions such as t.week are used to extract temporal features; space is divided according to the region where the fans are located. It is assumed that the regional information is stored in the "region" field in the database table. At the same time, the continuous fan behavior data is discretized according to the time and space granularity, organized into new tables and stored in the corresponding database table. Based on these discretized data, the Apriori association rule mining algorithm is used, and with the help of Python's mlxtend library, each discretized fan behavior data is assigned a corresponding behavior symbol, and the behavior symbol corresponding to each fan is converted into a symbol sequence. The correlation between the symbol sequences is analyzed to find the frequently occurring behavior pattern combinations of each fan, and obtain the fan behavior pattern symbol association data set. Finally, based on this data set, a behavior portrait is constructed for each fan. The portrait includes the fan's active time, main behavior pattern, location and other features, and a recommendation algorithm based on collaborative filtering is used, such as the KNNWithMeans algorithm in Python's Surprise library, to generate personalized fan behavior interaction operation recommendation content for each fan based on the fan user portrait.

[0069] S4: Emotional recognition and interactive response module, used to recommend content based on fan behavior interaction operations and combine the star idol digital avatar model to perform emotional recognition and interactive response analysis between fans and idols, so as to generate the emotional interaction response scenario status between fan behavior interaction and star idols.

[0070] In an embodiment of the present invention, by using Python's data analysis and machine learning library in the emotion recognition interactive response stage, based on the fan behavior interactive operation recommendation content and combined with the star idol digital avatar model to perform emotion recognition interactive response analysis between fans and idols, the recommended content data is read and the star idol digital avatar model is loaded. During the interaction between the star idol and the fans, the fan reply content and interaction status are monitored in real time, so as to identify abnormal replies by using semantic analysis tools in natural language processing technology, such as HanLP library, to calculate the deviation between the semantics of the fan reply and the normal context; to identify sensitive topics by using keyword matching and sentiment analysis tools; to calculate the negative value of the fan reply by sentiment analysis tools, and to monitor the fan language at the same time. The speed increase is used to judge the situation of fans' questioning, etc. Different interaction situations are processed according to the pre-set decision tree logic. For example, for abnormal replies, if the semantic deviation is greater than 0.7 and lasts for 10 seconds, the latest three interaction data are retrieved from the star idol's interaction record database, and the corresponding personality abnormality repair operations are performed according to the personality set in the star idol's digital avatar model, such as changing the star idol's language style and expression, etc. These fan behavior interactions and the emotional interaction response scene status between the star idol are recorded in the corresponding database table, listed as fan ID, interaction scene, response measures, response results and other fields. By continuously optimizing the decision tree logic and model parameters, the emotional interaction experience between star idols and fans will be ultimately improved.

[0071] Furthermore, the personality analysis module includes the following functions:

[0072] Obtain public comments, social media data, and interview content corresponding to celebrity idols;

[0073] De-noise, remove duplicates, and fill in missing values on the public statements, social media data, and interview content corresponding to the celebrity idols to obtain the pre-processed public statements, social media data, and interview content of the idols;

[0074] Determine the personality 3D vector based on the pre-processed public statements, social media data, and interview content of the idols to generate the personality 3D vector of the star idols;

[0075] Based on the three-dimensional personality vector of star idols, personality baseline fluctuation control is performed to obtain the idol personality characteristics after fluctuation control.

[0076] As an embodiment of the present invention, refer to Figure 2 As shown, Figure 1 Schematic diagram of the functional flow of the personality analysis module in this embodiment. The personality analysis module includes the following functions:

[0077] S11: Obtain the corresponding public statements, social media data and interview content of celebrity idols;

[0078] In an embodiment of the present invention, Python's Scrapy framework is used to obtain public comments, social media data, and interview content corresponding to celebrity idols. For public comments, specific search rules are set on major news websites and information platforms. Taking a celebrity idol as an example, the name of the celebrity is entered in the Baidu News search bar and keywords such as "comments" are added to traverse the search result page according to the set rules to extract the news content containing the celebrity's comments. For social media data, if the celebrity is active on the Weibo platform, after obtaining authorization by applying for the Weibo open platform interface, Python's Tweepy library (similar to the Weibo data acquisition library) is used to crawl the Weibo content, comments, and likes data posted by the celebrity according to his Weibo account ID. When obtaining interview content, On video websites such as Tencent Video and Youku, by searching for the star’s name and the keyword “interview”, the corresponding interview video link is obtained, and with the help of video download tools such as you-get, the interview video is downloaded locally. At the same time, audio extraction software such as FFmpeg is used to separate the audio file from the downloaded video. For expression data, computer vision technology is used, and the OpenCV library is used to analyze the interview video frame by frame. Through the facial key point detection algorithm, such as the 68 facial key point detection model in the dlib library, the key parts of the face are located, the coordinate data of the expression-related feature points are extracted, and the expression status information at different time points is recorded. The obtained data are stored in different folders, and the folders are named after the star idols for easy subsequent processing.

[0079] S12: De-noise, remove duplicates, and fill in missing values for the public statements, social media data, and interview content corresponding to the celebrity idols to obtain the pre-processed public statements, social media data, and interview content of the idols;

[0080] In an embodiment of the present invention, Python's pandas library is used to denoise, remove duplicates, and supplement missing values for public speech and social media data. Taking public speech data as an example, it is assumed that the data is stored in a CSV file, and each line represents a speech. During denoising, regular expressions are used to clean up special characters and garbled characters in the text, such as using the re module to delete HTML tags and special symbols in the text. The deduplication operation is performed through the drop_duplicates function of pandas. According to the speech content column, duplicate speech records are removed. For missing value supplementation, if the content of a speech is missing, the previous and next speech records are analyzed. Meaning, using the text generation model in natural language processing technology, such as GPT-2 (local deployment version), input the relevant text before and after to generate supplementary content. For the audio data in the interview content, if there are audio interruptions, noise, etc., use the audio processing software Audacity to perform denoising processing, and remove background noise by adjusting the noise reduction parameters. For expression data, if there is a frame of expression data missing, use the interpolation method to linearly calculate the coordinate value of the missing frame based on the coordinates of the expression feature points of the previous and next adjacent frames to supplement it. After a series of processing, the pre-processed idol public statements, social media data and interview content will be obtained and stored in a new folder.

[0081] S13: Determine the personality three-dimensional vector based on the pre-processed public statements, social media data, and interview content of the idol to generate the celebrity idol personality three-dimensional vector;

[0082] In an embodiment of the present invention, the three-dimensional personality vector is determined based on various types of pre-processed data using natural language processing technology and computer vision technology. For the text content in public speeches and social media data, a pre-trained language model, such as the BERT model (locally deployed), is used to input the text into the model to obtain a semantic vector representation of the text. For the expression data in the interview video, a deep learning model, such as an expression recognition model based on a convolutional neural network (pre-trained), is used to convert the coordinate data of the expression feature point into an expression vector. For audio data, the emotional features of the audio, such as speaking speed and intonation, are extracted and converted into audio emotion vectors using an audio feature extraction tool, such as the Librosa library. Then, the text semantic vector, expression vector and audio emotion vector are fused using a weighted average method. For example, the weight of the text semantic vector is set to 0.5, the weight of the expression vector is set to 0.3, and the weight of the audio emotion vector is set to 0.2. Assuming that the text semantic vector is [0.2, 0.3, 0.1], the expression vector is [0.1, 0.2, 0.3], and the audio emotion vector is [0.3, 0.1, 0.2], the fused vector is [0.2×0.5+0.1×0.3+0.3×0.2, 0.3×0.5+0.2×0.3+0.1×0.2, 0.1×0.5+0.3×0.3+0.2×0.2]=[0.21, 0.23, 0.18]. The fused vector is further processed by a fully connected neural network for dimensionality reduction to obtain a three-dimensional vector, and finally a three-dimensional vector of the star idol personality is generated.

[0083] S14: Perform personality baseline fluctuation control based on the star idol personality three-dimensional vector to obtain the idol personality characteristics after fluctuation control.

[0084] In an embodiment of the present invention, a statistical method is used to control personality baseline fluctuations. First, the three-dimensional personality vector data of the star idol at multiple different periods is collected. Assuming that data from 10 periods is collected, the standard deviation of these vectors is calculated to obtain the personality baseline vector. For example, these 10 three-dimensional vectors are [0.2, 0.3, 0.1], [0.22, 0.28, 0.12], etc., and the personality baseline vector obtained by calculating the standard deviation is [0.21, 0.29, 0.11]. For the newly obtained three-dimensional personality vector of the star idol, such as [0.23, 0.32, 0.09], the value corresponding to each personality dimension is compared with the value between plus and minus 1.5 times the standard deviation. If the value corresponding to the personality dimension is within the threshold range corresponding to plus or minus 1.5 times the standard deviation, the current vector is considered to be within a reasonable fluctuation range, and the current vector is directly used as the personality characteristic of the idol after fluctuation control; if the value corresponding to the personality dimension is not within the threshold range corresponding to plus or minus 1.5 times the standard deviation, a new vector is generated based on the personality baseline vector and the current vector through linear interpolation to make it closer to the personality baseline. This is used as the personality characteristic of the idol after fluctuation control. The personality characteristic of the idol after fluctuation control is updated and stored in the original text file, overwriting the original three-dimensional personality vector data, providing a stable personality characteristic basis for the subsequent multimodal response of star idols based on emotion recognition.

[0085] Furthermore, the interview content includes live video, audio and expression data.

[0086] Furthermore, determining the three-dimensional personality vector based on the pre-processed idol's public statements, social media data, and interview content includes:

[0087] Perform semantic and sentiment dictionary matching on the corresponding text content in the pre-processed idols' public speeches, social media data, and interviews to obtain the corresponding sentiment word polarity and intensity metadata in the idols' speeches, data, and content;

[0088] In an embodiment of the present invention, Python is combined with the natural language processing toolkit NLTK and a pre-built sentiment dictionary to perform semantic and sentiment dictionary matching on the corresponding text contents in the pre-processed idol public speeches, social media data and interview content. Taking public speech data as an example, the pandas library is used to read the file content. For each speech text, the NLTK word segmentation tool is first used to split the text into individual words. For example, the speech "Today's activities are very interesting and I am very happy" is segmented into ["today", "of", "activity", "very", "interesting", "I", "very", "happy"], Then, we traverse each word and match it with the sentiment dictionary. Assuming that the sentiment dictionary is stored in the form of a Python dictionary, the sentiment word polarity corresponding to "interesting" is "positive" and the intensity is 0.8; the sentiment word polarity corresponding to "happy" is "positive" and the intensity is 0.9. Through matching, we obtain the corresponding sentiment word polarity and intensity metadata in the speech and organize them into a list form, such as [("interesting", "positive", 0.8), ("happy", "positive", 0.9)]. The same operation is performed on the text in the social media data and interview content. Finally, we obtain the corresponding sentiment word polarity and intensity metadata in the idol's speech, data, and content.

[0089] Preferably, facial expression recognition analysis is performed on the pre-processed idol public speeches, social media data, and corresponding interview videos and expressions in the interview content to obtain the corresponding interview facial expression information in the idol speeches, data, and content;

[0090] In an embodiment of the present invention, a facial expression recognition tool based on deep learning is used to perform facial expression recognition analysis on the corresponding interview videos and expressions in the pre-processed idol public speeches, social media data, and interview content. By using a pre-trained facial expression recognition model, such as the FER2013 model (local deployment) based on a convolutional neural network, for each interview video, the OpenCV library is used to read the video frames in sequence. For example, for an interview video with a duration of 1 minute and a frame rate of 30 frames per second, a total of 1800 frames are read, and each frame image is input into the facial expression recognition model. The model outputs the category of the facial expression in the frame image, such as "happy", "surprised", "angry", etc. For the feature point coordinate data obtained by the facial key point detection algorithm in the expression data, it is also converted into an input format acceptable to the model and input into the model for expression category judgment. The facial expression category corresponding to each video and expression data is organized into a table form, and the time point and category of the expression appearance are recorded. For example, at the 10th second of the video, the expression is detected as "smiling", and the corresponding interview facial expression information in the idol speech, data, and content is finally obtained.

[0091] Preferably, voice intonation and semantic keywords are extracted from the pre-processed idol public speeches, social media data, and interview content corresponding to interviews, variety shows, social media, and live broadcast audio content to obtain the audio voice intonation and semantic keywords corresponding to the idol speeches, data, and content;

[0092] In an embodiment of the present invention, the audio processing library Librosa and natural language processing tools are used to extract speech intonation and semantic keywords from the pre-processed idol public speeches, social media data, and interview content corresponding to interviews, variety shows, social media, and live broadcast audio content. The audio files are read using the Librosa library, and speech intonation information is extracted by calculating features such as the Mel-Frequency Cepstral Coefficients (MFCC) of the audio. For example, the mean and variance of the MFCC features are calculated to reflect changes in speech intonation. For semantic keyword extraction, the audio is first converted into text content using a speech-to-text tool, such as the Baidu Speech Recognition API (applied for and obtained authorization). Then, a keyword extraction algorithm in natural language processing technology, such as the TextRank algorithm (using the Python TextRank4ZH library), is used to process the converted text. For example, for a text converted from an interview audio, "I think this work is very creative, I hope everyone will like it," the TextRank algorithm extracts semantic keywords such as "work," "creativity," and "like." The audio speech intonation and semantic keywords corresponding to the idol's speech, data, and content are organized into a table, and ultimately the audio speech intonation and semantic keywords corresponding to the idol's speech, data, and content are obtained.

[0093] Preferably, based on the corresponding facial expression information in the idol's speech, data and content, a deep behavioral pattern analysis is performed on the corresponding speech, audio and video in the idol's public speech, social media data and interview content to obtain the corresponding behavioral semantic intention pattern in the idol's speech, data and content;

[0094] In an embodiment of the present invention, a behavioral pattern analysis algorithm is used to perform an in-depth behavioral pattern analysis on the corresponding speeches, audio and video in the pre-processed public speeches, social media data and interview content of the idol, and the corresponding speeches, audio and video content are read from the corresponding text, audio and video data storage files. For example, when it is detected that the facial expression of the interview at a certain moment is "laughing", and the corresponding speech content at that moment is telling an interesting story, and the voice tone in the audio is also relatively cheerful, through the pre-established behavioral pattern analysis rules, such as when the facial expression is "laughing" and the speech content is telling an interesting story, and the voice tone is cheerful, the behavioral semantic intention pattern is determined to be "sharing humor". Such analysis is performed on all speeches, audio and video data, and the corresponding behavioral semantic intention patterns in the idol's speeches, data and content are organized into a list form.

[0095] Preferably, the personality three-dimensional vector is determined based on the corresponding emotional word polarity, intensity metadata, interview facial expression information, audio voice intonation and semantic keywords and behavioral semantic intention patterns in the idol's speech, data and content, so as to analyze and calculate the personality dimensions of the star idol corresponding to the publicity, affinity and humor value, and map the publicity, affinity and humor value into the corresponding personality three-dimensional vector to generate the star idol personality three-dimensional vector.

[0096] In the embodiment of the present invention, a mathematical model is used to comprehensively analyze the corresponding emotional word polarity, intensity metadata, interview facial expression information, audio voice intonation and semantic keywords and behavioral semantic intention patterns in idol speeches, data and content to determine the personality dimensions of star idols corresponding to publicity, affinity and humor value (such as Figure 4 For example, if there are many emotional words with positive polarity and high intensity, friendly expressions such as "smile" are often seen in interview facial expressions, the audio voice tone is relatively gentle, and the behavioral semantic intention pattern is mostly "caring for others", etc., a preset scoring mechanism is used, such as multiplying the total intensity of positive emotional words by 0.4, multiplying the frequency of friendly expressions by 0.3, multiplying the characteristic value of gentle voice tone by 0.2, and multiplying the number of occurrences of the behavioral semantic intention pattern of "caring for others" by 0.1, to calculate the affinity score. Similarly, the extroversion (for example, frequent use of emphatic words and large body movements in speech) and humor (for example, use of puns and exaggerated expressions) scores are calculated. Assuming that the extroversion is calculated to be 0.6, the affinity is 0.7, and the humor is 0.5, these three values are mapped to form the corresponding personality three-dimensional vector [0.6, 0.7, 0.5] to generate the celebrity idol personality three-dimensional vector, which is stored in a text file named after the celebrity idol, overwriting the previously existing personality three-dimensional vector file, and finally generating the celebrity idol personality three-dimensional vector.

[0097] Furthermore, the personality baseline fluctuation control based on the celebrity idol personality three-dimensional vector includes:

[0098] Normalize the three-dimensional personality vector of the star idol at each time point to obtain the three-dimensional standard vector of the idol personality at each time point;

[0099] In an embodiment of the present invention, the three-dimensional personality vectors of celebrity idols at each time point are standardized by using the Python scikit-learn library. The pandas library is used to read file data, where each row represents the three-dimensional personality vector at a time point. For example, a row of data is [0.6, 0.7, 0.5]. The StandardScaler function in the scikit-learn library is used to create a StandardScaler object. The three-dimensional personality vectors at all read time points are combined into a two-dimensional array and input into the fit_transform method of the StandardScaler object. The method calculates the mean and standard deviation of each dimension and standardizes the data. For example, assuming that the first dimension (exposure) in the three-dimensional personality vector has a mean of 0.5 and a standard deviation of 0.1, the extroversion value of 0.6 in the vector [0.6, 0.7, 0.5] is standardized to (0.6-0.5) / 0.1=1. This standardization operation is performed on the three-dimensional personality vectors at all time points, ultimately obtaining the three-dimensional standard personality vector of the idol at each time point.

[0100] Preferably, a personality change baseline is drawn for the personality dimension corresponding to the three-dimensional standard vector of the idol personality at each time point to obtain the three-dimensional change baseline of the idol personality;

[0101] In an embodiment of the present invention, by using Python's matplotlib library, the personality change baseline is plotted for the personality dimensions corresponding to the three-dimensional standard vector of the idol personality at each time point, so as to extract the three dimensions of the three-dimensional standard vector of personality, namely, the standard values corresponding to the publicity, affinity and humor value. For example, the standard values of publicity at different time points are extracted as [0.8, 1.2, 0.9, 1.1], etc., and by using the plot function of the matplotlib library, with the time point as the horizontal coordinate and the standard value of the personality dimension as the vertical coordinate, the curves of publicity, affinity and humor value changing with time are plotted respectively. For example, for publicity, the curve color is set to red and the label is "publicity change curve". The following code is used to implement it:

[0102] import matplotlib.pyplot as plt

[0103] time_points=[1,2,3,4]#Assumed time points

[0104] extroversion_values=[0.8,1.2,0.9,1.1]

[0105] plt.plot(time_points,extroversion_values,'r',label='Extroversion change curve')

[0106] Similarly, draw the change curves of affinity and humor value, integrate these curves into a chart, set the chart title to "Three-dimensional change baseline of star idol personality", the horizontal axis label to "time point", the vertical axis label to "personality dimension standard value", and add a legend. Save the drawn chart as an image file, and finally get the three-dimensional change baseline of idol personality.

[0107] Preferably, the standard values corresponding to the three personality dimensions are obtained by the personality dimensions corresponding to the three-dimensional standard vector of the idol personality at each time point, and the standard values corresponding to the three personality dimensions are set as the personality fluctuation threshold to obtain the three-dimensional fluctuation threshold of the idol personality;

[0108] In an embodiment of the present invention, by reading the personality dimension data corresponding to the three-dimensional standard vector of the idol personality at each time point, the standard deviations of the three personality dimensions (publicity, affinity, and humor value) are calculated respectively. For example, the standard deviation of all standard values of the publicity dimension is calculated, assuming that the calculation result is 0.15; the standard deviation of all standard values of the affinity dimension is calculated, assuming that the calculation result is 0.12; the standard deviation of all standard values of the humor value dimension is calculated, assuming that the calculation result is 0.1. These three standard deviations are used as the personality fluctuation thresholds corresponding to the three personality dimensions, that is, the publicity personality fluctuation threshold is 0.15, the affinity personality fluctuation threshold is 0.12, and the humor value personality fluctuation threshold is 0.1, and finally the three-dimensional fluctuation threshold of the idol personality is obtained.

[0109] Preferably, based on the three-dimensional fluctuation threshold of the idol personality and in combination with the three-dimensional change baseline of the idol personality, the personality baseline fluctuation control is performed on the three-dimensional vector of the star idol personality at each time point, so as to compare and judge based on the three-dimensional vector of the star idol personality at each time point with the corresponding three-dimensional fluctuation threshold of the idol personality. If the corresponding personality dimension is less than plus or minus 1.5 times the three-dimensional fluctuation threshold of the idol personality, the personality dimension is determined to be stable; otherwise, the personality dimension is determined to be unstable, and the personality baseline trigger correction is performed on the corresponding personality dimension determined to be unstable in combination with the three-dimensional change baseline of the idol personality until the fluctuation of the corresponding personality dimension is stabilized and used as a personality characteristic, so as to obtain the personality characteristic of the idol after fluctuation control.

[0110] In the embodiment of the present invention, a program is written in Python to perform personality baseline fluctuation control on the three-dimensional vector of the star idol personality at each time point based on the three-dimensional fluctuation threshold of the idol personality and in combination with the three-dimensional change baseline of the idol personality. Taking the three-dimensional vector of the star idol personality at a certain time point [0.7, 0.6, 0.4] as an example, the personality fluctuation threshold of publicity is 0.15, the personality fluctuation threshold of affinity is 0.12, and the personality fluctuation threshold of humor value is 0.1. The difference between each dimension in the vector and the corresponding three-dimensional personality change baseline value is calculated and compared with the plus or minus 1.5 times the personality fluctuation threshold. Assuming that at this time point The baseline value of the personality dimension of assertiveness is 0.6, the difference is 0.7-0.6=0.1, and the 1.5-fold fluctuation threshold of the personality dimension of assertiveness is 1.5×0.15=0.225. 0.1<0.225, so the personality dimension of assertiveness is determined to be stable. Similarly, the dimensions of affinity and humor are judged. If a dimension is unstable, for example, the calculated difference of the humor dimension is greater than 1.5-fold the personality fluctuation threshold of humor, the value of that dimension is adjusted by linear interpolation according to the three-dimensional baseline of the idol personality, so that it is close to the baseline value until the fluctuation of that dimension is stable. After such processing, the idol personality characteristics after fluctuation control are finally obtained.

[0111] Furthermore, the digital avatar generation module includes the following functions:

[0112] Access real-time multimodal data on celebrity idols, including interviews, variety shows, social media, and live streaming data.

[0113] The idol personality traits after fluctuation control are deconstructed and coded in time and space, so as to deconstruct the personality traits corresponding to different time and context according to time series and spatial context, and convert them into digital symbol sequences that can be processed by computers to obtain the idol personality traits time and space coding dataset;

[0114] Based on the spatiotemporal coding dataset of idol personality traits, a prototype framework for the corresponding star idol digital avatar is built, including an input layer, an intermediate processing layer, and an output layer. The input layer receives the spatiotemporal coding dataset of the corresponding idol personality traits, the intermediate processing layer performs feature conversion and calculation, and the output layer generates a virtual image and behavior that matches the idol's personality.

[0115] Based on the star idol digital avatar prototype framework and combined with variational autoencoders and generative adversarial networks, the real-time multimodal data corresponding to star idols are used to fuse digital avatars to generate digital avatars. The star idol digital avatar prototype framework is combined with variational autoencoders and generative adversarial networks. According to the real-time multimodal personality characteristics obtained from the analysis of the real-time multimodal data corresponding to the star idols, the digital avatar characteristics corresponding to the virtual image and behavioral performance are generated. The corresponding personality characteristic encoding and decoding process is adjusted in combination with the variational autoencoder to better capture and reconstruct the personality characteristics corresponding to the idols. At the same time, the generator and discriminator of the generative adversarial network are optimized, so that the generator can generate more realistic star idol performances and the discriminator can more accurately judge the authenticity of the generator to generate a star idol digital avatar model.

[0116] As an embodiment of the present invention, refer to Figure 3 As shown, Figure 1 Schematic diagram of the functional flow of the digital avatar generation module in this embodiment. The digital avatar generation module includes the following functions:

[0117] S21: Access real-time multimodal data corresponding to celebrity idols, including real-time interviews, variety shows, social media, and live broadcast dynamic data corresponding to celebrity idols;

[0118] In an embodiment of the present invention, real-time multimodal data corresponding to celebrity idols is obtained by using platform interface calls. For real-time interview data corresponding to celebrity idols, rules are set on video websites such as iQiyi and Bilibili using the Python Scrapy framework to monitor the release of celebrity idol-related interview videos in real time. For example, every 10 minutes, the celebrity's name and the keyword "interview" are entered in the website search bar. If a new video is released, the video link, title, release time, and other information are immediately captured, and the video is downloaded locally using the video download tool you-get. For variety show data, if the variety show in which the celebrity participates is broadcast on Mango TV, after applying for the Mango TV open platform interface and obtaining authorization, the corresponding Python API call library is used to obtain the latest episode content of the variety show, the celebrity's clips in the show, and other data in real time based on the variety show name or program ID of the celebrity. For social media data, taking Weibo as an example, the Weibo open platform interface is used to use the Python Tweepy library to capture the celebrity's latest Weibo content, pictures, videos, and fan comments and likes based on the celebrity's Weibo account ID. For live streaming dynamic data, if a celebrity broadcasts live on the Douyin platform, we can use the live streaming data acquisition interface provided by the Douyin open platform and write a program in Python to obtain data such as the number of viewers of the live streaming, the content of the barrage, the celebrity's live streaming behavior, etc. in real time, and finally obtain real-time multimodal data corresponding to the celebrity idol.

[0119] S22: Deconstruct and encode the spatiotemporal characteristics of the idol personality traits after fluctuation control, so as to deconstruct the personality traits corresponding to different times and situations according to the time sequence and spatial context, and convert them into a sequence of digital symbols that can be processed by a computer to obtain the spatiotemporal encoding dataset of the idol personality traits;

[0120] In this embodiment of the present invention, Python's data analysis and machine learning libraries are used to deconstruct and encode the spatiotemporal characteristics of the idol's personality traits after fluctuation control. File data is read using the pandas library. Each line in the file represents a personality trait at a specific time point. For example, [0.6, 0.7, 0.5] corresponds to outspokenness, likeability, and humor value, respectively. The personality traits at different time points are extracted according to the time series, forming a time series array. For example, the personality traits at time point 1 are [0.6, 0.7, 0.5], and those at time point 2 are [0.62, 0.68, 0.52]. Spatial context is categorized based on data source, such as interview context, variety show context, and social media context. Natural language processing techniques are used to identify context within the text descriptions in the data. For example, if the text contains the words "interview scene," it is determined to be an interview context. The personality traits corresponding to different time points and contexts are organized into a two-dimensional table, listed as time point, context, outspokenness, likeability, and humor value. Then, we use techniques such as one-hot encoding to convert these features into a sequence of digital symbols that can be processed by computers. For example, the context of "interview" is encoded as [1,0,0]; "variety show" is encoded as [0,1,0], etc. The personality feature values are normalized and combined with the context coding. For example, the personality features at time point 1 and in the interview context are encoded as [1,0,0,0.6,0.7,0.5]. All such encoded data are organized into a dataset, and finally a spatiotemporal coding dataset of idol personality features is obtained.

[0121] S23: Based on the spatiotemporal coding dataset of idol personality traits, a prototype framework for the corresponding star idol digital avatar is constructed, which includes an input layer, an intermediate processing layer, and an output layer. The input layer receives the spatiotemporal coding dataset of the corresponding idol personality traits, the intermediate processing layer performs feature conversion and calculation, and the output layer generates a virtual image and behavior performance that matches the idol personality.

[0122] In this embodiment of the present invention, the corresponding star idol digital avatar prototype framework is built by using the deep learning framework TensorFlow. By writing a program in Python, the input layer is first defined. The input layer receives the corresponding idol personality feature spatiotemporal coding dataset. In TensorFlow, the input layer is defined using the tf.keras.layers.Input function, for example:

[0123] import tensorflow as tf

[0124] input_layer=tf.keras.layers.Input(shape=(6,))#Assume that the dimension of the encoded data is 6

[0125] The intermediate processing layer uses multiple fully connected layers for feature conversion and calculation. For example, add two fully connected layers, each using the ReLU activation function. The code is as follows:

[0126] hidden_layer1=tf.keras.layers.Dense(32,activation='relu')(input_layer)

[0127] hidden_layer2=tf.keras.layers.Dense(16,activation='relu')(hidden_layer1)

[0128] The output layer generates an avatar and behavior that matches the idol's personality based on the calculation results of the intermediate processing layer. The output layer also uses a fully connected layer. The output dimension is determined by the number of parameters of the avatar and behavior, assuming there are 10 parameters, such as the avatar's facial expression parameters, body movement parameters, etc. Use the following code to define the output layer:

[0129] output_layer=tf.keras.layers.Dense(10)(hidden_layer2)

[0130] Combine the input layer, intermediate processing layer, and output layer into a model using the tf.keras.Model function:

[0131] model=tf.keras.Model(inputs=input_layer, outputs=output_layer)

[0132] The constructed star idol digital avatar will eventually obtain the corresponding star idol digital avatar prototype framework.

[0133] S24: Based on the star idol digital avatar prototype framework and combined with the variational autoencoder and the generative adversarial network, the real-time multimodal data corresponding to the star idol is used to fuse and generate digital avatars, so as to combine the star idol digital avatar prototype framework with the variational autoencoder and the generative adversarial network, and generate digital avatars corresponding to the virtual image and behavioral performance according to the real-time multimodal personality characteristics obtained from the analysis of the real-time multimodal data corresponding to the star idol. The corresponding personality characteristic encoding and decoding process is adjusted in combination with the variational autoencoder to better capture and reconstruct the personality characteristics corresponding to the idol, and at the same time optimize the generator and discriminator of the generative adversarial network, so that the generator can generate more realistic star idol performances, and the discriminator can more accurately judge the authenticity of the generator, so as to generate a star idol digital avatar model.

[0134] In an embodiment of the present invention, by using the TensorFlow framework based on the star idol digital twin prototype framework and combining the variational autoencoder and the generative adversarial network, the star idol digital twin prototype framework is loaded, and the star idol digital twin prototype framework is combined with the variational autoencoder and the generative adversarial network. For the variational autoencoder, TensorFlow is used to rebuild the structure and define the encoder part, for example:

[0135] encoder_input=tf.keras.layers.Input(shape=(6,))

[0136] encoder_hidden=tf.keras.layers.Dense(32,activation='relu')(encoder_input)

[0137] mu=tf.keras.layers.Dense(16)(encoder_hidden)

[0138] log_var=tf.keras.layers.Dense(16)(encoder_hidden)

[0139] Generate the encoding vector through the reparameterization trick, and then define the decoder part:

[0140]

[0141] For the generative adversarial network, a generator and a discriminator are defined. The generator receives random noise and personality features encoded by the variational autoencoder to generate the performance of a celebrity idol:

[0142] generator_input = tf.keras.layers.Input(shape = (16+16,)) # Assume that the noise dimension is 16 and the feature dimension after encoding is 16

[0143] generator_hidden=tf.keras.layers.Dense(64, activation='relu')(generator_input)

[0144] generator_output=tf.keras.layers.Dense(10)(generator_hidden)

[0145] generator=tf.keras.Model(generator_input,generator_output)

[0146] The discriminator determines whether the star idol performance generated by the generator is realistic:

[0147] discriminator_input=tf.keras.layers.Input(shape=(10,))

[0148] discriminator_hidden=tf.keras.layers.Dense(32,activation='relu')(discriminator_input)

[0149] discriminator_output=tf.keras.layers.Dense(1,activation='sigmoid')(discriminator_hidden)

[0150] discriminator=tf.keras.Model(discriminator_input,discriminator_output)

[0151] By combining the star idol digital avatar prototype framework, variational autoencoder and generative adversarial network, and using the real-time multimodal data corresponding to the star idol for training, the real-time multimodal personality characteristics are obtained by analyzing the real-time multimodal data, and the encoding and decoding process of the variational autoencoder is adjusted to enable it to better capture and reconstruct the personality characteristics corresponding to the idol. At the same time, during the training process, the generator and discriminator of the generative adversarial network are continuously optimized, so that the generator can generate more realistic star idol performances and the discriminator can more accurately judge the authenticity of the generator. After multiple rounds of training, the star idol digital avatar model is finally generated, which is used for subsequent star idol multimodal responses based on emotion recognition.

[0152] Furthermore, the fan operation management module includes the following functions:

[0153] Obtain fan behavior data;

[0154] In an embodiment of the present invention, fan behavior data is obtained by utilizing the backend data interfaces of social media platforms, celebrity idol official websites, and related applications at the data collection end. For social media platforms such as Weibo, the API interface provided by the Weibo open platform is used with the Python Tweepy library to obtain fan behavior data such as the time of attention, Weibo likes, Weibo comments, and Weibo reposts based on the fan list of the celebrity idol's official account. For example, through API calls, it is obtained that fan A liked a promotional video Weibo posted by the celebrity idol at 3:30 PM on August 10, 2024. For the celebrity idol's official website, the website server's logging function is utilized to collect data such as the time fans visited the website, the pages viewed, and the length of stay. In terms of applications, if the celebrity idol has a dedicated app, the app's built-in data collection module records fan operations within the app, such as watching the celebrity idol's live broadcasts, purchasing virtual goods, and participating in interactive activities. The fan behavior data obtained from various channels is integrated and stored in a database in the data processing center, with each row recording one piece of behavior data for each fan, including fields such as the behavior time, behavior type, and behavior object.

[0155] Preferably, fan behavior data is subjected to spatiotemporal granularity analysis and discretization to analyze the distribution of fan behavior data in time and space dimensions. Time is subdivided into different time periods, including hours, days, and weeks. Space can be divided according to the region where fans are located. Continuous fan behavior data is discretized according to the corresponding spatiotemporal granularity to obtain spatiotemporal discretized fan behavior data.

[0156] In the embodiment of the present invention, by using Python's pandas library to perform spatiotemporal granularity analysis and discretization on fan behavior data, the time dimension is subdivided. For time subdivision to hours, the dt.hour function of pandas is used to extract the hours of each behavior data. For example, if the time of a behavior data is 2024-08-10 15:30:00, the extracted hour is 15. For subdivision to days, the dt.date function is used to obtain the date, such as 2024-08-10. For subdivision to weeks, the dt.week function is used to obtain the week number. In the spatial dimension, assuming The information about the fan's region is stored in the "region" field in the database table and divided according to the region, such as "North China", "East China", "South China", etc. For continuous fan behavior data, it is discretized according to the corresponding spatiotemporal granularity. For example, the behavior data of all fans on August 10, 2024 are grouped together, and the fan behavior data in North China are grouped together. The discretized data are organized into a new table format, listed as time granularity (hour / day / week), spatial granularity (region), fan ID, behavior type, etc., and finally the spatiotemporal discretization data of fan behavior is obtained.

[0157] Preferably, based on the spatiotemporal discretization data of fan behavior, symbolic mapping and association mining are performed on each corresponding fan behavior pattern in the fan behavior data, so as to assign a corresponding behavior symbol to each discretized fan behavior data, convert the behavior symbol corresponding to each fan into a symbol sequence, and use an association rule mining algorithm to analyze the association relationship between the symbol sequences, thereby finding the corresponding behavior pattern combination that frequently appears for each fan, so as to obtain a fan behavior pattern symbol association data set;

[0158] In an embodiment of the present invention, Python's machine learning and data mining libraries are used to perform symbolic mapping and association mining on the spatiotemporal discretized fan behavior data. Data is read from the corresponding database table, and corresponding behavior symbols are assigned to each discretized fan behavior data. For example, the like behavior is symbolized as "L", the comment behavior is symbolized as "C", and the purchase of virtual goods behavior is symbolized as "B". For each fan, the behavior symbols at different spatiotemporal granularities are converted into symbol sequences in chronological order. For example, the behavior sequence of fan A on a certain day is "LCL". The Apriori association rule mining algorithm is applied, using the apriori function in Python's mlxtend library, to analyze the association relationship between these symbol sequences. The minimum support is set to 0.1 and the minimum confidence is set to 0.6. Frequent item sets and association rules are mined through calculation. For example, it is found that the behavior pattern combination "LC" (like first, then comment) frequently appears in many fans, with a confidence of 0.7. These frequently occurring behavior pattern combinations are organized into a data set and stored in a database table in a data processing center, listed as fields such as behavior pattern combination, support, and confidence, to ultimately obtain a fan behavior pattern symbol association data set.

[0159] Preferably, a fan behavior profile is performed for each fan behavior pattern corresponding to the fan behavior data according to the fan behavior pattern symbol association data set to obtain a fan user profile;

[0160] In an embodiment of the present invention, a fan behavior portrait is performed for each fan behavior pattern corresponding to the fan behavior data according to the fan behavior pattern symbol association data set by utilizing Python's data analysis library, so as to read the frequently occurring behavior pattern combination data from the database table, and read the fan's spatiotemporal behavior data. Taking fan A as an example, if the "LC" (like and then comment) behavior pattern combination frequently appears, and is mainly concentrated in the weekend evenings from 8 to 10 o'clock, and the region is East China, a fan behavior portrait is constructed based on this information. The portrait includes the fan's active time (8 to 10 o'clock on weekends), main behavior pattern (like and then comment), region (East China) and other features. Such a portrait is constructed for each fan, and the behavior portraits of all fans are organized into a table form, listed as fan ID, active time, main behavior pattern, region, etc., and finally a fan user portrait is obtained.

[0161] Preferably, fan behavior interaction recommendations are made for fan user portraits to generate fan behavior interaction operation recommendation content.

[0162] In an embodiment of the present invention, a recommendation algorithm is used to perform fan behavior interaction recommendations on fan user portraits. A recommendation algorithm based on collaborative filtering is used, such as the KNNWithMeans algorithm in Python's Surprise library. Assuming that interactive operation content is to be recommended for fan B, based on fan B's behavior profile, other fans with similar behavior patterns to fan B are found in the database, such as fan C, fan D, etc., and the interactive activities or virtual goods purchased by these similar fans are checked. For example, fan C and fan D have both participated in an online singing competition held by a celebrity idol, while fan B has not. This online singing competition is recommended to fan B as recommended content. By traversing the user profiles of all fans, personalized fan behavior interactive operation recommendation content is generated for each fan and stored in the corresponding database table of the data processing center, listed as fields such as fan ID and recommended content, so that it can be used later to guide fans to participate in interactions and enhance the influence of celebrity idols and fan activity.

[0163] Furthermore, the fan behavior interaction recommendation for the fan user portrait includes:

[0164] Conduct fan behavior interaction statistics on fan user portraits to obtain the frequency of fan behavior interaction;

[0165] In an embodiment of the present invention, fan behavior interaction statistics are performed on fan user portraits by utilizing Python's pandas library. The portrait includes fields such as fan ID, active time, main behavior pattern, and region. In order to count the frequency of fan behavior interaction, the behavior type that needs to be counted is first determined, such as likes, comments, purchase of virtual goods, participation in interactive activities, etc. For each behavior type, each row of data in the database table is traversed. If the "behavior type" field in the row of data is consistent with the currently counted behavior type, the interaction frequency of the corresponding fan ID is increased by 1. For example, when counting the interaction frequency of like behavior, it is found that fan A has 3 records of the behavior type of like, then the like interaction frequency of fan A is 3. The interaction frequencies of all fans for different behavior types are organized into a new table, listed as fan ID, like frequency, comment frequency, virtual goods purchase frequency, participation in interactive activities frequency, etc., providing a data basis for subsequent mining of fan behavior interaction interest points.

[0166] Preferably, based on the frequency of fan behavior interaction, the fan user portrait is mined for behavior interaction interest points to obtain fan behavior interaction interest points;

[0167] In an embodiment of the present invention, a data mining algorithm is used to mine behavioral interaction interest points for fan user portraits based on the frequency of fan behavioral interactions, and this is achieved by using the FP-Growth algorithm in the association rule mining algorithm with the help of Python's pymining library. The algorithm efficiently mines frequent item sets by constructing a frequent pattern tree (FP-tree), and sets the minimum support to 0.1, that is, when the proportion of a certain behavior type combination appearing in fans reaches 10% or more, it is considered to be frequently occurring. For example, during the analysis process, it was found that 15% of fans have a high frequency of purchasing virtual goods and participating in interactive activities at the same time, then the behavior type combination of "purchasing virtual goods-participating in interactive activities" is regarded as a fan behavior interaction interest point, and all the mined fan behavior interaction interest points are sorted into a list, and each interest point records the relevant behavior type combination and the corresponding support, which are listed as fields such as behavior type combination and support, providing key information for the subsequent generation of fan behavior interaction operation recommendation content.

[0168] Preferably, fan behavior interaction recommendations are made based on fan behavior interaction interest points to generate fan behavior interaction operation recommendation content.

[0169] In an embodiment of the present invention, a program is written in Python to perform fan behavior interaction recommendations based on fan behavior interaction points of interest. For each fan, recommended content is generated based on the main behavior patterns and region in their behavior portraits, combined with fan behavior interaction points of interest. For example, for a fan whose main behavior patterns are likes and comments and whose region is South China, if the combination of "likes-comments-participation in online voting activities" is found to have a high support rate in the fan behavior interaction points of interest, and the fan has not participated in online voting activities, then the online voting activities held by the star idol are recommended to the fan as recommended content. By traversing the user portraits of all fans, personalized fan behavior interaction operation recommendation content is generated for each fan, and the recommended content is organized into a table, listing fan ID, recommended content, etc., overwriting previously existing recommended content, so that these recommended contents can be pushed to fans through the star idol's official channels, such as social media, official websites, exclusive APPs, etc., to guide fans to participate in interactions, enhance the stickiness between fans and star idols, and improve the star idol's influence and fan activity.

[0170] Furthermore, the emotion recognition interactive response module includes the following functions:

[0171] Through the RFM model, including recency, frequency, and engagement, we conduct fan interaction identification and analysis on the recommended content of fan behavior interaction operations to identify the corresponding fan life cycle stage and the corresponding fan interaction data, including visit time, number, and spending power, to obtain the prediction results of fan interaction recommendation behavior;

[0172] In an embodiment of the present invention, by utilizing Python's pandas and scikit-learn libraries, fan interaction recognition and analysis is performed on fan behavior interaction operation recommendation content based on the RFM model (Recency, Frequency, Monetary), and the recommended content data is read. At the same time, combined with the corresponding fan information, for recency, the number of days between the time of each fan's most recent interaction with the star idol and the current time is calculated. For example, the last time fan A participated in the star idol online voting activity was 3 days ago, so its recency value is 3. For frequency, the total number of various interactive behaviors of fans in a certain time period (assuming it is the past 30 days) is counted, such as fan A liked 5 times, commented 3 times, and participated in activities 2 times in the past 30 days. times, the total interaction frequency is 10 times, and the participation is measured by the fans' consumption amount. Assume that fan A spends 50 yuan to buy virtual goods. According to the set consumption level classification standard, its participation level is determined, and these data are organized into a feature matrix. Using the clustering algorithm in the scikit-learn library, such as the K-Means algorithm, the number of clusters is set to 3 (representing different fan life cycle stages, such as the active period, the silent period, and the churn period). After calculation, the life cycle stage of each fan is identified. At the same time, the fans' visit time, number of interactions, consumption power and other interactive data are extracted. This information is integrated to finally obtain the fan interaction recommendation behavior prediction results, which are stored in the corresponding database table and listed as fan ID, life cycle stage, visit time, number of interactions, consumption power and other fields.

[0173] Preferably, based on the prediction results of fan interaction recommendation behavior and combined with the star idol digital avatar model, star idol contact scenarios are constructed to generate star idol active contact scenarios corresponding to different fan behavior interaction conditions;

[0174] In an embodiment of the present invention, a star idol engagement scenario is constructed by utilizing a Python data analysis and visualization library based on the predicted results of fan interaction recommendation behavior and in combination with a star idol digital avatar model. For fans in an active period with a high number of interactions, assuming fan B is in an active period with a high number of interactions and strong spending power, based on the characteristics of their region and visit time, if this fan often visits between 8 and 10 p.m. and is located in East China, a engagement scenario is generated in which the star idol proactively pushes exclusive promotions at 8 p.m. against a backdrop of scenes characteristic of East China. For fans in a silent period, such as fan C, whose last interaction was 15 days ago and whose interaction frequency is low, based on their previous interaction preferences, if they were previously interested in the star idol's live broadcast, a engagement scenario is generated in which the star idol proactively pushes a live broadcast preview with personalized invitation language. The corresponding star idol proactive engagement scenarios under different fan behavior interaction conditions are stored in the corresponding database table in the form of images, video scripts, or text descriptions, with fields such as fan ID, engagement scenario description, and scenario resource path (if any), providing a basis for subsequent emotion recognition and interactive response analysis.

[0175] Preferably, based on the corresponding star idol active contact scenarios under different fan behavior interaction conditions and combined with the decision tree, the star idol digital avatar model is analyzed for emotional recognition interaction response between fans and idols, so as to construct the corresponding multi-layer star idol interaction response jump logic according to the decision tree, and generate the corresponding emotional response state according to the corresponding multi-layer star idol interaction response jump logic, so as to generate the emotional interaction response scene state between fan behavior interaction and star idol.

[0176] In an embodiment of the present invention, by utilizing Python's decision tree algorithm library, such as DecisionTreeClassifier in scikit-learn, based on the corresponding star idol active contact scenarios under different fan behavior interaction conditions and combining the decision tree, the star idol digital twin model is analyzed for emotion recognition and interactive response between fans and idols. During the interaction between star idols and fans, the fan reply content and interaction status are monitored in real time. For abnormal replies, semantic analysis tools in natural language processing technology, such as the HanLP library, are used to calculate the deviation between the fan reply semantics and the normal context. If the fan reply semantic deviation is greater than 0.7 and the abnormal reply lasts for 10 seconds, the latest three interaction data are retrieved from the star idol's interaction record database. According to the personality set in the star idol digital twin model, corresponding personality abnormality repair operations are performed, such as changing the star idol's language style, expression, etc.; for active contact, if the fan has not interacted for 7 days and the sign-in is interrupted through fan interaction records, the fan-loving mode in the star idol digital twin model is activated, and customized wake-up eggs are generated according to pre-set rules and sent to fans; for sudden interruptions, if the service interruption time is monitored by the system, Between 55 and 65 seconds, scenario-based apology scripts for recalling stage accidents are retrieved from the pre-set script library and presented to fans through celebrity idols; for sensitive topics, keyword matching and sentiment analysis tools are used to identify them. Once sensitive topics are detected, celebrity idols are switched to agent agency mode and compliant scripts are implanted; for fan questions, sentiment analysis tools are used to calculate the negative value of fan replies and monitor the increase in fan speech speed. If the sum of the two is greater than 40%, the celebrity idol’s corresponding emotion transfer strategy is triggered, such as diverting fans’ attention through humorous words; for group events, if a celebrity idol responds to the same question in 1 Repeat the reply 500 times within 10 minutes to activate the scene-shifting words corresponding to the star idol and guide fans to pay attention to other topics; for cultural conflicts, use foreign language recognition tools, such as Baidu Translate API (apply for and obtain authorization). If the foreign language recognition confidence is less than 0.3, call the star idol's corresponding star imitation dialect style function to alleviate communication barriers; and for emotional overload, by recording the continuous interaction time between the star idol and fans, if it exceeds 120 minutes, start the star idol's corresponding backstage lounge virtual scene to show a relaxed atmosphere and gradually cool down the interaction, so as to continuously optimize the interactive experience between the star idol and fans.

[0177] Furthermore, the corresponding multi-layer star idol interaction response jump logic includes: for abnormal replies, if the semantic deviation is greater than 0.7 and lasts for 10 seconds, the star idol's corresponding three most recent interactions will be called to perform abnormal repair of the star idol's corresponding personality; for active contact, if the fan has not interacted for 7 days and the sign-in is interrupted, the star idol's corresponding fan-loving mode will be activated and a customized wake-up easter egg will be sent; for sudden interruptions, if the service interruption is greater than 55 seconds and less than 65 seconds, a stage accident recall scenario apology will be generated; for sensitive topics, the star idol will be switched to the agent agent mode In response to fan doubts, if the sum of the negative sentiment analysis value and the increase in speech speed is greater than 40%, the corresponding emotional transfer of the star idol will be triggered; in response to group events, if the star idol repeats the same question 500 times within 10 minutes, the corresponding scene transfer speech will be activated; in response to cultural conflicts, if the foreign language recognition confidence is less than 0.3, the corresponding celebrity will be called to imitate the dialect style; and in response to emotional overload, if the corresponding star idol's continuous interaction exceeds 120 minutes, the corresponding backstage lounge virtual scene will be activated to gradually cool down the interaction.

[0178] The present invention is therefore intended to be illustrative and non-restrictive in all respects, with the scope of the invention being defined by the appended claims rather than the foregoing description, and all changes that come within the meaning and range of equivalents of the application documents are intended to be embraced therein.

[0179] The foregoing description is intended only to provide specific embodiments of the present invention, which will enable those skilled in the art to understand and implement the present invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention is not intended to be limited to the embodiments shown herein, but is to be construed in the widest possible manner consistent with the principles and novel features disclosed herein.

Claims

1. A multimodal response system for celebrity idols based on emotion recognition, characterized in that: Includes the following modules: The personality analysis module is used to obtain the public statements, social media data and interview content corresponding to the star idols, and conduct personality fluctuation control analysis based on the public statements, social media data and interview content corresponding to the star idols to obtain the personality characteristics of the idols after fluctuation control; A digital avatar generation module is used to obtain real-time multimodal data corresponding to celebrity idols through access, and generate personality digital avatars based on the celebrity idols' personality characteristics after fluctuation control, so as to generate a celebrity idol digital avatar model; The fan operation management module is used to obtain fan behavior data and create fan behavior profiling based on the fan behavior data to obtain fan user profiles; Conduct fan behavior interaction recommendations for fan user portraits to generate recommended content for fan behavior interaction operations; The emotion recognition and interactive response module is used to recommend content based on fan behavior interaction operations and combine the star idol digital avatar model to conduct emotion recognition and interactive response analysis between fans and idols, so as to generate the emotional interaction response scenario status between fan behavior interaction and star idols.

2. The star idol multimodal response system based on emotion recognition according to claim 1 is characterized in that: The personality analysis module includes the following functions: Obtain public comments, social media data, and interview content corresponding to celebrity idols; De-noise, remove duplicates, and fill in missing values on the public statements, social media data, and interview content corresponding to the celebrity idols to obtain pre-processed public statements, social media data, and interview content; Determine the personality 3D vector based on the pre-processed public statements, social media data, and interview content of the idols to generate the personality 3D vector of the star idols; The personality baseline fluctuation is controlled based on the three-dimensional personality vector of the star idol to obtain the idol personality characteristics after fluctuation control.

3. The star idol multimodal response system based on emotion recognition according to claim 2 is characterized in that: The interview content includes live video, audio and expression data.

4. The star idol multimodal response system based on emotion recognition according to claim 2 is characterized in that: Determining the three-dimensional personality vector based on pre-processed public statements, social media data, and interview content of the idol includes: Perform semantic and sentiment dictionary matching on the corresponding text content in the pre-processed idols' public speeches, social media data, and interviews to obtain the corresponding sentiment word polarity and intensity metadata in the idols' speeches, data, and content; Perform facial expression recognition analysis on the pre-processed idols’ public statements, social media data, and corresponding interview videos and expressions in the interview content to obtain the corresponding interview facial expression information in the idols’ statements, data, and content; Extract speech intonation and semantic keywords from pre-processed idol public statements, social media data, and corresponding interview, variety show, social media, and live broadcast audio content to obtain the corresponding audio speech intonation and semantic keywords in the idol's speech, data, and content; Based on the corresponding facial expression information in the idols' speeches, data, and content, we conduct in-depth behavioral pattern analysis of the corresponding speeches, audio, and videos in the idols' public speeches, social media data, and interviews to obtain the corresponding behavioral semantic intention patterns in the idols' speeches, data, and content; The personality three-dimensional vector is determined based on the corresponding emotional word polarity, intensity metadata, interview facial expression information, audio voice intonation and semantic keywords, and behavioral semantic intention patterns in the idol's speech, data and content, so as to analyze and calculate the personality dimensions of the star idol corresponding to the degree of publicity, affinity and humor value, and then map the publicity, affinity and humor value into the corresponding three-dimensional personality vector to generate the star idol's three-dimensional personality vector.

5. The star idol multimodal response system based on emotion recognition according to claim 2 is characterized in that: The personality baseline fluctuation control based on the celebrity idol personality three-dimensional vector includes: Normalize the three-dimensional personality vector of the star idol at each time point to obtain the three-dimensional standard vector of the idol personality at each time point; Draw the personality change baseline for the personality dimension corresponding to the three-dimensional standard vector of the idol personality at each time point to obtain the three-dimensional change baseline of the idol personality; Obtaining the standard values corresponding to the three personality dimensions in the three-dimensional standard vector of the idol personality at each time point, and setting the standard values corresponding to the three personality dimensions as the personality fluctuation threshold to obtain the three-dimensional fluctuation threshold of the idol personality; Based on the three-dimensional fluctuation threshold of idol personality and combined with the three-dimensional change baseline of idol personality, the personality baseline fluctuation control is performed on the three-dimensional vector of star idol personality at each time point, so as to compare and judge based on the three-dimensional vector of star idol personality at each time point with the corresponding three-dimensional fluctuation threshold of idol personality. If the corresponding personality dimension is less than plus or minus 1.5 times the three-dimensional fluctuation threshold of idol personality, the personality dimension is judged to be stable; otherwise, the personality dimension is judged to be unstable, and the personality baseline trigger correction is performed on the corresponding personality dimension judged to be unstable in combination with the three-dimensional change baseline of idol personality until the fluctuation of the corresponding personality dimension is stabilized and used as a personality characteristic, so as to obtain the personality characteristics of the idol after fluctuation control.

6. The star idol multimodal response system based on emotion recognition according to claim 1 is characterized in that: The digital avatar generation module includes the following functions: Access real-time multimodal data on celebrity idols, including interviews, variety shows, social media, and live streaming data. The idol personality traits after fluctuation control are deconstructed and coded in time and space, so as to deconstruct the personality traits corresponding to different time and context according to time series and spatial context, and convert them into digital symbol sequences that can be processed by computers to obtain the idol personality traits time and space coding dataset; Based on the spatiotemporal coding dataset of idol personality traits, a prototype framework for the corresponding star idol digital avatar is built, including an input layer, an intermediate processing layer, and an output layer. The input layer receives the spatiotemporal coding dataset of the corresponding idol personality traits, the intermediate processing layer performs feature conversion and calculation, and the output layer generates a virtual image and behavior that matches the idol's personality. Based on the star idol digital avatar prototype framework and combined with variational autoencoders and generative adversarial networks, the real-time multimodal data corresponding to star idols are used to fuse digital avatars to generate digital avatars. The star idol digital avatar prototype framework is combined with variational autoencoders and generative adversarial networks. According to the real-time multimodal personality characteristics obtained from the analysis of the real-time multimodal data corresponding to the star idols, the digital avatar characteristics corresponding to the virtual image and behavioral performance are generated. The corresponding personality characteristic encoding and decoding process is adjusted in combination with the variational autoencoder to better capture and reconstruct the personality characteristics corresponding to the idols. At the same time, the generator and discriminator of the generative adversarial network are optimized, so that the generator can generate more realistic star idol performances and the discriminator can more accurately judge the authenticity of the generator to generate a star idol digital avatar model.

7. The star idol multimodal response system based on emotion recognition according to claim 1 is characterized in that: The fan operation management module includes the following functions: Obtain fan behavior data; Perform spatiotemporal granularity analysis and discretization on fan behavior data to analyze the distribution of fan behavior data in time and space dimensions. Time can be divided into different time periods, including hours, days, and weeks. Space can be divided according to the fans' location. Continuous fan behavior data is discretized according to the corresponding spatiotemporal granularity to obtain spatiotemporal discretized fan behavior data. Based on the spatiotemporal discretization of fan behavior data, symbolic mapping and association mining are performed on each corresponding fan behavior pattern in the fan behavior data to assign corresponding behavior symbols to each discretized fan behavior data. The behavior symbols corresponding to each fan are converted into symbol sequences, and the association rule mining algorithm is used to analyze the association relationship between the symbol sequences. The corresponding behavior pattern combinations that frequently appear for each fan are found to obtain the fan behavior pattern symbol association dataset; Perform fan behavior profiling for each fan behavior pattern corresponding to the fan behavior data according to the fan behavior pattern symbol association dataset to obtain a fan user profile; Conduct fan behavior interaction recommendations for fan user portraits to generate fan behavior interaction operation recommendation content.

8. The celebrity idol multimodal response system based on emotion recognition according to claim 7 is characterized in that: The fan behavior interaction recommendation for the fan user portrait includes: Conduct fan behavior interaction statistics on fan user portraits to obtain the frequency of fan behavior interaction; Based on the frequency of fan behavior interaction, the fan user portrait is mined for behavior interaction interest points to obtain fan behavior interaction interest points; Fan behavior interaction recommendations are made based on fan behavior interaction interest points to generate fan behavior interaction operation recommendation content.

9. The celebrity idol multimodal response system based on emotion recognition according to claim 1 is characterized in that: The emotion recognition interactive response module includes the following functions: Through the RFM model, including recency, frequency, and engagement, we conduct fan interaction identification and analysis on the recommended content of fan behavior interaction operations to identify the corresponding fan life cycle stage and the corresponding fan interaction data, including visit time, number, and spending power, to obtain the prediction results of fan interaction recommendation behavior; Based on the prediction results of fan interaction recommendation behavior and combined with the star idol digital avatar model, star idol contact scenarios are constructed to generate star idol active contact scenarios corresponding to different fan behavior interaction conditions; Based on the corresponding star idol active contact scenarios under different fan behavior interaction conditions and combined with the decision tree, the star idol digital avatar model is used to analyze the emotional recognition interaction response between fans and idols, so as to construct the corresponding multi-layer star idol interaction response jump logic according to the decision tree, and generate the corresponding emotional response state according to the corresponding multi-layer star idol interaction response jump logic, so as to generate the emotional interaction response scenario state between fan behavior interaction and star idols.

10. The celebrity idol multimodal response system based on emotion recognition according to claim 9 is characterized in that: The corresponding multi-layer star idol interaction response jump logic includes: for abnormal replies, if the semantic deviation is greater than 0.7 and lasts for 10 seconds, the star idol's corresponding three most recent interactions will be called to perform abnormal repair of the star idol's personality; for active contact, if the fan has not interacted for 7 days and the sign-in is interrupted, the star idol's corresponding fan mode will be activated and a customized wake-up easter egg will be sent; for sudden interruptions, if the service interruption is greater than 55 seconds and less than 65 seconds, a stage accident recall scenario apology will be generated; for sensitive topics, the star idol will be switched to the agent agency mode and Embed compliant language; for fan questioning, if the sum of the negative sentiment analysis value and the increase in speech speed is greater than 40%, the corresponding emotional transfer of the star idol will be triggered; for group events, if the star idol repeats the same question 500 times within 10 minutes, the corresponding scene transfer language of the star idol will be activated; for cultural conflict, if the foreign language recognition confidence is less than 0.3, the corresponding star of the star idol will be called to imitate the dialect style; and for emotional overload, if the corresponding star idol interacts for more than 120 minutes continuously, the corresponding backstage lounge virtual scene of the star idol will be activated to gradually cool down the interaction.

Images