Emotion and hotspot-based short video generation method and device, equipment and medium

Abstract

This application discloses a method, apparatus, device, and medium for generating short videos based on emotion and trending topics. The method includes: collecting emotional video data to display a user's emotional state and trending topic information data for the current time period; determining a user's emotional vector and trending keywords based on the emotional video data and the trending topic information data, respectively; generating a creative script and a storyboard description based on the user's emotional vector, the trending keywords, and a trained large language model; generating visual and audio materials based on the creative script and the storyboard description; and synchronously synthesizing the visual and audio materials to obtain a short video. This solution, by integrating the user's current emotional state with social trending content, achieves personalized generation and emotion-driven expression of short video content, enhancing the contextual resonance and user appeal of the content. It can be widely applied in scenarios such as intelligent content creation, social recommendation, and emotional dissemination.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a method, apparatus, device, and medium for generating short videos based on emotions and trending topics. Background Technology

[0002] With the rapid development of the short video industry, users' demands for personalized, efficient, and emotionally resonant content creation are increasing, prompting the application of intelligent content production technology in this field to gradually become a research hotspot. Especially against the backdrop of rapid iteration in the content ecosystem, achieving efficient generation of short video ideas and timely response to trending content has become a key means to enhance platform activity and user stickiness. Currently, in the short video creation process, users typically rely on existing templates, material libraries, or manual editing tools, resulting in cumbersome production paths, heavy creative burdens, and a lack of systematic support in areas such as creative conception, trend capture, and emotional expression, making it difficult to meet the increasingly diverse creative needs of users.

[0003] In related technologies, most short video content creation tools rely on splicing materials and combining templates to achieve content output. While this method lowers the creation threshold and increases content production speed to some extent, significant technical bottlenecks remain. For example, the creative generation process heavily depends on user participation and lacks automated content conception capabilities, leading to homogenization in style and theme, and a poor creation experience. The hot topic response mechanism is imperfect, failing to quickly capture and link social topics and trends, impacting dissemination efficiency. Weak emotion modeling capabilities make it difficult to accurately match content with users' emotional states, affecting viewer resonance. Furthermore, the overall production process demands high levels of skill in editing, art direction, and language expression, making it difficult for ordinary users to quickly complete the entire creation process, limiting the universality and ease of use of intelligent creation tools. Therefore, how to build a short video intelligent generation system with automatic creative generation capabilities, real-time hot topic adaptation capabilities, and an emotion-driven mechanism, while improving content richness and user participation experience, lowering the production threshold, and enhancing dissemination effectiveness, has become a core issue that urgently needs to be addressed in the current evolution of short video technology. Summary of the Invention

[0004] This application provides a method, apparatus, device, and medium for generating short videos based on emotion and trending topics. By collecting user emotional state data and current trending information, it enables creative generation and video construction tailored to user emotional expression and content trends. This solution improves the efficiency of short video creation and its adaptability for dissemination while ensuring the accuracy and timeliness of the generated content's emotional expression. It is suitable for short video creation scenarios with high requirements for content personalization, response speed, and dissemination effectiveness.

[0005] Firstly, this application provides a short video generation method based on emotion and trending topics, including: Collect emotional video data and hot topic information data for the current time period to display the user's emotional state, and determine the user's emotional vector and hot topic keywords based on the emotional video data and the hot topic information data respectively; Based on the user sentiment vector, the hot keywords, and the trained large language model, generate creative scripts and shot descriptions; Visual and audio materials are generated based on the creative script and the storyboard description, and the visual and audio materials are synchronously synthesized to obtain a short video.

[0006] Secondly, this application provides a short video generation device based on emotion and trending topics, comprising: The data acquisition module is used to collect emotional video data to display the user's emotional state and hot topic information data for the current time period; The determination module is used to determine the user's emotional vector and hot keywords based on the emotional video data and the hotspot information data, respectively; The generation module is used to generate creative scripts and shot descriptions based on the user sentiment vector, the hot keywords, and the trained large language model. The compositing module is used to generate visual and audio materials based on the creative script and the storyboard description, and to synchronously compose the visual and audio materials to obtain a short video.

[0007] Thirdly, this application provides a short video generation device based on emotion and trending topics, including: One or more processors; A memory that stores one or more programs that, when executed by one or more processors, enable the one or more processors to implement the emotion- and trending short video generation method as described in the first aspect.

[0008] Fourthly, this application provides a storage medium containing computer-executable instructions, which, when executed by a computer processor, are used to perform the short video generation method based on emotion and trending topics as described in the first aspect.

[0009] This application collects emotional video data reflecting user emotional states and trending information data for the current time period, extracting and constructing emotional vectors reflecting users' subjective emotional states and trending keywords representing public attention, forming the dual-driven foundational data for short video generation. Based on the extracted user emotional vectors and trending keywords, combined with a trained large language model, creative scripts and storyboard descriptions with emotional guidance and topic relevance are dynamically generated, constructing a logical framework for video content oriented towards diverse contexts. During the generation process, video segments and audio corpora adapted to the storyboard content are automatically matched, achieving segmented generation and structural assembly of visual and audio materials. Subsequently, through synchronous synthesis of visual and audio timelines, short video content highly aligned with individual user emotions and social trends is produced. This method not only improves the generation efficiency and customization capabilities of short video content but also significantly enhances the emotional resonance and dissemination value of the content, exhibiting good adaptability, flexibility, and promotional potential, especially suitable for scenarios such as intelligent media generation and personalized content recommendation. Attached Figure Description

[0010] Figure 1 This is a flowchart of a short video generation method based on emotion and trending topics provided in an embodiment of this application; Figure 2 This is a flowchart of a user sentiment vector determination method provided in an embodiment of this application; Figure 3 This is a flowchart of a method for determining emotion vector components provided in an embodiment of this application; Figure 4 This is a flowchart of a user sentiment vector determination method provided in an embodiment of this application; Figure 5 This is a flowchart of a creative script and storyboard description generation method provided in an embodiment of this application; Figure 6 This is a flowchart illustrating a user emotion tag and a method for determining the intensity of emotion, as provided in an embodiment of this application. Figure 7 This is a flowchart illustrating a method for generating visual and audio materials according to an embodiment of this application; Figure 8 This is a flowchart illustrating the steps of a short video generation method based on emotion and trending topics provided in an embodiment of this application. Figure 9 This is a structural block diagram of a short video generation device based on emotion and trending topics provided in an embodiment of this application; Figure 10 This is a schematic diagram of the structure of a short video generation device based on emotion and trending topics provided in an embodiment of this application. Detailed Implementation

[0011] To make the objectives, technical solutions, and advantages of this application clearer, specific embodiments of this application will be described in further detail below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are merely for explaining this application and not for limiting it. It should also be noted that, for ease of description, only the parts relevant to this application are shown in the drawings, not all of them. Before discussing exemplary embodiments in more detail, it should be mentioned that some exemplary embodiments are described as processes or methods depicted as flowcharts. Although the flowcharts describe operations (or steps) as being processed sequentially, many of these operations can be performed in parallel, concurrently, or simultaneously. Furthermore, the order of the operations can be rearranged. A process can be terminated when its operation is completed, but it may also have additional steps not included in the drawings. A process can correspond to a method, function, procedure, subroutine, subroutine, etc.

[0012] The terms "first," "second," etc., used in the specification and claims of this application are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such use of data can be interchanged where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first," "second," etc., are generally of the same class and the number of objects is not limited; for example, a first object can be one or more. Furthermore, in the specification and claims, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship.

[0013] Currently, the creation and dissemination of short video content faces multiple challenges, including a growing exhaustion of creative output, high production barriers, and lagging responses to trending topics. Existing short video production tools generally rely on fixed templates and pre-set materials for content splicing, lacking intelligent models with contextual understanding and creative generation capabilities. This results in low user creation efficiency, homogenized content styles, and limited personalized expression, failing to meet users' continuously growing demand for high-frequency, high-quality, and diverse content. Simultaneously, trending topics exhibit dynamic characteristics of high timeliness and high volatility. Traditional short video tools lack the responsiveness to extract topics and link content effectively, severely restricting the accuracy of capturing content popularity and the ability to control the pace of dissemination.

[0014] At the user level, existing methods generally lack sophisticated mechanisms for recognizing users' emotional states and perceiving specific scenarios. This makes it difficult for short video content to deeply resonate with users' inner emotions, impacting user engagement and the potential for viral spread. Furthermore, the short video production process still relies heavily on human intervention and professional processing skills, creating significant operational barriers for ordinary users in content creation and editing, thus limiting the vitality and expansion of the UGC ecosystem. Therefore, there is an urgent need for an intelligent short video generation method with capabilities for emotion recognition, trend-following, and creative generation. This method should integrate individual user emotional states with current social concerns to automatically generate scripts and content structures with strong communicative power and resonance, creating a low-barrier, high-efficiency, and highly empathetic video creation experience. Such a systematic and intelligent short video generation solution can significantly improve creators' productivity and content differentiation, enhance the emotional impact and trend-following adaptability of short video content, provide crucial support for platform operation and content ecosystem development, and comprehensively promote the transformation and upgrading of short video creation from tool-driven to intelligence-driven.

[0015] To address the aforementioned issues, this embodiment provides a short video generation method based on emotion and trending topics. It constructs a content generation system integrating emotion perception, semantic linkage, and material generation, achieving intelligent end-to-end driving from user perception to final output. This method introduces multimodal emotion recognition, trending topic extraction, and language model inference mechanisms, combined with video synthesis and adaptive optimization strategies, to comprehensively enhance the personalized expressiveness and dissemination adaptability of short video content generation. First, through a real-time emotion and trending topic acquisition module, based on multi-source inputs such as facial expression recognition, speech tone analysis, and text semantic interpretation, it integrates deep neural networks and emotion analysis algorithms to generate a high-dimensional user emotion vector. Simultaneously, it applies web crawling and keyword extraction technologies to capture social trending keywords in real time, constructing an emotion-trending joint feature model reflecting the individual user's state and the public opinion environment. Subsequently, based on the aforementioned joint feature vector, the method generates creative scripts and storyboard content through a large language model. Utilizing an emotion-driven Prompt construction mechanism and template guidance strategy, it outputs a content skeleton with emotional resonance and topic focus, laying the foundation for the logical structure and expressive tone of the short video. In the material generation stage, this method automatically drives an image generation model and a deep speech synthesis engine, matching background video, character animation, and environmental sound effects according to the script content to achieve intelligent generation and timeline-level synchronous synthesis of audio and video materials. It also combines a material style library and a sound effect resource library to achieve consistent adaptation of visual style and emotional tags. Finally, this method introduces a feedback loop mechanism, using user behavior data to drive a reinforcement learning algorithm to adaptively adjust generation parameters, continuously optimizing emotional weight allocation, style configuration, and semantic content selection strategies, thereby improving the acceptance and dissemination power of short videos across different user groups. This method not only significantly lowers the barrier to entry for short video creation and improves content generation efficiency and emotional impact, but also constructs a new paradigm of content production driven by user emotions, possessing broad application prospects and commercial promotion potential, especially suitable for scenarios such as personalized content recommendation, social media dissemination guidance, and intelligent marketing.

[0016] The emotion- and trending topic-based short video generation method provided in this embodiment can be executed by an emotion- and trending topic-based short video generation device. This device can be implemented through software and / or hardware, and can consist of two or more physical entities, or a single physical entity. For example, the emotion- and trending topic-based short video generation device could be an operations and maintenance server used to maintain the normal operation of business.

[0017] The short video generation device based on emotions and trending topics has at least one type of operating system installed, including but not limited to Android, Linux, and Windows. The device can install at least one application on the operating system; this application can be a built-in application or an application downloaded from a third-party device or server. In this embodiment, the short video generation device based on emotions and trending topics has at least one application capable of executing the short video generation method based on emotions and trending topics.

[0018] For ease of understanding, this embodiment uses the operation and maintenance server as the main body for executing the short video generation method based on emotion and hot topics as an example.

[0019] Figure 1 A flowchart of a short video generation method based on emotion and trending topics, provided in an embodiment of this application, is given. (Reference) Figure 1 The short video generation method based on emotion and trending topics specifically includes: S110. Collect emotional video data and hot topic information data for the current time period to display the user's emotional state, and determine the user's emotional vector and hot topic keywords based on the emotional video data and the hot topic information data respectively.

[0020] In one embodiment, emotional video data and trending information data for the current time period are first collected to display the user's emotional state. The emotional video data characterizes the emotional content expressed by the user within a specific time period, while the trending information data characterizes highly relevant keywords or event features in the current social or online environment. The emotional video data and trending information data to be collected can originate from video clips uploaded by users and their associated social media information. The users can be mobile applications, web platforms, or other client systems with upload capabilities. After acquiring the emotional video data and trending information data, user emotional vectors and trending keywords are determined. The user emotional vector characterizes the multidimensional emotional components contained in the video, constructing an emotion mapping model to drive the generation of short video content; the trending keywords characterize representative topic features in the social environment within the current time period, enhancing the relevance and dissemination of the video content.

[0021] In one embodiment, the user's emotion vector can be determined by: extracting features from facial expressions, voice intonation, and text information in emotional video data using an emotion recognition model, and constructing a vector representation structure that combines emotion labels and numerical weights using a multimodal fusion strategy.

[0022] In one embodiment, hot keywords can be determined by using a keyword extraction algorithm to select a set of keywords with high attention and contextual relevance based on search trend data, social media text content, and real-time topic popularity indicators within the current time period.

[0023] Optionally, Figure 2 A flowchart of a user sentiment vector determination method provided in an embodiment of this application is given. (Reference) Figure 2 The method for determining user sentiment vectors specifically includes: S1101. Extract several facial expression data and several voice audio data from the emotional video data, and identify the voice video data to obtain several voice intonation data and several voice text data.

[0024] For example, several facial expression data and several audio data are extracted from emotional video data, and the audio data is recognized to obtain several speech tone data and several speech-text data. Facial expression data is used to characterize the visual emotional features of the user in the video, including the temporal changes of facial movements such as smiling, frowning, and surprise; audio data is used to characterize the user's speech content and emotional expression, including acoustic parameters such as speech rate, pitch, and intensity; speech tone data is used to characterize the emotional prosodic features in the speech; and speech-text data is used to characterize the textual content information conveyed by the speech. After acquiring the emotional video data, a multimodal feature separation mechanism is used to separate the facial image sequences and corresponding audio segments related to the user's emotions, which are then fed into an image analysis model and a speech processing model for feature extraction and recognition processing, respectively.

[0025] In one embodiment, facial expression data can be extracted by: locating the facial region in a video frame using a face detection and keypoint tracking algorithm, and extracting the encoded features of facial expressions in each frame based on an expression recognition model to construct a temporal feature stream of facial expressions.

[0026] In one embodiment, the method for extracting speech intonation data can be: performing short-time Fourier transform or Mel frequency cepstral coefficient analysis on the audio data to extract acoustic feature parameters such as pitch, volume, and rhythm, and then mapping them into intonation emotion vectors by combining them with an emotion classification model.

[0027] In one embodiment, the method for recognizing speech-text data can be: using an automatic speech recognition model to decode the speech audio into corresponding natural language text for subsequent semantic analysis and sentiment vector construction.

[0028] S1102. Based on the facial expression data, the speech tone data, and the speech text data, determine the facial expression emotion vector, the speech tone emotion vector, and the text emotion vector, respectively.

[0029] For example, based on facial expression data, speech intonation data, and speech-text data, facial emotion vectors, speech intonation emotion vectors, and text emotion vectors are determined, respectively. The facial emotion vector represents the user's facial emotional state at the visual level, the speech intonation emotion vector represents the user's emotional tendency at the speech acoustic level, and the text emotion vector represents the emotional expression of the speech content at the semantic level. These emotion vectors collectively constitute a multimodal emotion recognition system, providing a foundation for subsequently generating a comprehensive user emotion vector. After acquiring facial expression data, speech intonation data, and speech-text data, the expression analysis model, speech intonation recognition model, and text emotion analysis model are respectively invoked to perform emotion label mapping and numerical encoding on various features, outputting a structured emotion vector representation.

[0030] In one embodiment, the expression emotion vector can be determined by inputting a sequence of facial images into a trained expression emotion recognition model, such as a deep network based on ResNet or 3D-CNN, extracting its probability distribution across multidimensional emotion categories, and mapping it to a low-dimensional emotion vector representation.

[0031] In one embodiment, the tone emotion vector can be determined by inputting the acoustic features of the speech into a tone emotion classification model, such as a BiLSTM or Transformer-based model, combining pitch, energy, and rhythm features to infer the emotional state, and then encoding and outputting it in vector form.

[0032] In one embodiment, the text sentiment vector can be determined by inputting the text obtained from speech recognition into a sentiment classification language model, such as a sentiment analysis network based on BERT or RoBERTa, to extract sentence-level sentiment semantic information and construct a sentiment vector representation.

[0033] Optionally, Figure 3 A flowchart of a method for determining emotion vector components provided in an embodiment of this application is given. (Reference) Figure 3 The method for determining the components of the emotion vector specifically includes: S11021. Determine the facial weight corresponding to each facial expression data based on the facial similarity between the facial expression data and historical facial expression data.

[0034] For example, based on the facial similarity between facial expression data and historical facial expression data, the facial weights corresponding to each facial expression data are determined. Facial similarity characterizes the feature matching degree between the facial expression extracted from the current video and historical emotion samples, measuring the representativeness of the current expression within the existing emotion distribution. Facial weights characterize the contribution strength of each frame or segment of facial expression data in generating the emotional expression vector, and are key parameters for constructing a stable and highly reliable emotional expression model. After obtaining the current facial expression data, a facial feature vector is generated using a facial feature extraction model, and its similarity is calculated with sample vectors in the historical facial emotion database to obtain the matching degree of the current data in the existing emotion corpus, thereby assigning corresponding weights.

[0035] In one embodiment, facial similarity can be calculated by extracting facial feature vectors through a face encoding model and using methods such as cosine similarity, Euclidean distance, or Mahalanobis distance to evaluate the similarity between the current expression and historical samples.

[0036] In one embodiment, facial weights can be determined by assigning normalized weights based on similarity scores, for example, by converting all facial similarities into weighted coefficients using the softmax function, which serve as the basis for subsequent weighted fusion of expression and emotion vectors.

[0037] S11022. Determine the voice weight corresponding to each of the voice audio data based on the voiceprint similarity between the voice audio data and the historical voice audio data.

[0038] For example, based on the voiceprint similarity between the current speech audio data and historical speech audio data, the corresponding voice weight for each speech audio data is determined. Voiceprint similarity characterizes the degree of similarity between the current speech segment and historical speech samples in terms of speaker features and emotional expression patterns, reflecting the degree of matching of the current speech within the existing emotion model. Voice weight characterizes the contribution strength of each speech data segment in the generation of intonation and emotion vectors, and is a key factor in improving the stability and individual adaptability of speech emotion recognition. After obtaining the current speech audio data, a feature vector of the speech segment is first constructed using a voiceprint extraction model, and then compared with the similarity of the classified and labeled emotional speech vectors in the historical speech sample library to determine the relative weight of the speech data in the overall emotion judgment.

[0039] In one embodiment, the method for calculating voiceprint similarity can be: using a voiceprint recognition model, such as an x-vector or ECAPA-TDNN architecture, to extract the embedding vector of each speech segment, and using cosine similarity or KL divergence to calculate its similarity score with historical sample vectors.

[0040] In one embodiment, the voice weights can be determined by: normalizing the similarity scores, and using a weighted strategy to map the similarity of each audio segment to a weight coefficient, which is then used for weighted fusion to generate a tone and emotion vector.

[0041] S11023. Based on each of the facial expression data and its corresponding facial weights, and each of the speech tone data and each of the speech text data and its corresponding speech weights, determine the facial expression emotion vector, the tone emotion vector, and the text emotion vector respectively.

[0042] For example, based on various facial expression data and their corresponding facial weights, as well as various speech tone data and speech-text data and their corresponding speech weights, facial expression emotion vectors, speech tone emotion vectors, and text emotion vectors are determined respectively. Facial weights and speech weights are used to characterize the importance of different modal inputs in their respective emotion recognition processes, improving the accuracy and robustness of emotion vector generation through a weighted fusion mechanism. Facial expression emotion vectors, speech tone emotion vectors, and text emotion vectors are used to characterize the user's emotional features at the visual, acoustic, and semantic levels, respectively, forming the fundamental components for constructing a multimodal user emotion profile. After completing feature extraction and weight calculation for facial expression and speech data, an intra-modal weighted aggregation strategy is adopted to perform weighted summation or weighted encoding on each emotion-related feature according to its weight, generating the final emotion vector output.

[0043] In one embodiment, the expression emotion vector can be determined by weighting and superimposing all facial expression feature vectors according to their corresponding facial weights to form a fusion vector representation in a multidimensional emotion label space, which serves as the emotion output of the facial channel.

[0044] In one embodiment, the way to determine the intonation emotion vector is to: weight the speech intonation features according to speech weights, extract the emotional expression of the current speech in the dimensions of pitch, volume and rhythm, and construct an acoustic emotion vector.

[0045] In one embodiment, the text sentiment vector can be determined by converting all speech text data into text sentiment vectors through a sentiment analysis model, and then performing a weighted average based on their respective speech weights to generate a text sentiment vector representing the semantic layer sentiment bias.

[0046] S1103. Determine the user's emotion vector based on the facial expression emotion vector, the tone emotion vector, and the text emotion vector.

[0047] For example, a user emotion vector is determined based on facial expression emotion vectors, intonation emotion vectors, and text emotion vectors. The facial expression emotion vector represents the user's facial emotional expression features in the visual channel, the intonation emotion vector represents the user's prosodic emotional features in the acoustic channel, and the text emotion vector represents the user's linguistic emotional tendency in the semantic channel. The user emotion vector is a unified expression form after fusing the above multimodal emotion features, used to comprehensively reflect the user's overall emotional tendency in the current state. After obtaining the three types of emotion vectors (facial expression, intonation, and text), a modal fusion strategy is used to encode them uniformly. The accuracy and expressive power of the fusion are improved through weight allocation, attention mechanisms, or feature alignment, thereby outputting a set of user emotion vectors with temporal and emotional consistency.

[0048] In one embodiment, the user sentiment vector can be determined by using a weighted fusion mechanism to linearly weight the facial expressions, tone of voice, and text sentiment vectors according to preset or dynamically calculated modal weights to obtain the fused user sentiment vector.

[0049] In one embodiment, the user sentiment vector can also be determined by using a multimodal attention mechanism, such as Transformer or multichannel attention network, to perform deep feature modeling and semantic alignment on each modality sentiment vector, thereby generating a final user sentiment vector that is more context-aware and emotionally nuanced.

[0050] Optionally, Figure 4 A flowchart of a user sentiment vector determination method provided in an embodiment of this application is given. (Reference) Figure 4 The method for determining user sentiment vectors specifically includes: S11031. Based on the recognition confidence of the facial expression emotion vector, the tone emotion vector, and the text emotion vector, determine the corresponding facial expression weight, speech tone weight, and speech text weight, respectively.

[0051] For example, based on the recognition confidence scores of facial expression emotion vectors, intonation emotion vectors, and text emotion vectors, corresponding facial expression weights, speech intonation weights, and speech-text weights are determined, respectively. The recognition confidence score characterizes the credibility of the emotion recognition results for each modality, typically determined by the output confidence score or the concentration of probability distribution given by the emotion recognition model during the inference phase. The corresponding modality weights are used to adjust the contribution ratio of each emotion channel in the user's emotion vector fusion process, improving the robustness and accuracy of multimodal fusion. After completing the initial recognition of emotion vectors, the weight distribution of each modality is dynamically calculated by analyzing the confidence scores of each vector, making the final fusion result closer to the expression tendency of the high-confidence channel and reducing interference from low-confidence inputs.

[0052] In one embodiment, facial expression weights can be determined by constructing facial expression weights that are proportional to the recognition reliability based on the entropy value, maximum class probability, or model confidence score of the emotion classification probability distribution in the emotion vector output.

[0053] In one embodiment, the speech intonation weights can be determined by using the mapping relationship between the matching degree of higher-order acoustic features in the intonation emotion vector and the confidence of the model output to calculate the credibility of the speech expression in the current context, and then mapping it to the speech intonation weights.

[0054] In one embodiment, the speech-text weights can be determined by analyzing the sentiment classification confidence probability output by the sentiment analysis language model or the sentiment consistency index of the text to determine the degree of trust of the semantic channel in this round of fusion, and then assigning text weights accordingly.

[0055] S11032. Determine the user's emotion vector based on the facial expression data and its corresponding facial expression weight, the voice tone data and its corresponding voice tone weight, and the voice text data and its corresponding voice text weight.

[0056] For example, a user's emotion vector is determined based on facial expression data and its corresponding facial expression weights, speech tone data and its corresponding speech tone weights, and speech text data and its corresponding speech text weights. The facial expression weights, speech tone weights, and speech text weights are used to characterize the confidence and importance of the corresponding modal data in the emotion recognition process. Weighted fusion improves the accuracy and robustness of the multimodal emotion vector representation. The user's emotion vector comprehensively reflects the user's overall emotional state at the visual, acoustic, and semantic levels. After obtaining the data and weights for each modality, the facial expression data, speech tone data, and speech text data are converted into emotion vector representations through their respective emotion encoding modules, and then weighted and fused according to their corresponding weights. This weighting process can employ methods such as weighted averaging, weighted summation, or dynamic weight allocation based on attention mechanisms to ensure that the final user emotion vector reflects multimodal emotional features while also considering the confidence of each modality.

[0057] In one embodiment, the user's emotion vector can be determined by multiplying the facial expression emotion vector, the voice tone emotion vector, and the text emotion vector by their respective weighting coefficients, and then concatenating or adding the weighted vectors to form a unified multimodal emotion fusion vector.

[0058] In one embodiment, the user sentiment vector can also be determined by inputting the weighted sentiment vectors of each modality into a multimodal fusion network, using an attention mechanism to dynamically adjust the importance of different modalities, and outputting a high-dimensional and context-aware user sentiment vector.

[0059] S120. Based on the user sentiment vector, the hot keywords, and the trained large language model, generate creative scripts and shot descriptions.

[0060] In one embodiment, a creative script and shot description are generated based on user sentiment vectors, trending keywords, and a trained large language model. The user sentiment vector represents the user's emotional tendency in a specific scenario, serving as the emotional tone input during content creation. Trending keywords represent highly relevant thematic elements in the current social context, enhancing the timeliness and dissemination of the content. The trained large language model refers to a pre-trained model with natural language understanding and generation capabilities, capable of automatically constructing structured script content and visual shot planning based on input semantic information. After obtaining the user sentiment vectors and trending keywords, the large language model is invoked for multi-dimensional semantic fusion and contextual reasoning to generate a creative script and shot description structure with emotional coherence and thematic focus. The creative script represents the overall plot development, character interactions, and emotional progression of the short video; the shot description represents the composition, dynamic elements, and timing information of each shot segment, supporting automatic synthesis and editing control of subsequent video content.

[0061] In one embodiment, the creative script can be generated by using user emotion vectors as emotion-driven input, hot keywords as content-guided input, and combining the semantic generation mechanism within the language model to output a complete text script structure that includes opening settings, conflict development, and emotional transitions.

[0062] In one embodiment, the method for generating shot descriptions may be: based on the scene semantics and emotional changes described in the creative script, calling the visual language alignment module of the language model to automatically construct components such as shot number, shot type, scene description and action cues, which are used to guide the video compositing system in content arrangement and rhythm control.

[0063] Optionally, Figure 5 A flowchart of a creative script and storyboard description generation method provided in an embodiment of this application is given. (Reference) Figure 5 The method for generating creative scripts and storyboard descriptions specifically includes: S1201. Map the user emotion vector to several user emotion labels and their corresponding user emotion intensities.

[0064] For example, user emotion vectors are mapped to several user emotion labels and their corresponding emotion intensities. The user emotion vector represents the user's current multimodal emotional state, containing multi-dimensional emotional feature information; the user emotion label represents the specific emotion category, such as joy, sadness, anger, surprise, etc.; and the user emotion intensity represents the degree or weight of emotion expression corresponding to each emotion label, reflecting the proportion and significance of that emotion in the overall emotional state. After obtaining the user emotion vector, the emotion classification mapping module is invoked. Through a pre-trained multi-classification model or emotion recognition network, the high-dimensional emotion vector is projected onto a predefined emotion label space, outputting the probability distribution or intensity score of each label, thereby determining the specific emotion label and its intensity.

[0065] In one embodiment, the method for mapping emotion labels and intensity can be as follows: calculate the probability value of each emotion category based on the Softmax classifier, use the probability as the intensity index of the corresponding emotion label, and filter out emotion labels that are higher than a preset threshold as the final output.

[0066] In one embodiment, the mapping of emotion labels and intensity can also be achieved by using a multi-label emotion recognition model that supports one user's emotion vector corresponding to multiple coexisting emotion labels, with each label equipped with an independent intensity score, which facilitates the description of complex and mixed emotional states.

[0067] Optionally, Figure 6 A flowchart illustrating a user emotion tag and its intensity determination method provided in an embodiment of this application is given. (Reference) Figure 6 The user's emotion tag and the method for determining its intensity specifically include: S12011. Extract several preset emotion tags from a preset emotion tag set, and calculate the matching confidence of the user's emotion vector with each preset emotion tag.

[0068] For example, several preset emotion tags are extracted from a pre-defined set of emotion tags, and the matching confidence score between the user's emotion vector and each preset emotion tag is calculated. The preset emotion tag set includes various emotion categories predefined by the system to standardize and unify the range of emotion expression; the matching confidence score characterizes the similarity or correspondence between the user's emotion vector and each emotion tag in the feature space, reflecting the suitability of the emotion tag to the current user's emotional state. After obtaining the user's emotion vector and preset emotion tags, the degree of matching between the user's emotion vector and each preset tag is quantified by calculating similarity indicators between vectors, such as cosine similarity, Euclidean distance, or correlation coefficient, and a matching confidence score distribution is generated accordingly.

[0069] In one embodiment, the matching confidence can be calculated by using cosine similarity to calculate the angle similarity between the user's sentiment vector and each sentiment tag vector, with a smaller angle corresponding to a higher confidence.

[0070] In one embodiment, the matching confidence score can also be calculated by using a trained emotion matching network and outputting the matching probability between the user's emotion vector and the label vector through a deep neural network model as a confidence score index.

[0071] S12012. Based on the matching confidence level and the preset confidence level threshold, determine several user emotion labels of the user emotion vector and their corresponding user emotion intensities.

[0072] For example, based on the matching confidence score and a preset confidence threshold, several user emotion labels and their corresponding user emotion intensities are determined from the user's emotion vector. The matching confidence score characterizes the degree of matching between the user's emotion vector and each preset emotion label, the confidence threshold is used to filter out emotion labels with higher confidence, and the user emotion intensity reflects the strength of the corresponding label's expression in the user's current emotional state. After obtaining the matching confidence score distribution, it is compared with the system's preset confidence threshold, and emotion labels with confidence scores higher than the threshold are selected as the final set of user emotion labels. The corresponding matching confidence score value, after normalization or linear mapping, serves as the user emotion intensity for each emotion label, used to quantify the significance of that emotion in the overall emotional state.

[0073] In one embodiment, the method for determining a user's emotion label and emotion intensity can be: using a threshold filtering strategy, outputting all labels with a matching confidence score greater than a preset threshold, and normalizing their confidence scores to obtain the emotion intensity distribution.

[0074] In one embodiment, the method for determining a user's emotion label and emotion intensity can also be: based on confidence ranking, select the top N labels with the highest confidence as output, and use the corresponding confidence as an emotion intensity indicator, supporting multi-label emotion expression.

[0075] S1202, The script prompt words are obtained by concatenating the user emotion tag, its corresponding user emotion intensity, and the hot keywords.

[0076] For example, user emotion tags, their corresponding emotion intensities, and trending keywords are concatenated to obtain script prompts. The user emotion tags and their intensity characterize the current user's emotional state and its intensity, while the trending keywords reflect current social hotspots or popular topics. The script prompts serve as input for subsequent creative script generation, guiding the language model to generate content that aligns with user emotions and trending topics. After obtaining the user emotion tags, emotion intensities, and trending keywords, predefined concatenation rules integrate various information according to a specific format or template to form structured and semantically rich text prompts, facilitating efficient understanding and utilization by the subsequent natural language processing module.

[0077] In one embodiment, the script prompt words can be concatenated as follows: the emotion labels, their intensities, and the hot keywords are arranged sequentially in the format of "emotion label 1: intensity 1, emotion label 2: intensity 2, ...; hot keyword 1, hot keyword 2, ..." to form a unified text string.

[0078] In one embodiment, the script prompt words can also be concatenated by using a natural language template to describe the emotion label and intensity as "The current user's emotion is... (emotion label and intensity), and the focus of attention includes... (hot keywords)" to generate more readable prompt text.

[0079] S1203. Generate creative scripts and shot descriptions based on the script prompts and the trained large language model.

[0080] For example, based on script prompts and a trained large language model, a creative script and shot descriptions are generated. The script prompts represent the user's current emotional state and trending keywords, serving as input to the generation model. The trained large language model refers to a pre-trained model with natural language understanding and generation capabilities, capable of automatically generating structured text content and visual narrative planning based on the prompts. After obtaining the script prompts, the large language model is invoked for semantic parsing and content construction, generating a creative script with emotional coherence and thematic focus, along with corresponding shot descriptions. The creative script represents the video's plot design, character actions, and emotional changes, while the shot descriptions specifically define the composition, shooting angle, and timing of each shot, supporting subsequent video compositing and editing control.

[0081] In one embodiment, the creative script can be generated by inputting script prompts into the text generation module of a language model and outputting detailed script content containing story clues, emotional build-up, and event development through a conditional text generation algorithm.

[0082] In one embodiment, the method for generating storyboard descriptions may be: based on the creative script content, calling the visual language alignment module to convert the scene descriptions, action cues, and emotional nodes in the text into structured storyboard information including shot numbers, shot types, and image details.

[0083] S130. Based on the creative script and the storyboard description, generate visual and audio materials, and synchronously synthesize the visual and audio materials to obtain a short video.

[0084] In one embodiment, visual and audio materials are generated based on a creative script and storyboard descriptions, and these materials are synchronously synthesized to obtain a complete short video. The creative script characterizes the overall narrative structure and emotional trajectory of the short video, providing the semantic basis for the visual and auditory content. The storyboard description characterizes the composition, transitions, and dynamic details of each shot, serving as a logical reference for material generation and editing. Visual materials refer to images, animations, or video clips used to construct the short video's visual content, while audio materials refer to auditory elements such as background music, dialogue, or ambient sound effects. After obtaining the creative script and storyboard descriptions, a multimodal generation model is used to construct semantically consistent visual and audio resources. Timeline alignment and rhythm control are performed based on the shot structure information to achieve automatic synthesis of a short video with consistent content and coordinated rhythm.

[0085] In one embodiment, visual materials can be generated by using a text-driven image / video generation model to generate image sequences or animation clips with corresponding styles and themes based on the composition, emotional atmosphere, and action information in the shot description.

[0086] In one embodiment, the audio material can be generated by: based on the dialogue information and emotional cues in the creative script, calling a text-to-speech system to generate the corresponding character voice, and at the same time, combining the background context and plot development, selecting matching background music and environmental sound effects through a music recommendation algorithm to form a complete audio track.

[0087] In one embodiment, the synchronous synthesis of visual and audio materials can be achieved by: splicing the visual segments sequentially according to the timeline and shot rhythm parameters defined in the shot description, aligning the audio materials to the corresponding shot segments, and using a synthesis engine to achieve synchronized audio and visual output and coordinated rhythm in a short video.

[0088] Optionally, Figure 7 A flowchart of a method for generating visual and audio materials according to an embodiment of this application is provided. (Reference) Figure 7 The method for generating visual and audio materials specifically includes: S1301. Based on the several described shot descriptions, match the corresponding video segments in the preset video material library, and cut and splice each video segment to obtain visual material.

[0089] For example, based on several shot descriptions, corresponding video clips are matched in a pre-set video material library, and each video clip is cropped and spliced ​​to obtain visual materials. The shot descriptions characterize the composition, action scenes, and timing of each shot in the short video, serving as a key basis for video clip retrieval and content construction. The pre-set video material library contains a large number of structured and annotated video clip resources to support shot-level content retrieval. Cropping and splicing operations are used to construct a coherent and rhythmically logical video visual sequence, forming complete visual content that conforms to the creative script's intent. After obtaining the shot descriptions, based on features such as shot keywords, composition, and emotional atmosphere, matching video clips are retrieved from the video material library. The selected materials are then cropped and optimized according to the target length, action rhythm, and visual focus of each shot. Subsequently, multiple video clips are seamlessly spliced ​​in shot order to generate a sequence of visual materials that can be used for subsequent synthesis.

[0090] In one embodiment, the method for matching video clips may be: based on semantic tags, emotion classifications, and action instructions in the shot description, keyword matching, image and text retrieval, or multimodal feature alignment models are used to select the most relevant set of video clips from the media library.

[0091] In one embodiment, the method of cropping and splicing visual materials can be as follows: the matched video segments are edited using the start and end times of the shots, the intensity of the rhythm, and the event boundaries, and the segments are organized in sequence and transition effects are set through automatic editing algorithms or template splicing logic to generate complete visual materials.

[0092] S1302. Based on the creative script and the user emotion vector, generate a dialogue audio track and a background audio track respectively, and mix the dialogue audio track and the background audio track to obtain an audio track.

[0093] For example, dialogue audio tracks and background audio tracks are generated based on the creative script and user emotion vectors, respectively. These tracks are then mixed to obtain the final audio track. The creative script provides the dialogue text and its corresponding contextual information, while the user emotion vector guides the emotional expression and intonation style during speech generation. The dialogue audio track recreates the character dialogue and plot narration, while the background audio track constructs the environmental atmosphere and emotional tone. The final audio track is the fused audio content, providing a complete acoustic expression for the short video. After obtaining the creative script and user emotion vector, the audio track is first generated based on the dialogue content. Simultaneously, the speech rate, tone, and emotional inflection during speech synthesis are controlled according to the emotion vector to ensure consistency between the dialogue audio track and the user's emotions. Subsequently, combining the script's plot segments and the user's emotional characteristics, music clips or ambient sounds matching the scene's atmosphere are selected from a pre-set background music library to generate the corresponding background audio track.

[0094] In one embodiment, the dialogue audio track can be generated by inputting the dialogue text in the creative script into the emotional speech synthesis model, and combining it with the user's emotional vector to control the emotional style of the speech, thereby generating dialogue audio with realistic tone and emotional color.

[0095] In one embodiment, the background music track can be generated by: selecting matching musical fragments or ambient sounds from structured background music resources based on the script scene type and user emotion dimension, and then cropping and aligning them according to the shot duration.

[0096] In one embodiment, the mixing process can be performed by using an audio mixing engine to overlay the dialogue track and the background track in a set ratio, automatically adjusting the volume balance and front-to-back layers to ensure that the dialogue is clearly distinguishable and the background atmosphere transitions naturally, and finally outputting a unified audio track.

[0097] S1303. Synchronize the audio track with the timeline of the visual material to obtain the audio material.

[0098] For example, the audio track is synchronized with the timeline of the visual materials to obtain the audio material. The audio track contains the complete acoustic content after mixing the dialogue and background audio tracks, while the visual material is a sequence of video frames generated by cropping and splicing based on the storyboard descriptions. Timeline synchronization ensures consistency between audio and video in the time dimension, achieving precise matching between dialogue and visual actions, and between background music and shot rhythm, generating video and audio resources suitable for final synthesis. After obtaining the audio track and visual materials, a unified timeline mapping relationship is established based on the storyboard structure and shot duration information in the creative script. The audio content undergoes time-series correction and segment alignment to ensure that each dialogue and background sound effect strictly corresponds to its corresponding video segment on the timeline.

[0099] In one embodiment, audio synchronization can be achieved by dividing the audio segment into several segments based on the start and end timestamps of the shot, and aligning each audio segment with anchor points according to the time frame precision of the visual material, ensuring that the dialogue is played during lip movements, and that the background music is dynamically adjusted according to the changes in the shot's atmosphere.

[0100] In one embodiment, the output audio material can be produced by re-encoding the synchronized audio segments into a continuous audio track, preserving the logical coordination between dialogue rhythm, background atmosphere and shot rhythm, and using it as audio input material for a short video synthesis engine.

[0101] Optionally, Figure 8 A step-by-step diagram of a short video generation method based on emotion and trending topics provided in an embodiment of this application is given. (Reference) Figure 8 The short video generation method based on emotion and trending topics specifically includes: S201. Collect emotional video data and hot topic information data.

[0102] For example, multi-source perception methods are used to acquire users' current emotional state and current social hot topics in real time, providing semantic and emotion-driven signals for subsequent content generation. Emotional video data and hot topic information data are collected. Emotional video data represents users' genuine emotional expressions in a specific time period or scenario, typically including multimodal emotional signals such as facial expressions, voice tone, and behavioral posture. Hot topic information data represents highly attention-grabbing events, topics, or keywords in the current social public opinion or online environment, reflecting the external semantic background of the user's environment. Users' audio and video streams are collected in real time through front-end terminal devices, and emotional video data is organized and labeled using time tags. Simultaneously, a hot topic information acquisition module connects to a network hot topic monitoring service to collect structured hot topic information such as high-frequency keywords, social media topics, and news summaries within a specified time period, constructing a hot topic corpus corresponding to the user's scenario.

[0103] In one embodiment, the method for collecting emotional video data may be: by calling the device's camera and microphone, recording audio and video data with the user's authorization, and constructing a structured emotional video sample set by combining contextual annotations.

[0104] In one embodiment, the method for collecting trending information data can be: accessing a network content aggregation platform, social media API, or news aggregation engine to extract popular keywords, event tags, and frequently followed content in real time, constructing a structured trending information flow for subsequent sentiment and semantic fusion modeling.

[0105] In one embodiment, while a user expresses "excitement" in front of the camera, a web crawler identifies "World Cup" as a current hot topic, thereby jointly constructing an emotion-semantic input vector.

[0106] S202. Generate creative scripts and storyboard descriptions using a large language model.

[0107] For example, by combining user sentiment vectors and current trending keywords, a large language model is used to automatically generate creative script content and corresponding storyboard planning. The large language model generates the creative script and storyboard descriptions. Here, the large language model refers to a deep neural network model trained on a large-scale corpus, possessing natural language understanding and generation capabilities. It can generate structurally complete and logically coherent creative text based on input contextual information and semantic prompts. The creative script represents the short video's storyline, character behavior, dialogue semantics, and emotional tone, while the storyboard description represents the visual content, shot type, timing, and visual instructions of each shot, supporting the automatic generation and editing of subsequent video content. After obtaining user sentiment data, trending information data, or other semantic prompts, script prompts are constructed and input into the large language model, guiding it to generate script content that matches the user's current emotional state and trending context, and outputting storyboard information according to the shot logic structure.

[0108] In one embodiment, the creative script can be generated by constructing structured script prompts from emotion tags, emotion intensities, and hot keywords, inputting them into a language generation model, and generating a complete text script containing structures such as opening guidance, emotional progression, and plot climax.

[0109] In one embodiment, the method for generating storyboard descriptions may be as follows: based on the creative script content, extract the corresponding scene information for each segment of the narrative through a natural language analysis and visual intent mapping module, and format it into storyboard descriptions such as shot number, shooting type, composition requirements, and duration.

[0110] In one embodiment, copywriting and visual designs depicting victory celebrations are generated based on the emotions of "excitement" and the keyword "World Cup," such as "fans running and celebrating with national flags in high spirits" and "players jumping up and hugging."

[0111] S203, Generate visual and audio materials.

[0112] For example, based on the generated creative script and storyboard description, the generation model and material library are invoked to automatically synthesize visual and audio content, and timeline alignment and style matching are completed. Visual and audio materials are generated. Visual materials represent the visual content of the short video, including character images, scene composition, action performance, and visual atmosphere, and are typically constructed based on the storyboard description. Audio materials represent auditory elements such as dialogue, background music, and environmental sound effects, and are generated in conjunction with the creative script content and the user's emotional state to support the video's emotional expression and rhythm control. After obtaining the creative script and storyboard description, the visual generation module and audio synthesis module are invoked respectively. Based on the visual requirements, character actions, dialogue content, and emotional settings of each shot, the corresponding visual image sequence and audio track are automatically constructed. Visual materials can come from video material library segment matching, virtual character generation, or AI image synthesis, while audio materials can be automatically generated based on text-to-speech, background music recommendation, or environmental sound effect synthesis.

[0113] In one embodiment, the visual material can be generated by: retrieving structured video clips from a video material library based on the shot description, or by synthesizing a sequence of images that meets the shot requirements based on an AI image generation model.

[0114] In one embodiment, the audio material can be generated by using an emotional speech synthesis model to generate dialogue tracks from the dialogue content in the creative script, and by combining the user's emotion vector and the script rhythm to match music fragments with consistent emotions from a background music resource library to synthesize a complete set of audio materials.

[0115] In one embodiment, images of "fans cheering" and "exciting goals on the field" can be synthesized, and combined with narration and rousing background music to generate video segments with high emotional intensity.

[0116] S204, Short Video Generation.

[0117] For example, a short video is generated. This short video presents complete multimodal content integrating visual and audio materials, characterized by coherent visual rhythm, consistent emotional tone, and clear thematic expression. It is the final output of the preceding creative script and emotional guidance. After acquiring the constructed visual and audio materials, they are synchronized and composited along the timeline to ensure that dialogue, visuals, background music, and shot rhythm are consistent in the temporal dimension. Subsequently, post-processing operations such as video encoding / decoding, transition processing, visual enhancement, and subtitle overlay are performed, ultimately outputting a video file that meets playback standards.

[0118] In one embodiment, the method for generating short videos may be: aligning audio materials with corresponding visual materials based on timestamp information to construct a unified media timeline, and then performing mixing and exporting through a video compositing engine.

[0119] In one embodiment, short video generation may further include: automatically inserting title pages, transition animations, and end markers according to the script structure, and adjusting the emotional style of the audio-visual performance by combining user emotion vectors to improve the emotional expressiveness and viewing experience of the final video.

[0120] S205. Collect user feedback data.

[0121] For example, user feedback data is collected. This data characterizes users' subjective evaluations and behavioral responses to generated short video content, serving as a crucial basis for assessing short video quality, optimizing generation strategies, and improving user satisfaction. Feedback data can include explicit and implicit feedback. Explicit feedback includes likes, comments, shares, and ratings, while implicit feedback includes completion rate, dwell time, bounce rate, and repeat viewing behavior. After the short video is generated and pushed to the user's device, feedback data related to short video interaction is acquired in real-time or periodically through data collection, interface interaction recording, and behavior analysis modules. This data is then structured and stored in conjunction with user characteristic information and contextual environment, providing foundational data support for subsequent sentiment modeling, content recommendation, and model training.

[0122] In one embodiment, user feedback data can be collected by embedding an interactive event listener in the short video playback interface to record user actions such as liking, commenting, favorited, and skipping, and marking the corresponding timestamps and video segment indexes.

[0123] In one embodiment, feedback data collection may further include: analyzing user viewing trajectories through a big data analytics platform, extracting implicit behavioral characteristics such as viewing ratio, number of pauses, and interaction hotspots, and associating them with video content characteristics to form feedback tags or preference models.

[0124] S206. Optimize the generation parameters of large language models using reinforcement learning algorithms.

[0125] For example, based on user interaction behaviors such as likes, shares, and viewing time, feedback signals are continuously collected, and content generation parameters are optimized through reinforcement learning mechanisms to achieve adaptive content evolution. Reinforcement learning algorithms are used to optimize the generation parameters of the large language model. Specifically, the reinforcement learning algorithm is used to continuously adjust and optimize the content generation strategy of the large language model in a feedback-driven closed-loop system to maximize the reward value represented by user feedback signals. Generation parameters refer to control variables that affect the structure, style, and emotional relevance of the large language model's output content, such as sampling strategies and sentiment bias weights, and are important factors driving the quality of model generation. After collecting user feedback data, a reinforcement learning framework with the large language model as the main strategy is constructed. User feedback is transformed into reward signals, guiding the model to dynamically adjust its generation strategy based on the feedback performance of historical generation results, thereby achieving continuous optimization of creative scripts and storyboard descriptions.

[0126] In one embodiment, reinforcement learning optimization can be achieved by constructing a reinforcement learning architecture based on policy gradient or proximal policy optimization, using user click-through rate, sentiment matching score, or video playback completion rate as reward functions to update the model's output policy.

[0127] In one embodiment, the optimization of generation parameters can be achieved by scoring different combinations of prompt words and generation results during the training phase, using high-scoring results as positive samples to guide the model to learn generation paths with higher emotional expression, structural coherence, and hot topic relevance, and improving the overall generation performance through continuous online learning.

[0128] In one embodiment, if users interact frequently with World Cup videos featuring a "passionate style," the system adjusts the parameters for "high-contrast visual style" and "high-frequency emotional transitions" accordingly to enhance the appeal of the next round of generated content.

[0129] Based on the above embodiments, Figure 9 This is a structural block diagram of a short video generation device based on emotion and trending topics, provided as an embodiment of this application. (Reference) Figure 9 The short video generation device based on emotion and hot topics provided in this embodiment specifically includes: a collection module 21, a determination module 22, a generation module 23, and a synthesis module 24.

[0130] The acquisition module 21 is configured to acquire emotional video data for displaying the user's emotional state and the number of hot topics in the current time period; the determination module 22 is configured to determine the user's emotional vector and hot keywords based on the emotional video data and the hot keywords; the generation module 23 is configured to generate a creative script and a storyboard description based on the user's emotional vector, the hot keywords, and the trained large language model; and the synthesis module 24 is configured to generate visual and audio materials based on the creative script and the storyboard description, and to synchronously synthesize the visual and audio materials to obtain a short video.

[0131] Based on the above embodiments, the determining module 22 includes: a data extraction unit configured to extract several facial expression data and several voice audio data from the emotional video data; a data recognition unit configured to recognize the voice video data to obtain several voice intonation data and several voice text data; a vector component unit configured to determine an expression emotion vector, a intonation emotion vector, and a text emotion vector based on the facial expression data, the voice intonation data, and the voice text data, respectively; and an emotion vector unit configured to determine a user emotion vector based on the expression emotion vector, the intonation emotion vector, and the text emotion vector.

[0132] Based on the above embodiments, the vector component unit includes: a facial weight subunit, configured to determine the facial weight corresponding to each of the facial expression data based on the facial similarity between the facial expression data and historical facial expression data; a voice weight subunit, configured to determine the voice weight corresponding to each of the voice audio data based on the voiceprint similarity between the voice audio data and historical voice audio data; and a vector component subunit, configured to determine an expression emotion vector, a tone emotion vector, and a text emotion vector based on each of the facial expression data and its corresponding facial weight, and each of the voice intonation data and each of the voice text data and its corresponding voice weight, respectively.

[0133] Based on the above embodiments, the emotion vector unit includes: a vector weight subunit, configured to determine corresponding facial expression weights, speech tone weights, and speech-text weights based on the recognition confidence of the facial expression emotion vector, the tone emotion vector, and the text emotion vector, respectively; and an emotion vector subunit, configured to determine a user emotion vector based on the facial expression data and its corresponding facial expression weights, the speech tone data and its corresponding speech tone weights, and the speech-text data and its corresponding speech-text weights.

[0134] Based on the above embodiments, the generation module 23 includes: an emotion tagging unit, configured to map the user emotion vector into several user emotion tags and their corresponding user emotion intensities; a prompt word splicing unit, configured to splice the user emotion tags and their corresponding user emotion intensities with the hot keywords to obtain script prompt words; and a creative generation unit, configured to generate creative scripts and storyboard descriptions based on the script prompt words and a trained large language model.

[0135] Based on the above embodiments, the emotion tagging unit includes: a matching confidence subunit, configured to extract several preset emotion tags from a preset emotion tag set and calculate the matching confidence between the user emotion vector and each of the preset emotion tags; and an emotion tagging subunit, configured to determine several user emotion tags of the user emotion vector and their corresponding user emotion intensities based on the matching confidence and a preset confidence threshold.

[0136] Based on the above embodiments, the synthesis module 24 includes: a visual material unit, configured to match corresponding video segments in a preset video material library based on several of the shot descriptions, and to crop and splice each of the video segments to obtain visual material; an audio track unit, configured to generate a dialogue track and a background track based on the creative script and the user emotion vector respectively, and to mix the dialogue track and the background track to obtain an audio track; and an audio material unit, configured to synchronize the audio track with the timeline of the visual material to obtain audio material.

[0137] The short video generation device based on emotion and trending topics provided in this application embodiment achieves efficient linkage analysis of user emotional states and social trending information, intelligent generation of script content, and automatic execution of material synthesis processes by constructing a structured system architecture composed of multiple functional units such as a collection module, a determination module, a generation module, and a synthesis module. Based on collecting user emotional video data and trending information of the current time period, the device uses the collection module to extract multimodal emotional signals reflecting the user's psychological state and simultaneously acquires high-frequency trending content in the network environment. Through vectorization analysis and keyword extraction of the collected data by the determination module, the device can accurately construct user emotional vectors and trending keyword sets as the semantic basis for subsequent creative generation. The generation module integrated within the device introduces a trained large language model to automatically generate script text and shot descriptions based on user emotional vectors and trending keywords, realizing human-computer collaboration and context awareness in the script creation process. Simultaneously, to ensure the structural integrity and expressiveness of the finished video, the synthesis module supports material mapping and multimodal content synthesis of the text script. Through timeline synchronization and unified output strategies for visual and audio materials, it achieves the fusion presentation of emotional expression and trending content.

[0138] The short video generation apparatus based on emotion and trending topics provided in this application can be used to execute the short video generation method based on emotion and trending topics provided in the above embodiments, and has corresponding functions and beneficial effects.

[0139] Figure 10 This is a schematic diagram of the structure of a short video generation device based on emotion and trending topics provided in an embodiment of this application. (Refer to...) Figure 10 The short video generation device based on emotion and trending topics includes: a processor 31, a memory 32, a communication device 33, an input device 34, and an output device 35. The number of processors 31 and the number of memories 32 in the short video generation device based on emotion and trending topics can be one or more. The processor 31, memory 32, communication device 33, input device 34, and output device 35 of the short video generation device based on emotion and trending topics can be connected via a bus or other means.

[0140] The memory 32, as a computer-readable storage medium, can be used to store software programs, computer-executable programs, and modules, such as program instructions / modules corresponding to the emotion- and hot-topic-based short video generation method in any embodiment of this application (e.g., the acquisition module 21, determination module 22, generation module 23, and synthesis module 24 in the emotion- and hot-topic-based short video generation device). The memory 32 may primarily include a program storage area and a data storage area. The program storage area may store the operating system and at least one application program required for a function; the data storage area may store data created based on device usage, etc. Furthermore, the memory 32 may include high-speed random access memory and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other non-volatile solid-state storage device. In some instances, the memory may further include memory remotely located relative to the processor, and these remote memories can be connected to the device via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0141] The communication device 33 is used for data transmission.

[0142] The processor 31 executes various functional applications and data processing of the device by running software programs, instructions and modules stored in the memory 32, thereby realizing the above-mentioned method for generating short videos based on emotions and hot topics.

[0143] Input device 34 can be used to receive input digital or character information, and to generate key signal inputs related to user settings and function control of the device. Output device 35 may include display devices such as a display screen.

[0144] The short video generation device based on emotion and trending topics provided above can be used to execute the short video generation method based on emotion and trending topics provided in the above embodiments, and has corresponding functions and beneficial effects.

[0145] This application embodiment also provides a storage medium containing computer-executable instructions. When executed by a computer processor, the computer-executable instructions are used to execute a short video generation method based on emotion and trending topics. The short video generation method based on emotion and trending topics includes: collecting emotional video data for displaying a user's emotional state and trending topic information data for the current time period; determining a user emotion vector and trending topic keywords based on the emotional video data and the trending topic information data, respectively; generating a creative script and a storyboard description based on the user emotion vector, the trending topic keywords, and a trained large language model; generating visual materials and audio materials based on the creative script and the storyboard description; and synchronously synthesizing the visual materials and the audio materials to obtain a short video.

[0146] Storage medium – any type of memory device or storage device. The term “storage medium” is intended to include: mounting media, such as CD-ROM, floppy disk, or magnetic tape devices; computer system memory or random access memory, such as DRAM, DDR RAM, SRAM, EDO RAM, Rambus RAM, etc.; non-volatile memory, such as flash memory, magnetic media (e.g., hard disk or optical storage); registers or other similar types of memory elements, etc. Storage medium may also include other types of memory or combinations thereof. Furthermore, storage medium may reside in a first computer system in which a program is executed, or it may reside in a different second computer system connected to the first computer system via a network (such as the Internet). The second computer system can provide program instructions to the first computer for execution. The term “storage medium” can include two or more storage media residing in different locations (e.g., in different computer systems connected via a network). Storage medium may store program instructions (e.g., specifically implemented as a computer program) executable by one or more processors.

[0147] Of course, the computer-executable instructions provided in the embodiments of this application are not limited to the short video generation method based on emotion and hot topics as described above, but can also execute related operations in the short video generation method based on emotion and hot topics provided in any embodiment of this application.

[0148] The emotion- and trending short video generation apparatus, storage medium, and emotion- and trending short video generation device provided in the above embodiments can execute the emotion- and trending short video generation method provided in any embodiment of this application. For technical details not described in detail in the above embodiments, please refer to the emotion- and trending short video generation method provided in any embodiment of this application.

[0149] The above description is merely a preferred embodiment and the technical principles employed in this application. This application is not limited to the specific embodiments described herein, and various obvious changes, readjustments, and substitutions that can be made by those skilled in the art will not depart from the scope of protection of this application. Therefore, although this application has been described in detail through the above embodiments, this application is not limited to the above embodiments, and may include many other equivalent embodiments without departing from the concept of this application. The scope of this application is determined by the scope of the claims.

Claims

1. A short video generation method based on emotion and trending topics, characterized in that, include: Collect emotional video data and hot topic information data for the current time period to display the user's emotional state, and determine the user's emotional vector and hot topic keywords based on the emotional video data and the hot topic information data respectively; Based on the user sentiment vector, the hot keywords, and the trained large language model, generate creative scripts and shot descriptions; Visual and audio materials are generated based on the creative script and the storyboard description, and the visual and audio materials are synchronously synthesized to obtain a short video.

2. The short video generation method based on emotion and trending topics according to claim 1, characterized in that, Determining the user's emotional vector based on the aforementioned emotional video data includes: Several facial expression data and several voice audio data are extracted from the emotional video data, and the voice video data is recognized to obtain several voice intonation data and several voice text data. Based on the facial expression data, the speech intonation data, and the speech-text data, respectively, the facial expression emotion vector, the speech intonation emotion vector, and the text emotion vector are determined; The user's emotion vector is determined based on the facial expression emotion vector, the tone emotion vector, and the text emotion vector.

3. The short video generation method based on emotion and trending topics according to claim 2, characterized in that, The step of determining the facial expression emotion vector, the tone emotion vector, and the text emotion vector based on the facial expression data, the speech tone data, and the speech text data respectively includes: The facial weights corresponding to each facial expression data are determined based on the facial similarity between the facial expression data and historical facial expression data. The voice weight of each voice audio data is determined based on the voiceprint similarity between the voice audio data and historical voice audio data. The facial expression vector, tone expression vector, and text expression vector are determined based on each of the facial expression data and its corresponding facial weights, as well as each of the speech tone data and each of the speech text data and its corresponding speech weights.

4. The short video generation method based on emotion and trending topics according to claim 2, characterized in that, The process of determining the user's emotion vector based on the facial expression emotion vector, the tone emotion vector, and the text emotion vector includes: The corresponding facial expression weight, speech tone weight, and speech-text weight are determined based on the recognition confidence scores of the facial expression emotion vector, the speech tone emotion vector, and the text emotion vector, respectively. The user's emotion vector is determined based on the facial expression data and its corresponding facial expression weights, the voice tone data and its corresponding voice tone weights, and the voice text data and its corresponding voice text weights.

5. The short video generation method based on emotion and trending topics according to claim 1, characterized in that, The process of generating creative scripts and shot descriptions based on the user sentiment vector, the hot keywords, and the trained large language model includes: The user emotion vector is mapped to several user emotion labels and their corresponding user emotion intensities; The script prompt words are obtained by concatenating the user emotion tags, their corresponding user emotion intensities, and the hot keywords. Based on the script prompts and the trained large language model, creative scripts and shot descriptions are generated.

6. The short video generation method based on emotion and trending topics according to claim 5, characterized in that, The step of mapping the user emotion vector to several user emotion labels and their corresponding user emotion intensities includes: Extract several preset emotion tags from a preset set of emotion tags, and calculate the matching confidence of the user's emotion vector with each preset emotion tag; Based on the matching confidence level and a preset confidence threshold, several user emotion labels and their corresponding user emotion intensities are determined from the user emotion vector.

7. The method for generating short videos based on emotion and trending topics according to any one of claims 1-6, characterized in that, The generation of visual and audio materials based on the creative script and the storyboard description includes: Based on the aforementioned shot descriptions, corresponding video segments are matched in a preset video material library, and each video segment is cropped and spliced ​​to obtain visual material. Based on the creative script and the user emotion vector, a dialogue audio track and a background audio track are generated respectively, and the dialogue audio track and the background audio track are mixed to obtain an audio track; The audio track is synchronized with the timeline of the visual material to obtain the audio material.

8. A short video generation device based on emotion and trending topics, characterized in that, include: The data acquisition module is used to collect emotional video data to display the user's emotional state and hot topic information data for the current time period; The determination module is used to determine the user's emotional vector and hot keywords based on the emotional video data and the hotspot information data, respectively; The generation module is used to generate creative scripts and shot descriptions based on the user sentiment vector, the hot keywords, and the trained large language model. The compositing module is used to generate visual and audio materials based on the creative script and the storyboard description, and to synchronously compose the visual and audio materials to obtain a short video.

9. A short video generation device based on emotion and trending topics, characterized in that, include: One or more processors; A memory that stores one or more programs, which, when executed by one or more processors, enable the one or more processors to implement the emotion- and trending short video generation method as described in any one of claims 1-7.

10. A storage medium containing computer-executable instructions, characterized in that, The computer-executable instructions, when executed by a computer processor, are used to perform the short video generation method based on emotion and trending topics as described in any one of claims 1-7.

Images