Method and system for automatic generation and matching of motion effect text labels for scene transitions

Abstract

This application provides a method and system for automatically generating and matching motion effect text tags for comic book storyboard transitions, belonging to the field of automated editing technology for comic book storyboards. The method includes: acquiring a sequence of comic book storyboard video frames, character dialogue audio, and synchronized subtitle text, and processing them separately to generate a video frame sequence, noise-reduced audio data, and standardized text data, which are then input into a visual emotion feature extraction network, a speech emotion recognition network, and a text emotion analysis network, respectively, to obtain micro-expression temporal features, body movement temporal features, intonation prosody features, audio spectrum energy features, lexical emotion polarity features, and sentence semantic emotion features, which are then input into a cross-modal feature fusion network to generate a fused emotion feature vector. Similarity matching is performed in an emotion tag library, and emotional intensity modulation is performed by determining the plot weight coefficient. Enhanced emotion tags are generated and input into a text tag generation model along with transition type information and scene theme information to generate transition motion effect text tags.

Description

Technical Field

[0001] This application relates to the technical field of automated editing of comic book storyboards, and in particular to a method and system for automatically generating and matching motion effect text tags for comic book storyboard transitions. Background Technology

[0002] Comic strips enhance the narrative atmosphere and viewing experience through dynamic transition effects, and motion effect text tags are the core basis for driving the automatic rendering of these transition effects. In current comic strip production, transition effect text tags rely on manual, subjective design and annotation based on visuals, dialogue, and audio, resulting in inefficiency, inconsistent standards, and strong subjectivity. Existing technologies often rely solely on single-dimensional information from text or visuals to determine emotions, failing to integrate multimodal emotional information such as character micro-expressions, body movements, vocal tone, and dialogue semantics, leading to low accuracy in emotion recognition. Furthermore, existing solutions cannot adaptively generate motion effect tags based on plot rhythm weighting, transition type, and scene theme changes, resulting in poor matching between transition effects and plot atmosphere / scene content. This leads to a disconnect between emotions and visual transitions, severely impacting the narrative coherence and visual quality of the final comic strip.

[0003] Therefore, there is an urgent need for a method and system for automatically generating and matching animated text tags for comic book scene transitions. Summary of the Invention

[0004] To address the aforementioned technical issues, this application provides a method and system for automatically generating and matching motion effect text tags for comic book scene transitions.

[0005] A first aspect of this application provides a method for automatically generating and matching motion effect text tags for comic book scene transitions, including: Obtain the raw data of the comic series, including storyboard video frame sequences, character dialogue audio, and synchronized subtitle text; The video sequence of the storyboard is decomposed and preprocessed frame by frame to generate a video frame sequence; the audio of the character dialogue is denoised to generate denoised audio data; and the synchronized subtitle text is normalized to generate normalized text data. The video frame sequence is input into a preset visual emotion feature extraction network to extract the micro-expression timing features and limb movement timing features of the character; the noise-reduced audio data is input into a preset speech emotion recognition network to extract intonation prosody features and audio spectrum energy features; the normalized text data is input into a preset text emotion analysis network to extract lexical emotion polarity features and sentence semantic emotion features. The micro-expression temporal features, body movement temporal features, intonation prosody features, audio spectrum energy features, lexical sentiment polarity features, and sentence semantic sentiment features are input into a cross-modal feature fusion network to generate a fused sentiment feature vector for the current scene. Based on the fused emotional feature vector, similarity matching is performed in a preset emotional tag library to obtain an initial emotional tag that matches the current storyboard scene. Obtain plot development information of the comic series, determine the plot weight coefficient of the current storyboard scene based on the plot development information, and use the plot weight coefficient to modulate the emotional intensity of the initial emotional tag to generate an enhanced emotional tag; Obtain the transition type information between the current storyboard and the next storyboard, as well as the scene theme information of the preceding and following storyboards. Input the enhanced emotion tag, the transition type information, and the scene theme information into a preset motion effect text tag generation model to generate transition motion effect text tags.

[0006] A second aspect of this application provides a system for automatically generating and matching motion effect text tags for comic book scene transitions, including: The data acquisition module is used to acquire the raw data of the comic series, including storyboard video frame sequences, character dialogue audio, and synchronized subtitle text; The data processing module is used to perform frame-by-frame decomposition and preprocessing on the storyboard video sequence to generate a video frame sequence; to perform noise reduction processing on the character dialogue audio to generate noise-reduced audio data; and to perform text normalization processing on the synchronized subtitle text to generate normalized text data. The feature extraction module is used to input the video frame sequence into a preset visual emotion feature extraction network to extract the micro-expression temporal features and limb movement temporal features of the character; input the noise-reduced audio data into a preset speech emotion recognition network to extract intonation prosody features and audio spectrum energy features; and input the normalized text data into a preset text emotion analysis network to extract lexical emotion polarity features and sentence semantic emotion features. The feature fusion module is used to input the micro-expression temporal features, body movement temporal features, intonation prosody features, audio spectrum energy features, lexical sentiment polarity features and sentence semantic sentiment features into the cross-modal feature fusion network to generate the fused sentiment feature vector of the current scene. The tag matching module is used to perform similarity matching in a preset emotional tag library based on the fused emotional feature vector to obtain an initial emotional tag that matches the current storyboard scene. The weight determination module is used to obtain plot development information of the comic series, determine the plot weight coefficient of the current storyboard scene based on the plot development information, and use the plot weight coefficient to modulate the emotional intensity of the initial emotional tag to generate an enhanced emotional tag. The tag generation module is used to obtain the transition type information between the current storyboard and the next storyboard, as well as the scene theme information of the preceding and following storyboards. The enhanced emotion tag, the transition type information, and the scene theme information are input into a preset motion effect text tag generation model to generate transition motion effect text tags.

[0007] A third aspect of this application provides an electronic device, including a memory, a processor, and a computer program stored in the memory and running on the processor, wherein the processor executes the computer program to implement the steps of the above-described method for automatically generating and matching animated text tags for comic book scene transitions.

[0008] A fourth aspect of this application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the above-described method for automatically generating and matching motion effect text tags for comic book scene transitions.

[0009] The beneficial effects of the automatic generation and matching method and system for animation text tags in comic book scene transitions provided in this application are as follows: This application preprocesses and extracts features from raw data in three modalities: video, audio, and subtitles, overcoming the shortcomings of existing single-modal analysis, which suffers from incomplete information and inaccurate emotion recognition. Through cross-modal fusion of multi-dimensional emotional features, a globally unified scene emotion feature vector is constructed. By modulating plot weights and linking transition and scene information in modeling, end-to-end automatic output from raw materials to standardized transition animation tags is achieved. This solves the problems of time-consuming and labor-intensive manual annotation, significant subjective differences, and poor tag adaptability. It unifies the animation style of the entire series, improves the automation level of post-production editing of comic books, adapts to industrialized mass production scenarios, reduces production costs, and enhances the visual atmosphere and narrative coherence of the final product. Attached Figure Description

[0010] Figure 1 A flowchart illustrating the automatic generation and matching method for animation text tags in comic book scene transitions provided in an embodiment of this application; Figure 2 A structural block diagram of an automatic generation and matching system for animation text tags for storyboard transitions in comics, provided in an embodiment of this application; Figure 3 This is a schematic block diagram of an electronic device provided in an embodiment of this application. Detailed Implementation

[0011] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted so as not to obscure the description of this application with unnecessary detail.

[0012] To make the purpose, technical solution, and advantages of this application clearer, the following will be described in conjunction with the appendix. Figure 1 - Appendix Figure 3 The following is an explanation using specific examples.

[0013] Please refer to Figure 1 , Figure 1 This is a flowchart illustrating a method for automatically generating and matching motion effect text tags for comic book scene transitions, provided in an embodiment of this application. The method includes: S101: Obtain the raw data of the comic series, including storyboard video sequence, character dialogue audio, and synchronized subtitle text.

[0014] In this embodiment, the comic series is a dynamic comic drama series, a continuous short video series formed by video synthesis, dubbing, and subtitle overlay of multiple static comic panels, featuring fixed panel shots, character dialogue, and plot narrative structure. The raw data is the original material file that has not undergone algorithmic processing or cleaning before the post-production of the comic series, and serves as the input data source.

[0015] In this embodiment, the storyboard video sequence is a collection of single-shot short video clips arranged in the order of the plot in the comic series; the comic series is edited in a shot-by-shot manner, with each shot corresponding to a storyboard video, which is arranged in chronological order to form a sequence.

[0016] In this embodiment, the character dialogue audio is a dedicated voice dubbing track for the comic series, which includes the characters' lines and voices, distinct from background music, sound effects, and ambient noise, and carries information about the characters' emotions, tone, and rhythm.

[0017] In this embodiment, the synchronized subtitle text is a subtitle text file that is strictly aligned with the timeline of the video footage and the dialogue, accurately corresponding to the start and end times and text content of each line of dialogue, and carrying the semantics of the plot and the emotions of the dialogue.

[0018] S102: Perform frame-by-frame decomposition and preprocessing on the storyboard video sequence to generate a video frame sequence; perform noise reduction processing on the character dialogue audio to generate noise-reduced audio data; perform text normalization processing on the synchronized subtitle text to generate normalized text data.

[0019] In this embodiment, frame-by-frame decomposition breaks down a continuous short video sequence into individual still images at a fixed frame rate, enabling pixel-level and keypoint-level feature analysis of each frame. Preprocessing (video preprocessing) involves standardizing and cleaning the decomposed original image frames, unifying their size, removing interference, correcting the image quality, and eliminating irrelevant noise such as material quality differences, watermarks, and black borders, ensuring consistent data specifications for the input network. The video frame sequence is a standardized set of still images arranged in the original playback time order after frame-by-frame decomposition and preprocessing; it serves as the standard input data for the visual feature network.

[0020] In this embodiment, audio noise reduction processing involves filtering out noise from the original dubbing audio, removing background noise, airflow sounds, environmental noise, and residual background music, retaining only the pure character's voice intonation information, and avoiding irrelevant audio interference with emotion recognition. The noise-reduced audio data is a standardized audio file that has been cleaned of interference noise and retains only the pure character's voice, used to extract intonation, rhythm, and spectral energy features.

[0021] In this embodiment, text normalization involves cleaning and correcting the original subtitles: removing special symbols, garbled characters, interjections, and extra spaces; standardizing sentence structure; correcting typos; standardizing punctuation; regularizing sentences; and eliminating invalid text interference. The normalized text data is a clean, regularized, and segmented standard pure dialogue text, semantically clean and free of redundant interference, enabling text sentiment networks to accurately extract semantic and emotional polarities.

[0022] S103: Input the video frame sequence into a preset visual emotion feature extraction network to extract the micro-expression temporal features and limb movement temporal features of the character; input the noise-reduced audio data into a preset speech emotion recognition network to extract intonation prosody features and audio spectrum energy features; input the normalized text data into a preset text sentiment analysis network to extract lexical sentiment polarity features and sentence semantic sentiment features.

[0023] In this embodiment, the preset visual emotion feature extraction network is a spatiotemporal convolutional neural network with fixed parameters, trained on a dataset of cartoon character images. It outputs dynamic emotion features of two types: facial micro-expressions and human body limbs, in sync with the temporal sequence of video frames. It also includes built-in sub-modules for face detection and skeletal key point tracking.

[0024] Specifically, the pre-defined visual emotion feature extraction network goes through six processing layers in sequence: audio preprocessing layer, time-frequency feature extraction layer, one-dimensional temporal convolutional layer, global temporal pooling layer, emotion feature encoding layer, and feature output layer, completing audio noise reduction, spectrum conversion, and temporal emotion information extraction layer by layer.

[0025] The pre-defined speech emotion recognition network is built using a temporal convolutional neural network. The backbone stacks multiple sets of one-dimensional convolutional modules to capture the temporal fluctuations in audio. An integrated Mel-frequency conversion operator extracts the time-frequency information of the sound, and pooling layers compress redundant data. Supervised training is conducted based on multiple types of emotional audio samples, resulting in stable outputs of two types of speech emotion features. The network input is denoised temporal audio waveform data; the output consists of two independent feature vectors: one representing the intonation prosody feature reflecting changes in tone, and the other representing the audio spectral energy feature representing fluctuations in sound intensity.

[0026] In this embodiment, micro-expression temporal features are a sequence of dynamic offset vectors of facial key points arranged in frame-time order, recording the continuous changes in minute deformations of the eyebrows, eyes, and mouth frame by frame, representing the character's subtle and fleeting hidden emotions, and possessing temporal continuity. Limb movement temporal features are a set of dynamic features of key points of the entire human skeleton changing over time, including temporal values ​​such as limb swing frequency and trunk tilt rate, representing the character's significant emotional limb expressions.

[0027] In this embodiment, the preset speech emotion recognition network is an acoustic feature extraction model trained using comic book dubbing corpus. The input is a denoised human voice waveform, which automatically analyzes the acoustic parameters of the speech's inherent emotion.

[0028] Specifically, the pre-defined speech emotion recognition network goes through six processing layers in sequence: audio noise reduction preprocessing layer, Mel time-frequency feature conversion layer, block temporal coding layer, multi-head self-attention Transformer coding layer, emotion temporal fusion layer, and feature normalization output layer, completing audio format conversion, temporal correlation modeling and emotion feature extraction layer by layer.

[0029] The pre-built speech emotion recognition network uses the Transformer speech model. First, the audio is converted into Mel-spectral temporal block embedding vectors. A multi-head self-attention mechanism is used to model the intonation correlation between different speech segments, eliminating the problem of long-distance temporal information loss. A feedforward network is used to perform emotion mapping on the time-frequency features. Fine-tuning training is completed using multi-emotion-annotated audio samples, and two independent speech emotion representations are output. The network input is normalized continuous audio waveform data after noise reduction; the output consists of two sets of independent temporal feature vectors: intonation prosodic features recording the speed of speech fluctuations, and audio spectral energy features carrying information on sound intensity fluctuations.

[0030] In this embodiment, the intonation prosodic features are temporal parameters composed of the fundamental frequency of speech, speech rate, pause duration, and pitch fluctuations, corresponding to the speaker's emotional fluctuations; the pitch rises when excited, and the speech rate slows down when sad. The audio spectral energy features are the energy distribution of each frequency band of the speech signal and the inter-frame energy change curves; high-frequency energy increases when excited / angry, and the overall spectral energy is lower when the energy is low.

[0031] In this embodiment, the preset text sentiment analysis network is the obtained text sentiment encoding network. The input is a regular subtitle sentence, and the sentiment representations at the word and sentence levels are extracted in layers.

[0032] Specifically, the pre-designed text sentiment analysis network goes through six processing layers: text normalization and cleaning layer, word segmentation and embedding layer, multi-head attention encoding layer, local semantic extraction layer, sentiment feature mapping layer, and feature output layer, gradually completing text normalization, semantic encoding, and separation and extraction of two types of sentiment features.

[0033] The pre-defined text sentiment analysis network employs a lightweight BERT Transformer text model. It first performs word segmentation and word embedding transformation on the normalized dialogue text. Multi-head self-attention is then used to mine lexical associations and semantic connections within the entire sentence. Shallow fully connected layers are then used to map lexical and sentence-level sentiment representations respectively. Fine-tuning is performed using annotated cartoon dialogue samples to separate and output two independent types of sentiment features. The network input is normalized dialogue text data with a uniform format and invalid symbols removed. The output consists of two sets of feature vectors: lexical sentiment polarity features representing the emotional tendency of individual words, and sentence semantic sentiment features carrying the overall sentiment of the entire dialogue.

[0034] In this embodiment, the lexical sentiment polarity feature is a word semantic sentiment vector that distinguishes between positive, negative, and neutral sentiment words and stores the sentiment strength value corresponding to each word in the dialogue. The sentence semantic sentiment feature is a global sentiment vector after fusing the entire sentence context. It eliminates ambiguity of single words through context and fully expresses the comprehensive emotional tendency of a single sentence.

[0035] S104: Input micro-expression temporal features, body movement temporal features, intonation prosody features, audio spectrum energy features, lexical emotional polarity features, and sentence semantic emotional features into a cross-modal feature fusion network to generate a fused emotional feature vector for the current scene.

[0036] In this embodiment, the cross-modal feature fusion network is used to unify features of three different dimensions and scales: visual, audio, and text, to eliminate data differences between modalities, and to weight and fuse various emotional information through an attention mechanism to output a feature vector with a unified dimension.

[0037] In this embodiment, the cross-modal feature fusion network sequentially passes through six network layers: modal feature normalization layer, feature dimension alignment layer, modal embedding encoding layer, cross-modal self-attention interaction fusion layer, global sentiment semantic aggregation layer, and fusion feature output layer, to complete multimodal feature normalization, cross-domain association modeling, and global sentiment vector generation step by step.

[0038] The pre-defined cross-modal feature fusion network is constructed using a cross-modal Transformer fusion model. It employs a multi-branch input structure designed for three heterogeneous emotional modalities: visual, speech, and text. First, it unifies and normalizes the dimensions of six independent emotional features, distinguishing different modal feature attributes through modality-specific embedding encoding. Then, it utilizes a multi-head self-attention mechanism to model the cross-modal associations and complementarities among video micro-expressions, body movements, speech prosody, audio energy, lexical polarity, and sentence semantics, suppressing modal redundancy noise and aggregating global emotional information. Finally, it outputs a unified-dimensional global fused emotional feature vector. The network input consists of six independent emotional features: micro-expression temporal features, body movement temporal features, intonation prosody features, audio spectral energy features, lexical emotional polarity features, and sentence semantic emotional features. The network output is a single-dimensional unified fused emotional feature vector, integrating multi-dimensional emotional information from visual, audio, and text sources, used for subsequent emotional tag similarity matching.

[0039] In this embodiment, the fused emotional feature vector is a fixed-dimensional one-dimensional numerical vector obtained by weighted integration of multimodal information. It includes all emotional information such as the expression, body movements, tone of voice, and semantics of the lines in the current storyboard scene. It is standardized data that represents the overall emotion of the storyboard scene and is used for emotional tag similarity matching.

[0040] In this embodiment, micro-expression temporal features, limb movement temporal features, intonation prosody features, audio spectrum energy features, lexical emotional polarity features, and sentence semantic emotional features are the six subdivided emotional features output, which correspond to subtle facial expressions, changes in human body movements, human voice intonation rhythm, audio frequency energy, word sense tendency, and the overall emotion of the entire sentence, respectively, and serve as the original emotional input data for the multimodal system to be fused.

[0041] S105: Based on the fused emotional feature vector, perform similarity matching in the preset emotional tag library to obtain the initial emotional tag that matches the current scene.

[0042] In this embodiment, the preset emotion tag library is a standardized emotion lexicon manually constructed and adapted to the plot scenes of the comic series. It has a three-dimensional tree structure (basic emotion, theme projection, and intensity modification). The preset emotion tag library stores all standard emotion words that can be used for transitions in the comic series and is the unique standard lexicon for matching output tags. Similarity matching is calculated using the cosine similarity of vector space. The output real-scene emotion vector is compared with the vector of each tag in the tag library in terms of spatial distance. The larger the value, the greater the emotional fit.

[0043] In this embodiment, the initial emotion tag is the raw emotion tag obtained solely from visuals, audio, and text AI recognition without any plot intensity modulation. It only represents the true emotional tone of the current scene and does not include plot climax / foreshadowing weighting; it is the prerequisite tag for strengthening the emotion tag.

[0044] S106: Obtain plot development information of the comic series, determine the plot weight coefficient of the current storyboard scene based on the plot development information, and use the plot weight coefficient to modulate the emotional intensity of the initial emotional label to generate an enhanced emotional label.

[0045] In this embodiment, plot development information is temporal plot data that characterizes the narrative rhythm of the comic series. It is divided into two categories: plot beat timeline with labeled global script; and plot node types predicted by a classifier from screen features when there is no labeled script, which are used to distinguish between the foreshadowing, development, climax and ending narrative stages.

[0046] In this embodiment, the plot weight coefficient is a pre-set dimensionless adjustment value. Different plot nodes correspond to different coefficients, with the coefficient for climax nodes being greater than that foreshadowing and development nodes. This is used to quantify and amplify / reduce the intensity of emotional expression.

[0047] In this embodiment, emotional intensity modulation is achieved by adjusting the basic emotional intensity value of the initial emotional tag through numerical calculation, changing the intensity of emotions through plot weight, and then matching the corresponding degree modifiers to complete the tag upgrade.

[0048] In this embodiment, the enhanced emotional label is a complete emotional label that is based on the initial emotional label, with intensity modifiers that are adapted to the rhythm of the plot and have emotional levels, so as to match the emotional tension of the narrative segment in which the storyboard is located.

[0049] S107: Obtain the transition type information between the current storyboard and the next storyboard, as well as the scene theme information of the preceding and following storyboards. Input the enhanced emotion tag, transition type information, and scene theme information into the preset motion effect text tag generation model to generate transition motion effect text tags.

[0050] In this embodiment, the current shot and the next shot are two adjacent shots in the sequence of the comic strip. The current shot is the ending shot of the preceding sequence, and the next shot is the subsequent shot that is about to be cut into. There is a unique transition relationship between the two.

[0051] In this embodiment, the transition type information is the lens transition style information calculated through inter-frame pixel difference, including four sub-classes of transition: switch, fade, wipe, and dissolve, which are used to define the visual form of transition between shots.

[0052] In this embodiment, the scene theme information of the preceding and following shots is the core semantic keywords of the scene obtained by performing object detection on the keyframes of the preceding and following shots respectively. These keywords represent the content of the current shot, the scene environment, and the main object, and are used to describe the content logic of the shot transition.

[0053] In this embodiment, the preset animation text tag generation model is used to integrate three types of information: emotion, scene, and transition, and automatically combine and output complete animation text tags that conform to the animation editing specifications.

[0054] Specifically, the preset animated text tag generation model goes through five processing layers in sequence: information regularization preprocessing layer, multi-source information encoding layer, slot matching mapping layer, template semantic splicing layer, and standardized tag output layer, which successively completes the organization of multiple types of heterogeneous information, rule matching and structured text generation.

[0055] The pre-defined motion effect text tag generation model is a template-rule-driven structured generation model that requires no deep learning training. It is built based on pre-defined tag templates, transition-action mapping relationships, and scene splicing rules. The model has an independent information parsing branch that classifies and parses enhanced emotion tags, transition type information, and scene theme information. Information is filled using a fixed slot matching mechanism, and the tags are automatically spliced ​​and integrated according to pre-defined word order rules, outputting structured text tags with a unified format and standardized semantics, exhibiting strong scene adaptability and stability. The model input consists of three types of heterogeneous information: modulated enhanced emotion tags, transition type information between adjacent shots, and scene theme information between preceding and following shots. The model output is standardized, structured transition motion effect text tags, which can be directly used for subsequent motion effect parameter parsing and editing rendering.

[0056] In this embodiment, the transition animation text tag is the final output of a complete standardized tag text that can be directly used for editing. It also includes the mood atmosphere, scene jump relationship, and transition action effect, and is the top-level text instruction that drives the post-production animation rendering.

[0057] As can be seen from the above, this application overcomes the shortcomings of existing single-modal analysis, such as one-sided information and inaccurate emotion recognition, by preprocessing and extracting features from raw data in three modalities: video, audio, and subtitles. Through cross-modal fusion of multi-dimensional emotional features, a globally unified scene emotion feature vector is constructed. By modulating plot weights and modeling transitions and scene information linkages, end-to-end automatic output from raw materials to standardized transition animation labels is achieved. This solves the problems of time-consuming and labor-intensive manual annotation, significant subjective differences, and poor label adaptability. It unifies the animation style of the entire series, improves the automation level of post-production editing of animated series, adapts to industrialized mass production scenarios, reduces production costs, and enhances the visual atmosphere and narrative coherence of the final product.

[0058] In one embodiment of this application, a video frame sequence is input into a preset visual emotion feature extraction network to extract the micro-expression temporal features and limb movement temporal features of a character, including: Perform face detection and facial key point localization on each frame in the video frame sequence, and extract the coordinate sets of eye key points, eyebrow key points, and mouth key points. Calculate the coordinate offset of each key point between the current frame and the previous frame, arrange the coordinate offsets in chronological order, and generate micro-expression temporal features; Perform human pose estimation on each frame in the video frame sequence and extract the two-dimensional coordinate sequence of key points of the bones in the shoulder, elbow, wrist, knee and ankle; Temporal difference analysis was performed on the two-dimensional coordinate sequence of each skeletal key point to extract the limb swing frequency feature and the trunk tilt angle change rate feature, thus forming the limb movement temporal feature.

[0059] In this embodiment, face detection refers to locating the face region of a person within a single frame, distinguishing the background from the character's face, and eliminating invalid frames without a face. Facial key point localization refers to accurately marking the pixel coordinates of the eyes, eyebrows, and lips within the detected face bounding box, quantifying subtle changes in facial expressions. The eye / eyebrow / mouth key point coordinate set is a two-dimensional set of pixel coordinates of all key points corresponding to the facial features in a single frame, providing the basic data for calculating dynamic changes in facial expressions.

[0060] In this embodiment, the coordinate offset is the difference between the horizontal and vertical coordinates of the same facial key point in two adjacent frames, representing the slight displacement of facial features over time, corresponding to micro-expression fluctuations. The micro-expression temporal feature is a sequence of key point coordinate offsets arranged in chronological order according to playback time, continuously characterizing the subtle changes in the character's facial expressions over a short period of time.

[0061] In this embodiment, human pose estimation is a module that identifies the human body contour in a single frame and outputs the two-dimensional coordinates of the joints of the entire skeleton. The two-dimensional coordinate sequence of skeletal key points consists of the pixel coordinates of each joint in the shoulder, elbow, wrist, knee, and ankle in a single video frame, which completely records the spatial position changes of the human limbs.

[0062] In this embodiment, temporal difference is performed by calculating the difference between adjacent frames of the same skeletal keypoint coordinates in consecutive frames to extract the amplitude and speed of limb dynamic changes. The limb swing frequency feature is calculated by statistically analyzing the number of limb swings per unit time after temporal difference of skeletal coordinates, distinguishing between calm, excited, and flustered limb states. The trunk tilt angle change rate feature is calculated by determining the trunk skeletal tilt angle frame by frame and then calculating the inter-frame angle change amplitude, reflecting emotional limb movements such as leaning forward, leaning back, and turning sideways. The limb movement temporal feature is formed by combining the limb swing frequency and trunk tilt angle change rate, uniformly representing the dynamic emotional expression of a person's limbs.

[0063] As can be seen from the above, this embodiment, through a defined and refined temporal visual emotion feature extraction strategy, calculates micro-expression temporal features by inter-frame offset of facial key points and extracts limb dynamic features based on temporal difference of human skeletal key points, which differs from existing single-frame static emotion recognition methods. It can accurately capture subtle changes in facial expressions, limb movements, and posture tilts of anime characters, fully adapting to the exaggerated and nuanced emotional performances characteristic of anime. It solves the shortcomings of static visual features in representing the gradual process of emotion and the lack of dynamic emotion capture, improving the subtlety and accuracy of visual emotion representation. It provides high-precision, highly temporally correlated visual feature support for cross-modal emotion fusion, ensuring that the overall emotion judgment fits the realistic atmosphere of the scene.

[0064] In one embodiment of this application, based on the fused emotional feature vector, similarity matching is performed in a preset emotional tag library to obtain an initial emotional tag that matches the current scene, including: The pre-defined emotional tag library is constructed as a three-dimensional tree structure, which includes a basic emotional layer, a theme projection layer, and an intensity modification layer. The basic emotional layer includes multiple basic emotional category nodes, the theme projection layer includes projected emotional nodes related to comic book themes, and the intensity modification layer includes degree modifier nodes. The tag encoder maps the tag text corresponding to each node in the preset emotion tag library to tag embedding vectors and stores them in the tag embedding vector index library. Calculate the cosine similarity between the fused sentiment feature vector and each tag embedding vector in the tag embedding vector index library, and select all tags with a cosine similarity greater than a preset similarity threshold to form a candidate tag set; The label with the highest cosine similarity is selected from the candidate label set as the initial sentiment label.

[0065] In this embodiment, the three-dimensional tree-structured emotion tag library is a hierarchical tree storage architecture, which is divided into three layers from top to bottom and linked hierarchically. It uniformly includes all emotion words applicable to comics and TV series. The layers correspond to basic emotions, theme-specific emotions, and emotion intensity modifications, so as to realize the orderly retrieval of emotion tags.

[0066] In this embodiment, the basic emotion layer is the first-level node in the tree structure, storing the underlying emotion vocabulary nodes. These represent the common basic emotions across all comics, such as happiness, sadness, anger, resentment, and calmness, forming the core foundation for emotional expression. The basic emotion category nodes are tree nodes under the basic emotion layer that independently store the text of a single basic emotion. Each node is bound to a unique basic emotion text and a corresponding number. The theme projection layer is the intermediate-level node in the tree structure. It derives emotion nodes adapted to specific themes within the comics by utilizing the upper-level basic emotions, thus refining the common emotions in conjunction with the plot theme, such as the thriller aspect of a suspense theme.

[0067] In this embodiment, the projected emotion node is an independent node within the theme projection layer that binds to the theme-specific subdivided emotional text, establishing a parent-child relationship with the corresponding basic emotion node in the upper layer. The intensity modification layer is the lowest level node in the tree structure, used to modify the intensity of the upper-level emotions, storing only degree-related modifiers. The degree modifier node is an independent node within the intensity modification layer that stores strong and weak modifiers, including gradient modifications such as slightly, somewhat, suddenly, and extremely.

[0068] In this embodiment, the tag encoder is a text encoding network that takes tag text as input and outputs fixed-dimensional numerical vectors, achieving a unified representation of text vectorization. The tag embedding vector is a low-dimensional numerical vector obtained by the encoder from the tag text; the vector space distance can represent the semantic similarity of sentiment words. The tag embedding vector index is a retrieval database that centrally stores the tag embedding vectors of all levels of nodes and binds them to the corresponding tag text, used for rapid similarity comparison.

[0069] In this embodiment, cosine similarity is a vector space similarity calculation index with a value range of [-1, 1]. The closer the value is to 1, the more closely the fused sentiment vector and the tag vector match the emotional semantics.

[0070] In this embodiment, the preset similarity threshold is a set similarity screening threshold used to filter irrelevant sentiment tags with too low a match.

[0071] Specifically, the preset similarity threshold is determined by cross-validation statistical calibration of multi-genre comic series samples. First, storyboard samples covering various plot styles are collected and manually labeled with standard emotional tags. Then, different threshold values ​​within the gradient range are traversed, and the tag matching precision and recall corresponding to each threshold are calculated. The value with the best combination of the two indicators is selected as the baseline threshold. Then, special samples with ambiguous emotions and niche themes are used to complete the calibration and correction. Finally, a similarity threshold that meets the matching needs of the entire comic series scene is obtained, balancing the filtering ability of irrelevant tags and the retention effect of niche emotional tags.

[0072] In this embodiment, the candidate tag set is a collection of all tags with a cosine similarity greater than a preset similarity threshold. Invalid tags with insufficient matching are filtered out, narrowing the scope of optimal tag retrieval. The initial sentiment tag is the tag text with the highest cosine similarity in the candidate set, which only represents the original emotional tone of the storyboard and has not been modified by plot intensity.

[0073] As can be seen from the above, this embodiment, through a three-dimensional tree-structured hierarchical emotion tag library, covers the emotional expression needs of all scenarios in animation and drama through a three-layer structure of basic emotion, theme projection, and intensity modification, solving the problems of coarse classification and limited theme adaptation in existing tag libraries. By using tag embedding vectorization and a cosine similarity intelligent matching mechanism, the optimal initial emotion tags can be accurately selected instead of manual subjective tag selection. Through threshold-based candidate set selection and optimal matching strategies, problems such as emotion mismatch, weak matching, and generalized matching are avoided, improving the fit and professionalism of scene-specific emotion tags. This provides a precise and standardized emotion tag foundation for subsequent emotion intensity modulation and motion effect tag generation, improving the overall system output accuracy.

[0074] In one embodiment of this application, obtaining plot development information of a comic strip and determining the plot weight coefficient of the current storyboard scene based on the plot development information includes: Obtain the global storyboard file of the comic. The global storyboard file includes a pre-marked plot beat timeline. The plot beat timeline is set with multiple plot node type identifiers in chronological order. The plot node type identifiers include at least the foreshadowing node, the development node, the climax node, and the ending node. Extract the start and end timestamps of the current storyboard scene, map the start and end timestamps to the plot beat timeline, and determine the plot node type to which the current storyboard scene falls; Query the preset plot weight mapping table to obtain the plot weight coefficient corresponding to the plot node type; among them, the plot weight coefficient of the climax node is set to be greater than the plot weight coefficient of the foreshadowing node and the development node.

[0075] In this embodiment, the global storyboard file is a complete narrative script carrier generated during the animation production stage. It records all the storyboard timing, screen content, and plot narrative rhythm related annotation information, and serves as the original file basis for determining the plot rhythm.

[0076] In this embodiment, the plot beat timeline is a linear time scale built within the script according to the playback sequence, which segments the duration of the entire comic series and binds them to the narrative rhythm annotations, so as to achieve a one-to-one correspondence between shot time and plot stage.

[0077] In this embodiment, the plot node type identifier is a classification mark marked at the segment position of the timeline, used to distinguish the narrative stage to which the shot belongs, including four basic identifiers: foreshadowing, development, climax, and ending.

[0078] In this embodiment, the setup nodes are the early scenes that lay the groundwork for the plot and foreshadow future developments, with generally low emotional tension. The development nodes are the middle scenes where the plot gradually advances and the conflict slowly escalates, with moderate emotional tension. The climax nodes are the core scenes where the conflict erupts and the emotional impact is strongest, with the highest emotional tension. The ending nodes are the final scenes where the story concludes and the conflict subsides, with the emotion becoming more subdued.

[0079] In this embodiment, the start timestamp and end timestamp are the start and end time sequence markers of a single segment of video within the entire comic series playback stream, used to locate the interval of the shot in the global timeline.

[0080] In this embodiment, the plot weight mapping table is a pre-constructed key-value lookup table, indexed by plot node type, with each type of node matched with a unique dimensionless weight coefficient to distinguish the emotional amplification ratio of different narrative stages.

[0081] In this embodiment, the plot weight coefficient is a dimensionless adjustment parameter used to amplify or contract the basic emotional intensity. The climax corresponds to a larger coefficient to enhance the emotional expression intensity of the scene.

[0082] As can be seen from the above, this embodiment divides multiple plot nodes into categories through the timeline of the comic book's plot rhythm, accurately maps plot stages through storyboard timestamps, and differentiates the weight coefficients of foreshadowing, development, climax, and ending. This solves the shortcomings of existing technologies that neglect narrative rhythm and lack uniformity and layering in emotional intensity throughout the film. By amplifying the emotional weight of climax nodes, it aligns with the narrative fluctuations of film and television dramas, achieving dynamic changes in emotional intensity with the plot rhythm. It avoids the problems of traditional motion effect labels resulting in flat emotional expression, weak atmosphere, and insufficient tension in climax scenes. It ensures that the intensity of transition motion effects aligns with the logical progression of the plot, enhancing the narrative layering and emotional impact of the comic book, achieving a deep integration of technical effects and cinematic artistic expression.

[0083] In one embodiment of this application, a method for automatically generating and matching motion effect text tags for comic book scene transitions further includes: When the global storyboard file does not include a pre-annotated plot beat timeline, perform time-intensive sampling on the video frame sequence to extract the image complexity features and image brightness change rate features of each sampled frame. The image complexity features and image brightness change rate features of each sampled frame are input into the pre-trained narrative rhythm classifier in chronological order. The narrative rhythm classifier outputs the predicted probability distribution of plot node types corresponding to each time window. Based on the predicted probability distribution of plot node types, the plot node type with the highest predicted probability is selected as the plot node type to which the current storyboard scene belongs, and the corresponding plot weight coefficient is obtained by querying the preset plot weight mapping table.

[0084] In this embodiment, temporal dense sampling involves uniformly extracting multiple keyframes from a continuous video frame sequence at fixed time intervals. This preserves the temporal changes in the shot without increasing computational load, and is used for subsequent rhythm feature analysis.

[0085] In this embodiment, the image complexity feature is a quantitative feature that characterizes the richness of content, element density, and texture complexity of a single frame. The stronger the plot conflict and the more frequent the scene transitions, the higher the image complexity.

[0086] In this embodiment, the image brightness change rate feature is the magnitude and speed of change of the overall image brightness between adjacent sampling frames. It is used to reflect the jumping of light and shadow in the lens and sudden changes in atmosphere, and is an important visual representation of the ups and downs of the plot rhythm.

[0087] In this embodiment, the narrative rhythm classifier is a multi-classification network trained with a large number of comic book scene rhythm samples. It automatically determines whether the current plot is in the prelude, development, climax, or ending stage based on the dynamic features of the scene.

[0088] Specifically, the narrative rhythm classifier goes through five processing layers in sequence: temporal sampling preprocessing layer, dual feature fusion encoding layer, time window sliding block layer, narrative rhythm feature inference layer, and probability normalization output layer, to complete the regularization of scene temporal features, local rhythm modeling, and probability prediction of multiple types of plot nodes layer by layer.

[0089] The pre-defined narrative rhythm classifier is constructed using a temporal convolutional classification network, designed specifically for the rhythmic changes in comic book scenes. It is capable of fully automatic plot identification without labeled scripts. The network uses temporal convolutional modules to capture the continuous fluctuations in scene complexity and brightness change rate, models local plot rhythm features through sliding time windows, and combines this with a fully connected classification layer to output multi-node probability distributions. Supervised training is conducted using a massive dataset of comic book temporal samples (foreshadowing, development, climax, and ending) to achieve accurate adaptive classification of the storyboard narrative stage. The network input consists of a sequence of sampled frames arranged chronologically, representing the scene complexity and brightness change rate features. The network output is the predicted probability distribution of the four plot node types corresponding to each time window, including the confidence probabilities of foreshadowing, development, climax, and ending nodes, used to select the optimal plot node type.

[0090] In this embodiment, the time window is a short analysis interval that is segmented into segments of the storyboard video according to time sequence. It is used to capture local changes in the rhythm of the scene and ensure that the plot judgment fits the local plot state.

[0091] In this embodiment, the plot node type prediction probability distribution is a multidimensional probability result output by the classifier, which represents the confidence probability that the current window belongs to one of the four types of nodes: setup, development, climax, and ending.

[0092] In this embodiment, the optimal probability screening strategy is to select the plot type corresponding to the maximum probability as the final judgment result, thereby achieving fully automatic plot node recognition under unlabeled scripts.

[0093] In this embodiment, the plot weight mapping table is a preset lookup table, which outputs the corresponding emotion weight coefficient based on the identified plot nodes.

[0094] As can be seen from the above, this embodiment, by adding an automatic plot recognition branch under unannotated scripts, automatically predicts the plot node types of storyboards by combining the temporal features of scene complexity and brightness changes with a pre-trained narrative rhythm classifier, thus overcoming the shortcomings of manually annotating rhythm scripts. It achieves adaptive plot weight calculation with and without annotated scripts, improving the system's versatility and adaptability to various scenarios. It solves the shortcomings of small-to-medium-sized animated series lacking standardized scripts and unable to modulate emotional weights, automatically distinguishing plot rhythms and dynamically adapting emotional intensity without manual pre-annotation, further improving the fully automated production chain and reducing the cost of pre-processing materials.

[0095] In one embodiment of this application, the initial sentiment label is modulated with sentiment intensity using a plot weighting coefficient to generate an enhanced sentiment label, including: The initial sentiment labels are extracted to obtain the basic sentiment intensity value; The modulated emotional intensity value is obtained by multiplying the base emotional intensity value by the plot weight coefficient. Based on the modulated emotional intensity value, a matching intensity modifier is selected from the intensity modification layer of the preset emotional tag library, and the intensity modifier is added to the preset position of the initial emotional tag to generate an enhanced emotional tag.

[0096] In this embodiment, the basic emotional intensity value is a fixed quantitative value that comes with the initial emotional tag. It is pre-configured by the tag library and is used to characterize the inherent intensity benchmark of this type of basic emotion, without distinguishing the rhythm of the plot narrative.

[0097] In this embodiment, emotion intensity modulation is a quantitative processing logic that adjusts the intensity of emotion tags by numerical calculation, and realizes the differentiation of emotion intensity based on the plot stage of the scene.

[0098] In this embodiment, the modulated emotional intensity value is a new value calculated by multiplying the basic emotional intensity value by the plot weight coefficient, which integrates the dual dimensions of the intensity of the emotional base itself and the tension of the plot narrative.

[0099] In this embodiment, the intensity modifier layer is the bottom-level node set of the three-dimensional tree-structured sentiment tag library, storing modifier words with varying degrees of intensity. Different numerical ranges correspond to different levels of strength in the modifier words. Intensity modifier words are text words used to modify basic sentiments and reflect the intensity of emotions, and they exist in a complete gradient hierarchy from weak to strong. The preset position is a predefined fixed word order for the modifier words, and a unified format for concatenating modifier words with basic sentiment tags.

[0100] In this embodiment, the enhanced emotion tag is a complete emotion text composed of a basic emotion tag and intensity modifiers that match the tension of the plot, which has both basic emotion type and layered emotion intensity information.

[0101] As can be seen from the above, this embodiment achieves quantitative emotional intensity modulation by multiplying the basic emotional intensity value by the plot weight coefficient. Based on the modulated value, it automatically matches hierarchical intensity modifiers, completing the intelligent enhancement and upgrade of the initial tags. This overcomes the shortcomings of existing tags, which only have emotional categories, lack intensity differentiation, and have a single level of complexity. It enables the same emotion to be expressed in multiple levels according to the intensity of the plot. This results in the final emotional tags having a precise intensity gradient, fitting the rhythmic changes of plot development, progression, and climax. It also makes the emotional concentration of transition effects differentiated and refined, enhancing the emotional expressiveness of animation transitions and avoiding the problem of homogenized and bland overall animation atmosphere.

[0102] In one embodiment of this application, obtaining transition type information between the current storyboard and the next storyboard, as well as scene theme information of the preceding and following storyboards, includes: Extract the last frame of the current storyboard and the first frame of the next storyboard; Calculate the pixel-by-pixel difference image between the last frame and the first frame, and count the proportion of pixels in the pixel-by-pixel difference image whose pixel values ​​are greater than a preset change threshold; When the pixel ratio is greater than or equal to the preset first ratio threshold, the transition type is determined to be a switch. Obtain the differential pixel state in the pixel-by-pixel differential image. The differential pixel state includes gradient diffusion distribution and directional sweep distribution. When the pixel ratio is greater than or equal to the preset second ratio threshold and less than the preset first ratio threshold, and the differential pixel state shows a gradual diffusion distribution, the transition type is determined to be a gradient. When the pixel ratio is greater than or equal to the preset second ratio threshold and less than the preset first ratio threshold, and the differential pixel state is directional sweep distribution, the transition type is determined to be wipe. When the pixel ratio is less than the preset third ratio threshold, the transition type is determined to be dissolve; The preset first ratio threshold is greater than the preset second ratio threshold, and the preset second ratio threshold is greater than the preset first ratio threshold. Obtain the keyframes of the current scene and the next scene, and input them into the pre-trained target detection network to obtain the category label and corresponding confidence score of each detected target; The category label corresponding to the detected target with the highest confidence score in the keyframe of the current scene is selected as the theme keyword of the previous scene, and the category label corresponding to the detected target with the highest confidence score in the keyframe of the next scene is selected as the theme keyword of the subsequent scene.

[0103] In this embodiment, the last frame of the current shot and the first frame of the next shot are the beginning and end images at the junction of two adjacent shots, serving as the comparison reference images for determining the transition form of the shots.

[0104] In this embodiment, the pixel-by-pixel difference image is a difference image generated by performing a difference operation on the grayscale / color values ​​of pixels at the same coordinate positions in two comparison frames. The larger the pixel difference, the more obvious the change in the image.

[0105] In this embodiment, the preset change threshold is a critical standard for pixel difference. Only pixels with a difference greater than this value are determined to have undergone image changes, thus filtering out invalid differences caused by slight light and shadow noise.

[0106] Specifically, the preset change threshold is determined by statistical calibration of multiple sets of comic book transition samples. The first and last frames of the storyboard, including various transition effects such as switching, fading, wipe, and dissolve, are collected. The pixel difference between frames is calculated frame by frame. The distribution range of the difference between effective changed pixels and noise pixels under various transitions is statistically analyzed. The effective difference generated by the real scene switching is distinguished from the small difference caused by light and shadow jitter and screen noise. The boundary value between the two difference ranges is taken as the benchmark threshold. Robust calibration is then performed using low-brightness and low-definition comic book material samples. Finally, the preset change threshold that can filter out irrelevant noise in the picture and accurately identify changes in the real shot picture is obtained.

[0107] In this embodiment, the pixel ratio is the percentage of the total number of pixels in the differential image that meet the change determination conditions out of all pixels in the entire image, and is used to measure the overall change range of the lens image.

[0108] In this embodiment, the preset first proportional threshold, the preset second proportional threshold, and the preset third proportional threshold represent three segmented critical proportional values ​​for determining the layer, which are used to divide the three different ranges of screen changes: switching, fading / wiping, and dissolve.

[0109] The preset first ratio threshold is determined by statistical calibration of multiple types of anime transition samples. First, a large number of first and last difference images corresponding to four types of standard transitions, namely switch, fade, wipe, and dissolve, are collected. The distribution range of effective change pixel ratio for each type of transition is statistically analyzed. The lowest change pixel ratio of the switch transition is extracted as the benchmark value. Then, robust calibration is carried out in combination with interference material samples such as low light, low image quality, and partial occlusion. The critical ratio that can completely distinguish hard cuts and smooth transitions is defined. In this way, the preset first ratio threshold that is suitable for various anime screen materials and can accurately distinguish between large screen changes and smooth shot transitions is obtained.

[0110] The preset second ratio threshold is determined by statistical calibration of multiple types of anime transition samples. First, batch collection of inter-frame difference pixel ratio data corresponding to three types of transitions—gradient, wipe, and dissolve—is conducted to distinguish the distribution boundaries between the low-change interval of dissolve and the medium-change interval of gradient and wipe. The minimum pixel change ratio of gradient and wipe transitions is used as the benchmark value. Then, local character occlusion, screen lighting fluctuations, and low-resolution materials are introduced to complete robustness calibration, and the critical ratio that can separate small-amplitude blending dissolves from medium-amplitude slow transitions is defined, thus achieving accurate definition of the gradient and wipe transition intervals.

[0111] The preset third ratio threshold is determined by statistical calibration of multiple types of smooth transition samples from comics. First, the inter-frame difference pixel ratio data corresponding to the dissolve transition is collected in batches. The pixel ratio range of invalid small changes such as pure screen light and shadow fluctuations and image noise is statistically analyzed. The lower limit of the effective change pixel ratio of the real dissolve transition is extracted as the benchmark value. Then, robustness calibration is completed through dark screen, partial occlusion of characters, and low-definition comic materials. The critical ratio that can distinguish between meaningless screen noise and real dissolve blending effect is defined, and dissolve transitions with only slight screen blending changes are accurately identified.

[0112] In this embodiment, the differential pixel state refers to the spatial distribution of differential pixels within the image, categorized into two types: gradient diffusion distribution and directional sweep distribution. Gradient diffusion distribution is a distribution pattern where the changing pixels diffuse uniformly from the center of the image outwards, corresponding to fade-in / fade-out transitions. Directional sweep distribution is a strip-like distribution where the changing pixels move along a single fixed direction (horizontal, vertical, or diagonal), corresponding to wipe transitions such as push-pull and venetian blind transitions.

[0113] In this embodiment, the transition is a hard cut, where a large area of ​​the preceding and following images is instantly and completely replaced without a slow transition process. The fade is a smooth transition, with the changed pixels spreading evenly and slowly, resulting in a gradual overall replacement speed. The wipe is a directional sliding transition, where the image gradually covers and switches along a fixed direction. The dissolve is a slow overlapping and blending of images, with only a small number of pixels changing slowly, resulting in a small overall change.

[0114] In this embodiment, a storyboard keyframe is a representative frame within a single storyboard that can fully represent the core scene and main object of that segment.

[0115] In this embodiment, the pre-trained object detection network is a detection model trained using animation scene, character, and object samples. It can identify the main subject within the image and output classification labels and confidence scores. The detected object category label is a standardized text label corresponding to the object, scene, or character identified by the network. The confidence score is a value between 0 and 1 output by the network, representing the accuracy and reliability of the recognition result; the higher the value, the stronger the recognition reliability.

[0116] Specifically, the pre-trained object detection network sequentially passes through a keyframe preprocessing layer, a shallow feature convolution extraction layer, a multi-scale feature fusion layer, a target candidate region screening layer, a category classification and discrimination layer, and a confidence normalization output layer, completing image purification, semantic feature extraction, target localization, and category confidence calculation layer by layer.

[0117] The pre-trained object detection network employs a lightweight, single-stage detection model structure. It uses a convolutional backbone network to extract multi-scale visual features from comic book scenes, integrating deep and shallow semantic information to adapt to diverse targets such as characters, scenes, and props. A pre-defined anchor box mechanism filters effective target regions, and a combination of classification and confidence branches completes target discrimination. Through fine-tuning training with large-scale comic book storyboard samples, it can accurately identify the core content of storyboard scenes. The network input is a standardized keyframe image of the storyboard; the network output is the category label of all detected targets in the image and their corresponding confidence scores, used to select the target with the highest confidence score as the scene theme keywords for the preceding and following storyboard scenes.

[0118] In this embodiment, the thematic keywords of the front and back shot scenes are the category labels corresponding to the subjects with the highest confidence in the front and back shots, which are used to summarize the core scene content of the two shots.

[0119] As can be seen from the above, this embodiment accurately identifies four types of transitions—switching, fading, wipe, and dissolve—by using the inter-frame pixel difference ratio and distribution pattern, overcoming the limitations of coarse-grained recognition that only distinguishes between hard cuts and fade-in / fade-out. Simultaneously, by combining keyframe target detection to automatically extract thematic keywords from the preceding and following scenes, it achieves dual accurate perception of transition styles and scene semantics. This solves the problems of low transition recognition accuracy, inability to subdivide transition types, and lack of association with scene content, providing precise scene and transition basis for generating exclusive animated text tags that fit the screen content and transition form, thus improving the scene adaptability of the tag content.

[0120] In one embodiment of this application, enhanced emotion tags, transition type information, and scene theme information are input into a preset motion effect text tag generation model to generate transition motion effect text tags, including: Get the preset tag template string, which includes the emotion prefix slot, scene description slot and transition action slot; Fill the emotion prefix slots with enhanced emotion tags; The scene theme keywords of the first and second scenes are concatenated with preset connectors to generate a scene description string and fill it into the scene description slot. Based on the transition type information, query the preset transition-action mapping table to obtain the corresponding transition action words; According to the preset arrangement order of the emotion prefix slot, scene description slot and transition action slot in the preset tag template string, the filling content of each slot is concatenated into a complete transition animation text tag.

[0121] In this embodiment, the motion effect text tag generation model is a rule-driven model with fixed tag templates and mapping tables as its core. It does not require deep learning to generate and is responsible for uniformly organizing three types of information: emotion, scene, and transition, and outputting standardized motion effect text tags in a fixed format.

[0122] In this embodiment, the tag template string is a standard text frame with a predefined fixed word order and reserved filling spaces, which unifies the output format of all parting animation tags and internally divides them into three independent filling slots.

[0123] In this embodiment, the emotion prefix slot is a fixed empty space in the template used to place enhanced emotion tags, uniformly placed at the beginning of the text, used to indicate the overall emotional atmosphere of the shot. The scene description slot is a reserved empty space in the middle section of the template, storing the scene transition description text after combining two shots. The transition action slot is a reserved empty space at the end of the template, placing the standardized action vocabulary corresponding to the transition form of the shot.

[0124] In this embodiment, the preset connector is a fixed linking text used to connect the keywords of the two consecutive shot scenes, clearly expressing the scene transition logic. The scene description string is a short sentence formed by combining the theme keywords of the consecutive shot scenes with connectors, which fully describes the scene changes during the shot transition.

[0125] In this embodiment, the transition-action mapping table is a preset key-value lookup table that uses the transition type identifier as an index to match unique and standardized transition action terms. Transition action terms are standardized professional descriptive terms corresponding to four types of transition effects: toggle, fade, wipe, and dissolve.

[0126] In this embodiment, the preset arrangement order of the slots is a fixed order of the three types of content—emotion, scene, and transition—as specified by the template, ensuring that the word order of all output tags is consistent. The transition animation text tag is a complete text composed of the content of each slot according to the template order, and includes three types of information: emotion, scene jump, and shot transition, serving as the instruction text for later rendering animation effects.

[0127] As can be seen from the above, this embodiment, through a multi-slot template splicing mechanism, structurally combines enhanced emotional tags, scene semantics, and transition actions to generate complete motion effect text tags, thus achieving the expression of emotional atmosphere, scene content, and transition forms. By standardizing template slot filling and fixing the word order, the format of all storyboard motion effect tags in the entire series is ensured to be uniform and logically standardized, solving the problems of messy styles, inconsistent expressions, and incomplete content in manual tagging. The generated tags possess emotional, scene, and motion effect attributes, are semantically complete, and highly adaptable, accurately corresponding to the artistic needs of different shot transitions, enhancing the atmosphere and uniformity of the transitions in the final animated series.

[0128] In one embodiment of this application, after generating the transition animation text tag, the method further includes: The transition animation text tags are parsed into structured animation parameter fields, which include at least the animation type identifier, duration parameter, motion curve type, and color offset direction. The duration parameter is determined based on the emotional intensity value of the enhanced emotional label, where the higher the emotional intensity value, the shorter the duration parameter. Based on the transition type information, determine the motion curve type: when the transition type is a switch or wipe, the motion curve type is set to linear or ease in and ease out; when the transition type is a fade or dissolve, the motion curve type is set to ease in and ease out. Encapsulate structured motion effect parameter fields into metadata files in a preset format; The metadata file is output to the editing software interface, enabling the editing software to call the corresponding transition effects for rendering.

[0129] In this embodiment, the structured motion effect parameter field consists of standardized, independent parameter items obtained by disassembling the transition motion effect text tags. Each item corresponds to a control instruction for transition rendering, used by the machine to identify and execute the rendering logic. The motion effect type identifier is an encoded mark corresponding to toggle, fade, wipe, and dissolve, used by the editing software to match the built-in transition rendering template.

[0130] In this embodiment, the duration parameter is the playback duration of the camera transition animation, which is dynamically adjusted by the emotional intensity value.

[0131] In this embodiment, the motion curve type is a parameter that controls the speed and rhythm of transition scenes, including four types: linear, ease in, ease out, and ease in ease out. The color shift direction is a parameter that controls the brightness and tone gradient shift of the scene during the transition process, adjusting the color transition tendency of the scene to match the emotional atmosphere.

[0132] In this embodiment, the metadata file is a file that encapsulates all structured motion effect parameters in a unified format, which can be directly read and parsed by professional editing software. The editing software interface is a data reading channel exposed by the editing software, used to receive externally imported transition control metadata.

[0133] As can be seen from the above, this embodiment automatically parses and converts motion effect text tags into structured motion effect parameters, adaptively adjusts transition duration based on emotional intensity, intelligently matches motion curves based on transition types, and outputs standardized metadata files that can be directly recognized by editing software. This solves the shortcomings of existing tags, which are merely text descriptions, cannot directly drive rendering, and require manual secondary parameter tuning. It automates the entire process from tag generation and parameter configuration to software rendering, reducing post-production manual debugging workload, improving the efficiency of industrial-scale animation editing, and ensuring a high degree of matching between motion effect parameters and shot emotions and transition types.

[0134] Corresponding to the method for automatically generating and matching motion effect text tags for comic book scene transitions in the above embodiment, Figure 2 This is a structural block diagram of an automatic generation and matching system for motion effect text tags in comic book scene transitions, provided as an embodiment of this application. For ease of explanation, only the parts relevant to the embodiment of this application are shown. References Figure 2 The animated storyboard transition animation text tag automatic generation and matching system 20 includes: a data acquisition module 21, a data processing module 22, a feature extraction module 23, a feature fusion module 24, a tag matching module 25, a weight determination module 26, and a tag generation module 27.

[0135] Among them, the data acquisition module 21 is used to acquire the raw data of the comic series, including storyboard video frame sequences, character dialogue audio, and synchronized subtitle text; Data processing module 22 is used to decompose and preprocess the storyboard video sequence frame by frame to generate a video frame sequence; to perform noise reduction processing on the character dialogue audio to generate noise-reduced audio data; and to perform text normalization processing on the synchronized subtitle text to generate normalized text data. The feature extraction module 23 is used to input video frame sequences into a preset visual emotion feature extraction network to extract the micro-expression temporal features and limb movement temporal features of the character; input noise-reduced audio data into a preset speech emotion recognition network to extract intonation prosody features and audio spectrum energy features; and input normalized text data into a preset text emotion analysis network to extract word emotion polarity features and sentence semantic emotion features. The feature fusion module 24 is used to input micro-expression temporal features, body movement temporal features, intonation prosody features, audio spectrum energy features, lexical sentiment polarity features and sentence semantic sentiment features into the cross-modal feature fusion network to generate the fused sentiment feature vector of the current scene. The tag matching module 25 is used to perform similarity matching in a preset emotional tag library based on the fused emotional feature vector to obtain an initial emotional tag that matches the current storyboard scene. The weight determination module 26 is used to obtain plot development information of the comic series, determine the plot weight coefficient of the current storyboard scene based on the plot development information, and use the plot weight coefficient to modulate the emotional intensity of the initial emotional label to generate an enhanced emotional label. The tag generation module 27 is used to obtain the transition type information between the current storyboard and the next storyboard, as well as the scene theme information of the preceding and following storyboards. It inputs the enhanced emotion tag, transition type information, and scene theme information into the preset motion effect text tag generation model to generate transition motion effect text tags.

[0136] See Figure 3 , Figure 3 This is a schematic block diagram of an electronic device provided according to an embodiment of this application. Figure 3 The electronic device 300 in this embodiment may include one or more processors 301, one or more input devices 302, one or more output devices 303, and one or more memories 304. The processors 301, input devices 302, output devices 303, and memories 304 communicate with each other via a communication bus 305. The memories 304 store computer programs, including program instructions. The processors 301 execute the program instructions stored in the memories 304. Specifically, the processors 301 are configured to invoke the program instructions to perform the functions of the modules in the aforementioned device embodiments, for example... Figure 2 The functions of the data acquisition module 21, data processing module 22, feature extraction module 23, feature fusion module 24, label matching module 25, weight determination module 26, and label generation module 27 are shown.

[0137] It should be understood that, in the embodiments of this application, the processor 301 may be a central processing unit (CPU), but it may also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor.

[0138] Input device 302 may include a touchpad, a fingerprint sensor (for collecting the user's fingerprint information and fingerprint orientation information), a microphone, etc., and output device 303 may include a display (LCD, etc.), a speaker, etc.

[0139] The memory 304 may include read-only memory and random access memory, and provides instructions and data to the processor 301. A portion of the memory 304 may also include non-volatile random access memory. For example, the memory 304 may also store device type information.

[0140] In specific implementations, the processor 301, input device 302, and output device 303 described in the embodiments of this application can execute the implementation methods described in any embodiment of the automatic generation and matching method for animation scene transition animation text tags provided in the embodiments of this application, or they can execute the implementation methods of the electronic devices described in the embodiments of this application, which will not be repeated here.

[0141] In another embodiment of this application, a computer-readable storage medium is provided. This computer-readable storage medium stores a computer program, which includes program instructions. When executed by a processor, the program instructions implement all or part of the processes in the methods described above. Alternatively, the computer program can instruct related hardware to complete the process. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include any entity or device capable of carrying computer program code, a recording medium, a USB flash drive, a portable hard drive, a magnetic disk, an optical disk, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium, etc.

[0142] The computer-readable storage medium can be an internal storage unit of the electronic device in any of the foregoing embodiments, such as a hard disk or memory of the electronic device. The computer-readable storage medium can also be an external storage device of the electronic device, such as a plug-in hard disk, smart media card (SMC), secure digital card (SD), flash card, etc., equipped on the electronic device. Furthermore, the computer-readable storage medium can include both internal and external storage units of the electronic device. The computer-readable storage medium is used to store computer programs and other programs and data required by the electronic device. The computer-readable storage medium can also be used to temporarily store data that has been output or will be output.

[0143] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this application.

[0144] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the electronic devices and units described above can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0145] In the several embodiments provided in this application, it should be understood that the disclosed electronic devices and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed may be indirect coupling or communication connection through some interfaces or units, or it may be an electrical, mechanical, or other form of connection.

[0146] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of the embodiments of this application, depending on actual needs.

[0147] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0148] The above are merely specific embodiments of this application, but the scope of protection of this application is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in this application, and these modifications or substitutions should all be covered within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method for automatically generating and matching motion effect text tags for comic book scene transitions, characterized in that, include: Obtain the raw data of the comic series, including storyboard video frame sequences, character dialogue audio, and synchronized subtitle text; The video sequence of the storyboard is decomposed and preprocessed frame by frame to generate a video frame sequence; the audio of the character dialogue is denoised to generate denoised audio data; and the synchronized subtitle text is normalized to generate normalized text data. The video frame sequence is input into a preset visual emotion feature extraction network to extract the micro-expression timing features and limb movement timing features of the character; the noise-reduced audio data is input into a preset speech emotion recognition network to extract intonation prosody features and audio spectrum energy features; the normalized text data is input into a preset text emotion analysis network to extract lexical emotion polarity features and sentence semantic emotion features. The micro-expression temporal features, body movement temporal features, intonation prosody features, audio spectrum energy features, lexical sentiment polarity features, and sentence semantic sentiment features are input into a cross-modal feature fusion network to generate a fused sentiment feature vector for the current scene. Based on the fused emotional feature vector, similarity matching is performed in a preset emotional tag library to obtain an initial emotional tag that matches the current storyboard scene. Obtain plot development information of the comic series, determine the plot weight coefficient of the current storyboard scene based on the plot development information, and use the plot weight coefficient to modulate the emotional intensity of the initial emotional tag to generate an enhanced emotional tag; Obtain the transition type information between the current storyboard and the next storyboard, as well as the scene theme information of the preceding and following storyboards. Input the enhanced emotion tag, the transition type information, and the scene theme information into a preset motion effect text tag generation model to generate transition motion effect text tags.

2. The method for automatically generating and matching motion effect text tags for comic book scene transitions according to claim 1, characterized in that, The step of inputting the video frame sequence into a preset visual emotion feature extraction network to extract the character's micro-expression temporal features and limb movement temporal features includes: For each frame in the video frame sequence, perform face detection and facial key point localization, and extract the coordinate sets of eye key points, eyebrow key points, and mouth key points. Calculate the coordinate offset of each key point between the current frame and the previous frame, arrange the coordinate offsets in chronological order, and generate micro-expression temporal features; Human pose estimation is performed on each frame in the video frame sequence to extract the two-dimensional coordinate sequence of key points of the bones in the shoulder, elbow, wrist, knee and ankle. Temporal difference analysis was performed on the two-dimensional coordinate sequence of each skeletal key point to extract the limb swing frequency feature and the trunk tilt angle change rate feature, thus forming the limb movement temporal feature.

3. The method for automatically generating and matching motion effect text tags for comic book scene transitions according to claim 1, characterized in that, The step of performing similarity matching in a preset emotion tag library based on the fused emotion feature vector to obtain an initial emotion tag matching the current scene includes: The preset emotion tag library is constructed as a three-dimensional tree structure, which includes a basic emotion layer, a theme projection layer, and an intensity modification layer. The basic emotion layer includes multiple basic emotion category nodes, the theme projection layer includes projected emotion nodes related to comic book themes, and the intensity modification layer includes degree modifier nodes. The tag encoder maps the tag text corresponding to each node in the preset emotion tag library to tag embedding vectors and stores them in the tag embedding vector index library. Calculate the cosine similarity between the fused sentiment feature vector and each tag embedding vector in the tag embedding vector index library, and select all tags with a cosine similarity greater than a preset similarity threshold to form a candidate tag set; The tag with the highest cosine similarity is selected from the candidate tag set as the initial sentiment tag.

4. The method for automatically generating and matching motion effect text tags for comic book scene transitions according to claim 1, characterized in that, The process of acquiring plot development information from the comic series and determining the plot weight coefficient of the current storyboard scene based on the plot development information includes: Obtain the global storyboard file of the comic series. The global storyboard file includes a pre-marked plot beat timeline. The plot beat timeline is set with multiple plot node type identifiers in chronological order. The plot node type identifiers include at least foreshadowing nodes, development nodes, climax nodes, and ending nodes. Extract the start and end timestamps of the current storyboard scene, map the start and end timestamps to the plot beat timeline, and determine the plot node type to which the current storyboard scene falls; Query the preset plot weight mapping table to obtain the plot weight coefficient corresponding to the plot node type; wherein, the plot weight coefficient of the climax node is set to be greater than the plot weight coefficients of the foreshadowing node and the development node.

5. The method for automatically generating and matching motion effect text tags for comic book scene transitions according to claim 4, characterized in that, Also includes: When the global storyboard file does not include a pre-annotated plot beat timeline, time-intensive sampling is performed on the video frame sequence to extract the image complexity features and image brightness change rate features of each sampled frame. The image complexity features and image brightness change rate features of each sampled frame are input into a pre-trained narrative rhythm classifier in chronological order. The narrative rhythm classifier outputs the predicted probability distribution of plot node types corresponding to each time window. Based on the predicted probability distribution of the plot node types, the plot node type with the highest predicted probability is selected as the plot node type to which the current storyboard scene belongs, and the corresponding plot weight coefficient is obtained by querying the preset plot weight mapping table.

6. The method for automatically generating and matching motion effect text tags for comic book scene transitions according to claim 1, characterized in that, The step of modulating the emotional intensity of the initial emotional tag using the plot weight coefficient to generate an enhanced emotional tag includes: The initial sentiment labels are extracted to obtain basic sentiment intensity values; The modulated emotional intensity value is obtained by multiplying the basic emotional intensity value by the plot weight coefficient. Based on the modulated emotional intensity value, a matching intensity modifier is selected from the intensity modification layer of the preset emotional tag library, and the intensity modifier is added to the preset position of the initial emotional tag to generate the enhanced emotional tag.

7. The method for automatically generating and matching motion effect text tags for comic book scene transitions according to claim 1, characterized in that, The process of obtaining the transition type information between the current shot and the next shot, as well as the scene theme information of the preceding and following shots, includes: Extract the last frame of the current storyboard and the first frame of the next storyboard; Calculate the pixel-by-pixel difference image between the last frame image and the first frame image, and count the proportion of pixels in the pixel-by-pixel difference image whose pixel values ​​are greater than a preset change threshold; When the pixel ratio is greater than or equal to a preset first ratio threshold, the transition type is determined to be a switch. Obtain the differential pixel state in the pixel-by-pixel differential image, wherein the differential pixel state includes a gradient diffusion distribution and a directional sweep distribution; When the pixel ratio is greater than or equal to a preset second ratio threshold and less than a preset first ratio threshold, and the differential pixel state exhibits a gradual diffusion distribution, the transition type is determined to be a gradient. When the pixel ratio is greater than or equal to a preset second ratio threshold and less than a preset first ratio threshold, and the differential pixel state is directional sweep distribution, the transition type is determined to be wipe image. When the pixel ratio is less than a preset third ratio threshold, the transition type is determined to be dissolve; Wherein the preset first ratio threshold is greater than the preset second ratio threshold, and the preset second ratio threshold is greater than the preset first ratio threshold; Obtain the keyframes of the current scene and the next scene, and input them into the pre-trained target detection network to obtain the category label and corresponding confidence score of each detected target; The category label corresponding to the detected target with the highest confidence score in the keyframe of the current scene is selected as the theme keyword of the previous scene, and the category label corresponding to the detected target with the highest confidence score in the keyframe of the next scene is selected as the theme keyword of the subsequent scene.

8. The method for automatically generating and matching motion effect text tags for comic book scene transitions according to claim 7, characterized in that, The step of inputting the enhanced emotion tag, the transition type information, and the scene theme information into a preset motion effect text tag generation model to generate transition motion effect text tags includes: Obtain a preset tag template string, which includes an emotion prefix slot, a scene description slot, and a transition action slot; Fill the enhanced emotion tag into the emotion prefix slot; The scene theme keywords of the first and second scenes are concatenated with preset connectors to generate a scene description string, which is then filled into the scene description slot. The corresponding transition action word is obtained by querying the preset transition-action mapping table based on the transition type information. According to the preset arrangement order of the emotion prefix slot, scene description slot and transition action slot in the preset tag template string, the filling content of each slot is concatenated into a complete transition animation text tag.

9. The method for automatically generating and matching motion effect text tags for comic book scene transitions according to claim 1, characterized in that, After generating the transition animation text tags, the method also includes: The transition animation text tags are parsed into structured animation parameter fields, which include at least the animation type identifier, duration parameter, motion curve type, and color offset direction. The duration parameter is determined based on the emotional intensity value of the enhanced emotional tag, wherein the larger the emotional intensity value, the shorter the duration parameter. Based on the transition type information, the motion curve type is determined: when the transition type is a switch or wipe, the motion curve type is set to linear or ease in and ease out; when the transition type is a fade or dissolve, the motion curve type is set to ease in and ease out. The structured motion effect parameter fields are encapsulated into a metadata file in a preset format; The metadata file is output to the editing software interface, enabling the editing software to call the corresponding transition effects for rendering.

10. A system for automatically generating and matching animated text tags for comic book scene transitions, characterized in that, include: The data acquisition module is used to acquire the raw data of the comic series, including storyboard video frame sequences, character dialogue audio, and synchronized subtitle text; The data processing module is used to perform frame-by-frame decomposition and preprocessing on the storyboard video sequence to generate a video frame sequence; to perform noise reduction processing on the character dialogue audio to generate noise-reduced audio data; and to perform text normalization processing on the synchronized subtitle text to generate normalized text data. The feature extraction module is used to input the video frame sequence into a preset visual emotion feature extraction network to extract the micro-expression temporal features and limb movement temporal features of the character; input the noise-reduced audio data into a preset speech emotion recognition network to extract intonation prosody features and audio spectrum energy features; and input the normalized text data into a preset text emotion analysis network to extract lexical emotion polarity features and sentence semantic emotion features. The feature fusion module is used to input the micro-expression temporal features, body movement temporal features, intonation prosody features, audio spectrum energy features, lexical sentiment polarity features and sentence semantic sentiment features into the cross-modal feature fusion network to generate the fused sentiment feature vector of the current scene. The tag matching module is used to perform similarity matching in a preset emotional tag library based on the fused emotional feature vector to obtain an initial emotional tag that matches the current storyboard scene. The weight determination module is used to obtain plot development information of the comic series, determine the plot weight coefficient of the current storyboard scene based on the plot development information, and use the plot weight coefficient to modulate the emotional intensity of the initial emotional tag to generate an enhanced emotional tag. The tag generation module is used to obtain the transition type information between the current storyboard and the next storyboard, as well as the scene theme information of the preceding and following storyboards. The enhanced emotion tag, the transition type information, and the scene theme information are input into a preset motion effect text tag generation model to generate transition motion effect text tags.

Images