A short play subtitle translation system based on artificial intelligence

Abstract

The application provides a short play subtitle translation system based on artificial intelligence, and relates to the technical field of artificial intelligence. The system comprises a subtitle recognition module, an erasing module, a translation module and an output module. The system can automatically recognize the time information, position coordinates and visual style parameters of the subtitles in the short play video, generate a picture sequence without original subtitles by combining the erasing process, and translate the recognized original subtitle text into a target language subtitle. The system generates target rendering parameters based on explicit style parameters and implicit style embedding vectors, realizes high restoration of font, stroke, shadow, gradient, transparency, texture and dynamic special effects, and performs adaptive adjustment according to the characteristics of the target subtitle text, so as to improve the visual consistency and cultural adaptability of the translated subtitles in the short play scene with complex subtitle styles, frequent dynamic changes and significant cross-cultural differences, improve the audience's viewing experience and reduce the workload of artificial post-processing.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, in particular to a short play subtitle translation system based on artificial intelligence. BACKGROUND

[0002] With the rapid popularization of short videos and streaming platforms, short plays have rapidly become popular due to their compact rhythm, concise plot, and high information density, and have become an important content type for cross-cultural communication. Compared with traditional films and television works, short plays often have more exaggerated and diversified subtitle designs in order to attract the attention of the audience in a short time, such as high-contrast outlines, shadow effects, gradient colors, semi-transparent processing, and even embedding subtitles directly into the surface of objects in the scene to create an immersive and visually striking effect. These expressive visual elements, while enhancing the viewing experience, have added challenges for subtitle detection and recognition.

[0003] In the prior art, subtitle translation usually involves the following process: first, extracting the original subtitle text through optical character recognition (OCR); second, generating target language subtitles using machine translation; and finally, performing style rendering and superimposition. However, traditional OCR algorithms often significantly reduce recognition accuracy when dealing with short play subtitles due to factors such as complex outlines of outlines and shadows, low contrast caused by gradient and transparency, and motion blur introduced by dynamic effects. When subtitles are highly integrated with background content or attached to the surface of scene objects in three-dimensional perspective form, the recognition difficulty is multiplied. This not only affects the accurate extraction of the original subtitle text, but also poses hidden dangers for subsequent translation and style restoration.

[0004] In addition, due to differences in character structure, character quantity, writing direction, and cultural expression habits between different languages, simply applying the translation result to the original subtitle style often leads to crowded subtitles, poor readability, and even a sense of discomfort with the picture. Misplacement or lack of cultural connotations can also weaken the audience's understanding and sense of immersion. Existing methods are difficult to ensure semantic accuracy while considering visual consistency and cross-cultural adaptability.

[0005] Therefore, how to improve the visual consistency and cultural adaptability of translated subtitles in the process of short play subtitle translation has become a technical problem that needs to be solved. SUMMARY

[0006] To overcome the shortcomings of the prior art, the present application provides a short play subtitle translation system based on artificial intelligence, which comprises:

[0007] a subtitle recognition module, configured to analyze an input short drama video, determine appearance time information of a subtitle and position coordinates of the subtitle in a picture, extract a subtitle region, and recognize original subtitle text; analyze a visual style of the subtitle region, and extract source style data representing the visual style;

[0008] an erasing module, configured to perform a filling process on the subtitle region to generate a picture frame sequence without original subtitles;

[0009] a translation module, configured to translate the original subtitle text to obtain target language subtitle data;

[0010] an output module, configured to generate target rendering parameters adapted to the target language subtitle data based on text features of the source style data and the target language subtitle data, perform style rendering on the target language subtitle data according to the target rendering parameters, and superimpose the target language subtitle after the style rendering on the picture frame sequence without original subtitles.

[0011] Optionally, the source style data representing the visual style includes:

[0012] explicit style parameters are obtained by using an image processing algorithm to analyze color, contour, and texture of the subtitle region, the explicit style parameters including foreground color parameters, outline parameters, soft special effect parameters, transparency parameters, and dynamic change parameters;

[0013] an implicit style embedding vector representing abstract aesthetic features of the subtitle region is generated by using a style encoding network to perform encoding processing on the subtitle region.

[0014] Optionally, the explicit style parameters include:

[0015] a layered component mask is generated by using a multi-class semantic segmentation network to segment the subtitle region, the layered component mask at least distinguishing a foreground filling region, a hard outline region, and a soft special effect region;

[0016] foreground filling color, outline color, and soft special effect color of the subtitle region are distinguished based on the layered component mask, and corresponding RGB values, pixel widths, and pixel offsets are calculated respectively to generate the foreground color parameters, the outline parameters, and the soft special effect parameters;

[0017] an alpha mask segmentation process is performed based on edge pixels of the subtitle region to separate and obtain the foreground color value and the transparency value, and the transparency parameters are generated;

[0018] Performing time series analysis on the foreground color parameter, the stroke parameter and the soft effect parameter on a video timeline corresponding to the appearance time information of the subtitle, and generating dynamic change parameters representing color flicker frequency, color gradient direction and rate, and periodic displacement amount of texture scrolling.

[0019] Optionally, the generating the foreground color parameter, the stroke parameter and the soft effect parameter comprises:

[0020] For the mask of the foreground filling area, performing classification of pure color filling, gradient filling or texture filling based on color distribution entropy and / or texture complexity analysis, and generating a texture descriptor vector through a pre-trained feature extraction network when it is texture filling;

[0021] For the masks of the hard stroke area and the soft effect area, calculating the primary color RGB value respectively, and performing gradient analysis and / or transparency decay curve fitting on the edge contour to generate stroke width and blur radius;

[0022] The layered component mask serves as a region limit for the foreground color parameter, the stroke parameter, the soft effect parameter and the transparency parameter;

[0023] The texture descriptor vector serves as a texture filling description parameter of the foreground color parameter; the primary color RGB value, the stroke width and the blur radius respectively serve as components of the foreground color parameter, the stroke parameter and the soft effect parameter, and as inputs of the time series analysis.

[0024] Optionally, the time series analysis comprises:

[0025] On a unified time reference aligned with the appearance time information, constructing a frame-by-frame sequence for the foreground color parameter, the stroke parameter and the soft effect parameter;

[0026] Performing time-frequency analysis on the frame-by-frame sequence to identify and parameterize the time point, intensity and duration of the transient flicker event of the subtitle brightness or chrominance, and generating color flicker frequency;

[0027] Within a preset time window, fitting and matching a color vector change sequence with a nonlinear interpolation function library including a fade curve to determine the best matching function type and its key frame parameters, and generating color gradient direction and rate;

[0028] When the foreground filling is determined to be texture filling, calculating an optical flow field between adjacent frames based on the foreground filling area and decomposing the optical flow field to obtain a composite motion parameter including translation, rotation and scaling components, and generating periodic displacement amount of texture scrolling;

[0029] The color flicker frequency, the color gradient direction and rate, and the periodic displacement amount of the texture scrolling are aggregated as the dynamic change parameter and output.

[0030] Optionally, the timing analysis further includes:

[0031] analyzing the synchronized audio data of the video on the time axis corresponding to the occurrence time information, extracting prosodic features of speech and / or beat transient points of music;

[0032] Based on the prosodic features and / or beat transient points, performing time axis alignment and synchronization relationship calibration on the dynamic change parameter;

[0033] Output the dynamic change parameter including the synchronization relationship.

[0034] Optionally, the generating of the implicit style embedding vector representing the abstract aesthetic features thereof includes:

[0035] extracting multi-scale feature maps within the spatial range aligned with the subtitle area, and performing spatial pooling on each scale feature to generate scale feature aggregation results;

[0036] calculating channel statistics based on the scale feature aggregation results to generate a first style statistical vector;

[0037] calculating inter-channel correlation statistics based on the scale feature aggregation results to generate a second style statistical vector; the second style statistical vector is composed of the vectorization results of the correlation statistics;

[0038] performing content decoupling processing on the scale feature aggregation results to generate a content-free feature vector; the content decoupling processing includes instance normalization, and low-pass filtering and / or multi-scale smoothing convolution in the frequency domain;

[0039] performing timing aggregation on the first style statistical vector, the second style statistical vector, and the content-free feature vector within the continuous time slice corresponding to the occurrence time information to generate an implicit style embedding vector;

[0040] The channel statistics include channel mean, channel variance, and channel skewness; the inter-channel correlation statistics include a correlation matrix and / or a Gram matrix.

[0041] Optionally, the timing aggregation includes:

[0042] calculating the first style statistical vector, the second style statistical vector, and the element-wise time average within the continuous time slice to generate a static style vector representing a static underlying style;

[0043] subtracting the static style vector from each time point in the continuous time slice to generate a residual vector sequence representing a dynamic superimposed style;

[0044] processing the residual vector sequence using a time series encoder based on a self-attention mechanism to calculate an attention weight for each time point in the residual vector sequence, the attention weight representing the dynamic significance of the time point;

[0045] performing weighted summation on the residual vector sequence based on the attention weight to generate a dynamic style vector representing a dynamic superimposed style; and,

[0046] concatenating or adding the static style vector and the dynamic style vector to generate a final implicit style embedding vector.

[0047] Optionally, generating a target rendering parameter adapted to the target language subtitle data based on the source style data and the text features of the target language subtitle data includes:

[0048] pre-rendering the target language subtitle data to obtain target text features representing the morphology of the target language subtitle data, the target text features including at least one of the number of characters, the width-height ratio of the rendering bounding box, and the pixel density;

[0049] inputting the explicit style parameter and the target text features into a parameter mapping model to calculate and generate an adjusted explicit style parameter adapted to the geometric morphology of the target language subtitle data;

[0050] inputting the implicit style embedding vector as a style condition and the target language subtitle data as a content condition into a generative style rendering network to generate a generative rendering parameter including font shaping and texture details; and,

[0051] fusing the adjusted explicit style parameter and the generative rendering parameter to form the target rendering parameter.

[0052] Optionally, performing style rendering on the target language subtitle data according to the target rendering parameter includes:

[0053] generating a foreground fill layer image of the target language subtitle data based on the font shaping and texture details included in the generative rendering parameter and in combination with the foreground color parameter in the adjusted explicit style parameter;

[0054] generating an outline layer image and a soft special effect layer image based on the stroke parameter and the soft special effect parameter in the adjusted explicit style parameter at the contour periphery or designated offset position of the foreground fill layer image; and,

[0055] The foreground filling layer image, the outline layer image and the soft special effect layer image are layer-composited in a preset vertical order to form a final style subtitle image.

[0056] Compared with the prior art, the present application automatically identifies and extracts the subtitle area in the short drama video, accurately obtains the appearance time, position and visual style parameter of the subtitle, generates a clean picture by combining the erasing processing, and translates the original subtitle text into a target language subtitle. The system generates target rendering parameters based on explicit style parameters and implicit style embedding vectors, realizes high restoration and dynamic adaptation of font, outline, shadow, gradient, transparency, texture and other effects, and can automatically adjust for different subtitle text length and format. Compared with the prior art, the present application can significantly improve the visual consistency and cultural adaptability of the translated subtitle under complex subtitle style, dynamic special effect and cross-cultural difference in the short drama, improve the viewing experience and reduce manual post-processing. BRIEF DESCRIPTION OF DRAWINGS

[0057] Figure 1 A flowchart of a short drama subtitle translation system based on artificial intelligence provided by the present application;

[0058] Figure 2 A flowchart of a method for extracting source style data representing the visual style provided by the present application;

[0059] Figure 3 A flowchart of a method for obtaining explicit style parameters provided by an embodiment of the present application;

[0060] Figure 4 A flowchart of a method for generating the foreground color parameter, outline parameter and soft special effect parameter provided by an embodiment of the present application. DETAILED DESCRIPTION

[0061] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, not all the embodiments of the present application.

[0062] Referring to Figure 1 The schematic diagram of a short drama subtitle translation system based on artificial intelligence provided by an embodiment of the present application is shown, which includes:

[0063] A subtitle recognition module is configured to analyze the input short drama video, determine the appearance time information of the subtitle and the position coordinates of the subtitle in the picture, extract the subtitle area, and recognize the original subtitle text. The visual style of the subtitle area is analyzed, and the source style data representing the visual style is extracted.

[0064] an erasing module configured to perform a filling process on the subtitle region to generate a picture frame sequence without original subtitles;

[0065] a translation module configured to translate the original subtitle text to obtain target language subtitle data;

[0066] an output module configured to generate target rendering parameters adapted to the target language subtitle data based on text features of the source style data and the target language subtitle data, perform style rendering on the target language subtitle data according to the target rendering parameters, and superimpose the target language subtitle after the style rendering onto the picture frame sequence without original subtitles.

[0067] In an optional embodiment, the subtitle recognition module can include a video frame analysis unit, a subtitle region detection unit, a text recognition unit, and a style analysis unit, which are connected through a data bus or a high-speed interface and perform a preset program by a processor to complete a processing flow.

[0068] First, the video frame analysis unit receives an input short drama video data stream, decodes it into a continuous picture frame sequence in chronological order, and synchronously analyzes audio and video time codes to establish a correspondence between frames and a time axis. The analyzed video frames can be cached in a lossless or low compression ratio format to prevent subsequent processing from introducing distortion.

[0069] The subtitle region detection unit identifies a region that may contain subtitles in each frame of the obtained picture based on a pre-trained subtitle detection model or a candidate region extraction method based on image processing. This process can combine traditional vision methods such as color threshold segmentation, edge detection, and connected component analysis to quickly exclude irrelevant regions; for suspected subtitle regions, a convolutional neural network can be further used for region classification confirmation. Through continuous frame comparison, the position coordinates of the subtitles in the picture and their appearance and disappearance time on the time axis can be determined to generate appearance time information of the subtitles.

[0070] The text recognition unit performs image enhancement processing on the positioned subtitle region, such as improving contrast, removing background noise, sharpening edges, etc., to improve the subsequent recognition accuracy. Subsequently, an optical character recognition (OCR) engine is used to convert the processed subtitle region image into text data. To adapt to the diversified fonts and decorative effects commonly seen in short dramas, the OCR engine can load a recognition sub-model trained for non-standard fonts and special characters to enhance the recognition robustness under interference factors such as outlines, shadows, and gradients. Finally, the original subtitle text is output and stored in association with the corresponding time information.

[0071] After identifying the subtitle region, the style analysis unit extracts features from the visual presentation of the region. This process can include sampling the color histogram of the subtitle region, extracting the contour shape, calculating the stroke width and color, determining whether there is a shadow and its offset direction and transparency, analyzing the filling method (solid color, gradient or texture), etc. If necessary, a feature encoding network can be used to convert the region image into a multi-dimensional feature vector to represent its overall visual style features. These source style data can be used as a reference for the subsequent rendering process to ensure that the translated subtitle maintains consistency with the original subtitle in visual effects.

[0072] In practical applications, the subtitle recognition module can adapt to the characteristics of subtitle diversification and dynamicization in short drama scenes, and can stably extract the time information, position coordinates, original text and visual style data of the subtitle even if the subtitle has complex special effects or is embedded on the surface of scene objects, thereby providing reliable input data for subsequent translation and rendering links.

[0073] In some embodiments, the feature encoding network can be constructed based on a convolutional neural network, a visual Transformer or a combination thereof. The input can be the aligned subtitle region image block, and the size of the image block can be, for example, 64x64 pixels, 128x32 pixels or other commonly used sizes in the art, in order to balance detail capture and computational efficiency. The image block can include three RGB channels, and an Alpha transparency channel can be added as a fourth channel when needed. The network output can be a one-dimensional or multi-dimensional vector, such as a floating-point number vector with a length of 128, 256 or 512, which is used to represent the overall style features of the subtitle region. Those skilled in the art can adjust the input size, channel number and output vector dimension according to different computing resources and task requirements, without changing the core idea of the present application.

[0074] In some embodiments, the OCR sub-model can be trained by supervised learning. The training data can come from public subtitle image datasets, manually annotated subtitle frames in short drama videos, or multi-style subtitle samples automatically generated by a synthesis engine. In order to ensure the generalization ability of the model, various style disturbances such as stroke thickness variation, color gradient, transparency variation, dynamic blur, etc. can be introduced in the training to simulate the complex visual characteristics of short drama subtitles. The loss function of the model can include character recognition accuracy loss and position coordinate regression loss, which are weighted and summed to optimize the comprehensive performance of the model in text recognition and positioning. The training process can be performed on a computing platform with GPU acceleration, and can be repeated to verify the convergence stability.

[0075] For the above erasing module:

[0076] In some embodiments, the erasing module includes a subtitle region filling unit and a background consistency repairing unit, both of which can be executed by the same processor or a dedicated image processing chip. The function of this module is to remove the located subtitle region from the picture frame without trace and generate a visually natural picture frame sequence without the original subtitle, providing a clean background for subsequent superimposed translation subtitles.

[0077] First, the subtitle region filling unit receives the subtitle position coordinate information and the corresponding frame image output from the subtitle recognition module, and marks the subtitle region with a mask. After the mask marking is completed, different initial filling methods can be selected according to the actual scene. For example, in some simpler scenes, the direct filling method using the mean or median value of the color of the pixels adjacent to the subtitle region can be used to quickly eliminate the subtitle content; in more complex scenes, the method based on texture synthesis can be used to copy and splice the surrounding background texture information to the subtitle region, thereby preliminarily restoring the picture continuity.

[0078] Next, the background consistency repairing unit performs more detailed repair processing on the preliminarily filled region to ensure that the generated picture is highly consistent with the original background in vision. This processing can use image repair algorithms, such as the sample filling method based on block matching or the repair model based on deep convolutional neural network.

[0079] In the implementation of the deep model, an encoder-decoder structure can be used, the input is the frame image containing the mask, and the output is the repaired complete frame. When training the repair model, short drama pictures without subtitles can be selected as samples, and virtual occlusion regions are randomly generated for learning to ensure that the model can restore the real background texture under different scenes and lighting conditions.

[0080] In some optional embodiments, to deal with special subtitle styles in short dramas, such as semi-transparent subtitles, gradient subtitles or subtitles embedded on the surface of scene objects, the erasing module can first separate the subtitle foreground from the background using the transparency estimation method to reduce the loss of background information.

[0081] For example, the transparency mask of the subtitle pixels can be predicted by analyzing the Alpha channel or using an image layer separation network, and more original background texture can be preserved during the background consistency repair stage.

[0082] Through the above steps, the erasing module not only removes the original subtitles in the short drama, but also maintains the structural integrity and natural details of the picture, avoiding the appearance of obvious repair traces, thereby providing high-quality background frames for subsequent subtitle rendering and superposition.

[0083] In some embodiments, the input and output data format of the erasing module can be explicitly as follows: the input includes the original video frame image with the subtitle region mask, the frame image can be stored in RGB or RGBA format, where the Alpha channel is used to identify the transparency of the subtitle pixels; the subtitle region mask can be a binary or grayscale image, which is used to distinguish the pixels that need to be filled in repair and the pixels that need to be kept. The output is the repaired complete frame image, which maintains the same resolution and color space as the input frame, so as to seamlessly replace the original frame.

[0084] In the implementation and training of the deep repair model, a publicly reproducible neural network structure can be used, such as U-Net or an image repair network based on attention mechanism. The training data can come from the scene frames without subtitles in a short drama video, and the effect of subtitle covering can be simulated by randomly generating virtual occlusion regions with irregular shapes. The shape, size and position of the occlusion region can be diversified to enhance the generalization ability of the model. The training target is to minimize the difference between the repaired frame and the original frame in terms of pixel value, texture feature and perceptual feature.

[0085] For example, the input size of the model, such as 256x256 pixels or 512x512 pixels, the number of channels (RGB or RGBA), the batch size, such as 8 or 16 frames per batch, and the optimizer parameters, such as learning rate, weight decay coefficient, etc. At the same time, the loss curve and the repair effect of the validation set can be recorded during the training process, and the model weight file can be exported after reaching the convergence standard.

[0086] For the above translation module:

[0087] In some embodiments, the translation module can be composed of a text preprocessing unit, a translation engine unit and a post-processing unit, and the processing results are transmitted through the memory data channel. After receiving the original subtitle text output from the subtitle recognition module, the text preprocessing unit will perform cleaning and standardization operations, such as removing line breaks, extra spaces, meaningless symbols and repeated characters caused by recognition errors in the subtitle, while keeping necessary punctuation marks to preserve the natural sentence breaks of the subtitle semantics.

[0088] In addition, in a short drama with multiple languages or mixed languages, the unit can also call a language detection model to automatically determine the source language category of each subtitle, so as to select the appropriate translation direction and model parameters for the translation engine.

[0089] The translation engine unit can be implemented based on the neural machine translation architecture, such as a model using an encoder-decoder network combined with an attention mechanism. The encoder maps the source language subtitle text to a semantic vector representation, and the decoder generates the target language subtitle text step by step based on the representation.

[0090] To improve the naturalness and colloquialism of the translation of short drama dialogues, the translation model can introduce a film and television subtitle corpus during training, and construct a training dataset by combining parallel corpora and alignment tools.

[0091] Exemplarily, the training data can come from publicly available film and television subtitle data, multilingual subtitle resources similar to the short drama type, and samples annotated by human translations. The data can be preprocessed, such as sentence splitting, annotation, denoising, and word form restoration, before use to reduce the impact of source language noise on translation accuracy.

[0092] The training process of the model can be executed in multiple rounds of iteration. The parameters are optimized by minimizing the difference between the predicted translation and the reference translation, and model checkpoints can be saved at different training rounds to verify convergence and repeatability.

[0093] The post-processing unit for translations formats and adapts the target language subtitles output by the translation engine, including adjusting the sentence breaking position, aligning the time codes, matching the subtitle display length limit, and replacing literal translations that do not conform to the target language usage. In some cases, this unit also calls a context consistency checking algorithm to conduct a contextual review of the translations of consecutive subtitles to ensure the unity of character appellations, key terms, and emotional tones. For special expressions, onomatopoeias, or culture-specific elements commonly found in short dramas, the post-processing unit can replace or annotate them based on a custom dictionary to ensure that the audience can understand the subtitle meaning without spoiling the viewing experience.

[0094] In a specific example, assume the input subtitle is the Chinese dialogue "你怎么来了?" (How come you are here?). The text preprocessing unit first removes the redundant symbols and identifies it as simplified Chinese. The translation engine selects a Chinese-to-English model based on this language category. The encoder encodes the Chinese sentence into a semantic vector, and the decoder generates the translation "How come you are here?". The post-processing unit determines that this sentence appears when friends meet according to the scene context, replaces "How come" with a more natural colloquial expression "What brings you here?", and aligns the translation with the time information of the original subtitle to generate the final target language subtitle data that can be rendered.

[0095] Such a design of the translation module not only ensures the accurate conveyance of the subtitle semantics but also retains the characteristics of the short drama in terms of language style, ensuring that the target language subtitles generated in different scenarios conform to the language usage and are also visually appealing.

[0096] In some embodiments, the output module obtains the source style data, target language subtitle data, appearance time information, and position coordinates. The output module can include a parameter generation unit, a style rendering unit, and an overlay synthesis unit, and each unit processes the data of the same subtitle in sequence.

[0097] The parameter generation unit extracts text features from the target language subtitle data, including the number of characters, the average word length, the punctuation density, the writing direction, the width-height ratio of the rendering bounding box, and the number of candidate line breaks. The parameter generation unit maps the text features to the source style data to obtain each sub-item of the target rendering parameters. Each sub-item includes font family and backup font, font size, font weight, kerning, line spacing, alignment, foreground color, transparency, outline width and color, shadow offset and blur radius, subtitle block margin and safe area constraints, and enabled time period on the time axis. The parameter generation unit adjusts the font size, line breaks, and kerning and line spacing of the subtitle when the width-height ratio or the number of characters exceeds the threshold according to the preset priority; when the writing direction of the target language is different from that of the source language, switch the layout direction and alignment; when the contrast between the foreground and background is lower than the threshold, adjust the foreground color, outline width, and shadow offset in turn until the contrast threshold is met. The parameter generation unit outputs complete target rendering parameters.

[0098] The style rendering unit receives the target rendering parameters and the target language subtitle data and generates a superimposable subtitle block image. The style rendering unit completes font shaping and line breaks based on the font family, font size, font weight, and writing direction to obtain glyph sequences and glyph displacements. The style rendering unit generates a foreground fill layer based on the foreground color and transparency, an outline layer based on the outline width and color, and a background effect layer based on the shadow offset and blur radius. The above layers are synthesized in a fixed overlay order to obtain the subtitle block image. During rendering, the style rendering unit uses the same color space and bit depth as the video frame and performs anti-aliasing processing on the glyph edges.

[0099] For parameters that need to be updated within the appearance time information, the style rendering unit reads the corresponding values in the target rendering parameters by frame and performs incremental redrawing; if a frame parameter has not changed, the last frame subtitle block is reused to reduce computational complexity. The style rendering unit outputs a sequence of subtitle block images consistent with the size of the video frame.

[0100] The superimposition and synthesis unit receives the sequence of subtitle block images and position coordinates and places the subtitle block at the specified position of the corresponding video frame. The position coordinates are defined in pixels relative to the video frame origin or the safe area origin. When the superimposition and synthesis unit detects that the subtitle block overlaps with a high-priority area in the picture, it fine-tunes the position according to the safe area and forbidden area rules, preferentially moving along the horizontal direction, then along the vertical direction, with an offset not exceeding a preset upper limit. The superimposition uses a premultiplied transparency blending method and maintains the same color management as the video frame. After superimposition, the superimposition and synthesis unit outputs a sequence of frames consistent with the input frame number, which contains the rendered and superimposed target language subtitles.

[0101] In one embodiment, the input is a subtitle within the time information of a landscape video segment, where the number of characters exceeds a single-line threshold. The parameter generation unit sets the line break number to two lines, reduces the font size and increases the line spacing, and selects a foreground color and outline color similar to the source style. After completing the font shaping, the style rendering unit generates three types of layers: foreground, outline, and background effects, and then composites them into a subtitle block image. The overlay compositing unit places the subtitles according to the bottom safe zone, and after detecting overlap with the watermark area, it performs a slight offset upwards, outputting a final frame sequence consistent with the video frame resolution.

[0102] Optional, see Figure 2 The flowchart of a method for extracting source style data characterizing the visual style provided in this application includes steps S101 to S102, wherein:

[0103] S101: Using an image processing algorithm, analyze the color, outline, and texture of the subtitle area to obtain explicit style parameters, including foreground color parameters, stroke parameters, soft effect parameters, transparency parameters, and dynamic change parameters.

[0104] S102: Using a style coding network, the subtitle region is encoded to generate an implicit style embedding vector that represents its abstract aesthetic features.

[0105] In an optional embodiment, to address the technical problem that conventional techniques lack a structured description system that can simultaneously consider specific, quantifiable features and abstract, overall aesthetics when analyzing subtitle styles, the subtitle recognition module provided in this application adopts a binary composite representation method for extracting source style data.

[0106] This implementation deconstructs the visual style of subtitles into two dimensions: one part consists of attributes that can be precisely calculated and have clear physical or geometric meaning; the other part consists of artistic features that can only be perceived holistically and abstractly modeled through deep learning models. By combining information from these two dimensions, a comprehensive and three-dimensional description of the subtitle style can be achieved, providing a complete and robust data foundation for subsequent high-fidelity style reconstruction.

[0107] Specifically, the process of extracting source style data includes the following two parallel processing paths:

[0108] First, obtain the explicit style parameters.

[0109] This approach aims to analyze and quantify specific, measurable visual attributes within the caption area. The explicit style parameters are a structured set of parameters, including foreground color parameters, stroke parameters, soft effects parameters, transparency parameters, and dynamic variation parameters.

[0110] In this stage of the implementation, the system performs a preliminary analysis of the color, outline, and texture of the subtitle area using image processing algorithms to identify and separate the corresponding values ​​or descriptors for each parameter. For example, the foreground color parameter can be used to record the dominant color value of the subtitle filling area, and the stroke parameter can be used to record the width and color of its outline. Subsequent implementations will further elaborate on the specific calculation process for generating these parameters.

[0111] Second, generate implicit style embedding vectors.

[0112] This approach aims to capture the overall, abstract aesthetic characteristics of subtitles. Technically, these abstract aesthetic characteristics refer to a high-dimensional mathematical representation that is insensitive to the content of the subtitle image—the specific text—but highly sensitive to visual presentation methods, such as the curvature and sharpness of the font design, the contrast and harmony of color combinations, and the integration of special effects elements.

[0113] To generate this vector, the system utilizes a pre-trained style coding network.

[0114] Specifically, the image data of the subtitle area, after preprocessing such as size normalization, is input into the style encoding network. This network can be built based on a deep convolutional neural network architecture, which gradually extracts higher-level, more abstract visual patterns from pixel-level details through its successive convolutional and pooling layers. The final output of the network is a fixed-length numerical vector, such as a 256-dimensional floating-point vector. The position and orientation of this vector in the feature space represent the abstract aesthetic style of the input subtitle. When training this style encoding network, supervised learning or self-supervised learning methods can be used. For example, it can be trained using a subtitle image dataset labeled with different style categories (such as "cartoon style," "tech style," etc.) to ensure that the network can learn a discriminative style representation.

[0115] Through the processing of the two paths mentioned above, the system finally outputs source style data that is rich in information, consisting of explicit style parameters and implicit style embedding vectors.

[0116] Optional, see Figure 3 The flowchart of a method for obtaining explicit style parameters provided in this application embodiment includes steps S201 to S204, wherein:

[0117] S201: The subtitle region is segmented using a multi-class semantic segmentation network to generate a layered component mask that at least distinguishes between the foreground fill region, the hard outline region, and the soft effect region;

[0118] S202: Based on the layered component mask, distinguish the foreground fill color, the stroke color and the soft effect color of the subtitle region, and calculate the corresponding RGB value, pixel width and pixel offset respectively, to generate the foreground color parameter, stroke parameter and soft effect parameter;

[0119] S203: Based on the edge pixels of the subtitle region, perform alpha mask segmentation processing to separate the foreground color value and the transparency value, and generate the transparency parameter;

[0120] S204: On the video timeline corresponding to the appearance time information of the subtitle, perform time series analysis on the foreground color parameter, stroke parameter and soft effect parameter to generate dynamic change parameters representing color flicker frequency, color gradient direction and rate, and texture scrolling periodic displacement amount.

[0121] In short drama subtitles, various visual elements such as foreground text, stroke, shadow or glow are closely coupled or even overlapped in space, making it difficult for traditional image analysis methods to separate them for independent and accurate parameter measurement. For example, at the edge of a text with a stroke, the color of the pixel is a mixture of the foreground and stroke colors, and direct measurement will produce serious bias.

[0122] The embodiment adopts a "segmentation first, then measurement" deconstruction analysis strategy. It does not directly measure on the original, mixed subtitle image, but first uses a deep learning model to "dissect" each visual component of the subtitle at the pixel level, separating it into independent regions, and then calculating parameters for each pure region. The beneficial effect is that it fundamentally avoids measurement interference caused by visual element overlap, so that the extraction of each style parameter is based on clear, unambiguous pixel regions, thereby improving the accuracy and reliability of the parameters.

[0123] Specifically, the process of obtaining explicit style parameters, in one embodiment, includes the following consecutive processing steps:

[0124] First step: generating a layered component mask based on multi-class semantic segmentation

[0125] The core of this step is to process the subtitle region image using a multi-class semantic segmentation network. This network is a specially trained deep learning model, and its architecture can be based on an encoder-decoder structure such as U-Net to achieve accurate pixel-level classification. In the training phase, a labeled dataset containing a large number of diverse style subtitles is needed, where each pixel is manually or semi-automatically labeled as a different class.

[0126] At runtime, the network receives a subtitle region image as input and outputs a set of layered component masks. This set of masks is a plurality of binary images, each of which corresponds to a visual component. For example, one mask identifies only the pixels belonging to the foreground fill region, another mask identifies only the pixels belonging to the hard stroke region, and a third mask identifies only the pixels belonging to the soft effect region. In this way, the originally mixed subtitle image is clearly decomposed into multiple independent layers.

[0127] Second step: Calculate static style parameters based on masks

[0128] After obtaining the layered component masks, the system can calculate various parameters based on these precise region definitions.

[0129] Generate foreground color parameters, stroke parameters, and soft effect parameters: The system overlays the layered component masks on the original subtitle region image as "templates".

[0130] To calculate the foreground color parameters, the system only extracts and analyzes the pixels identified by the foreground fill region mask, calculates the average color value or dominant color value of these pixels as RGB color values.

[0131] Similarly, the system extracts the pixels of the hard stroke region and the soft effect region respectively, and calculates their color values. At the same time, by using image morphology or distance transform, etc., the geometry of these regions is analyzed to calculate the pixel width and the pixel offset relative to the foreground fill region. These calculation results together constitute the foreground color parameters, stroke parameters, and soft effect parameters. For example, the pixel width can be the thickness of the stroke, and the pixel offset can be the displacement of the shadow.

[0132] To accurately calculate the transparency of the subtitle, the system performs an alpha matting process. This process first determines which pixels are absolute foreground, which are absolute background, and which are unknown regions on the edge of the two based on the layered component masks. Then, an alpha matting algorithm focuses on analyzing the pixels in the unknown regions to calculate the true foreground color value and the corresponding transparency value of each pixel. Finally, by statistically analyzing these transparency values, such as taking the average, the transparency parameter can be generated.

[0133] Third step: Generate dynamic change parameters through timing analysis

[0134] To capture the dynamic effects of the subtitle, the above-mentioned static parameter calculation process based on masks is repeated on multiple key frames on the video timeline corresponding to the appearance time information of the subtitle. This generates a sequence of values over time for each static parameter, such as foreground color, stroke width, etc.

[0135] Subsequently, a temporal analysis unit processes the parameter sequences to extract and quantify their dynamic patterns. For example, by analyzing the periodic fluctuation of the color intensity parameter sequence, the color flicker frequency can be determined; by analyzing the persistent directional movement of the color values in the color space, the color gradient direction and rate can be determined; by analyzing the positional translation of the subtitle region between consecutive frames, the periodic displacement amount of the texture scrolling can be determined. These quantified dynamic patterns are collected together to form the dynamic variation parameters.

[0136] Optionally, referring to Figure 4 A flowchart of a method for generating the foreground color parameter, the outline parameter and the soft effect parameter is provided for the embodiments of the present application, comprising steps S301-S304, wherein:

[0137] S301: For the mask of the foreground filling region, the foreground filling is classified as solid color filling, gradient filling or texture filling based on color distribution entropy and / or texture complexity analysis, and when it is texture filling, a texture descriptor vector is generated by a pre-trained feature extraction network;

[0138] S302: For the masks of the hard outline region and the soft effect region, the primary color RGB value is calculated respectively, and the edge profile is analyzed by gradient and / or the transparency decay curve is fitted to generate the outline width and the blur radius;

[0139] S303: The layered component mask serves as the region limitation of the foreground color parameter, the outline parameter, the soft effect parameter and the transparency parameter;

[0140] S304: The texture descriptor vector serves as the texture filling description parameter of the foreground color parameter; the primary color RGB value, the outline width and the blur radius serve as the components of the foreground color parameter, the outline parameter and the soft effect parameter respectively, and as the input of the temporal analysis.

[0141] Further, the visual style of the short drama subtitle is not only diverse, but also extremely rich in details. For example, the filling content of the subtitle can be a complex image or texture, and the edge effect can be a clear hard outline or a blurred soft glow. The preliminary parameter generation in the previous embodiment still needs more refined analysis means to accurately classify and quantify these complex visual details.

[0142] The embodiment introduces a classification-driven deep analysis strategy, that is, before parameter quantization, the type of visual component is intelligently classified first, and then according to the classification result, the most suitable deep analysis algorithm is called to extract the feature. Its beneficial effect is that the parameter extraction process is more targeted and adaptive, which can generate a parameter set with richer information dimension and more accurate description, thereby providing the possibility of reproducing highly complex texture filling and delicate edge effects.

[0143] Specifically, the deep generation process first analyzes the mask of the foreground filling area generated by the previous embodiment. The system first needs to judge the filling type of the area, for which one or more analysis methods can be used.

[0144] One method is color distribution entropy-based analysis. The system calculates the information entropy of all pixel colors in the mask. If the entropy value is very low, it is determined to be solid color filling. If the entropy value is relatively high and the color distribution shows regular changes, it is determined to be gradient filling. If the entropy value is very high and the color distribution has no obvious rules, it can be determined to be texture filling.

[0145] Another method is texture complexity-based analysis. The system can use Gabor filters or local binary pattern operators to process the area in the mask, and judge its complexity according to the response strength of the texture features, thereby classifying. When the filling is classified as texture filling, since its visual information cannot be described by simple color values, the system will call a pre-trained feature extraction network to generate a texture descriptor vector.

[0146] In a specific embodiment, the network can be a deep convolutional neural network pre-trained on a large image dataset, such as VGG-19. The system inputs the image block of the foreground filling area into the network and extracts the activation output of the intermediate convolutional layer. After processing, the numerical vector formed by these activation outputs is the texture descriptor vector as a mathematical "fingerprint".

[0147] While analyzing the foreground filling, the system also processes the masks of the hard edge region and the soft special effect region to calculate more detailed geometric and appearance parameters. The system calculates the dominant color of the pixels in these two mask regions as their respective primary color RGB values, and analyzes the edge profile to distinguish hard edges and soft effects.

[0148] For example, the system can calculate the image gradient extending outward from the edge of the filling area. The edge gradient curve of the hard edge will be very steep, while the edge gradient of the soft special effect will be relatively flat. By analyzing the width and slope of the gradient, the edge width and blur radius can be calculated.

[0149] Optionally, the system can also analyze the change of pixel transparency value from the edge of the fill region outward, and fit a preset function curve, such as a Gaussian function, to this transparency decay data, and the parameters of the function can also be used to determine the blur radius.

[0150] To ensure the data integrity and consistency of the subsequent processing flow, the embodiment finally performs structured assignment to the extracted values. The hierarchical component mask itself is designated as the region limit of the foreground color parameter, the stroke parameter, the soft effect parameter and the transparency parameter. The texture descriptor vector calculated in the foregoing steps is formally assigned as a special component of the foreground color parameter, i.e. the texture fill description parameter. Similarly, the calculated values of the main color RGB value, the stroke width and the blur radius are respectively assigned as the corresponding components of the foreground color parameter, the stroke parameter and the soft effect parameter. Finally, this complete parameter set which has been deeply analyzed and assigned in detail will be passed down as the input data of the next time sequence analysis.

[0151] Optionally, the time sequence analysis comprises:

[0152] Constructing a frame-by-frame sequence for the foreground color parameter, the stroke parameter and the soft effect parameter on a unified time reference aligned with the occurrence time information;

[0153] Performing time-frequency analysis on the frame-by-frame sequence to identify and parameterize the time point, intensity and duration of the transient flicker event of the subtitle brightness or chrominance, and generating a color flicker frequency;

[0154] Within a preset time window, fitting and matching the color vector change sequence with a nonlinear interpolation function library including a fade curve, determining the best matching function type and its key frame parameters, and generating a color gradient direction and rate;

[0155] When the foreground fill is determined to be a texture fill, calculating an optical flow field between adjacent frames based on the foreground fill region and decomposing the optical flow field to obtain a composite motion parameter including a translation, rotation and scaling component, and generating a periodic displacement amount of texture scrolling;

[0156] Collecting the color flicker frequency, the color gradient direction and rate, and the periodic displacement amount of texture scrolling into the dynamic change parameter and outputting.

[0157] Further, the dynamic effects of short film subtitles, such as flickering, gradient and texture scrolling, are often instantaneous, nonlinear and complex in motion pattern. The conventional analysis method based on simple interframe difference is difficult to accurately capture and quantify the artistic details of these dynamic effects, such as the animation rhythm of "fade in and fade out", sudden pulse flickering, or texture motion with rotation and scaling.

[0158] The embodiment does not use a single general algorithm to analyze all dynamic effects, but rather invokes the most suitable analysis model in each field for different types of dynamic changes (periodic / transient changes, continuous gradual changes, complex motion), and finally collects the results of each special analysis.

[0159] In this way, through fine classification and special processing of dynamic effects, a set of dynamic change parameters with richer dimensions and more accurate description can be extracted, thereby making it possible to faithfully reproduce the animation rhythm and visual rhythm of the original subtitle.

[0160] Specifically, the process of the timing analysis, in one embodiment, includes the following processing steps:

[0161] The timing analysis process starts with data preparation. The system first extracts or interpolates the values of the foreground color parameters, the stroke parameters, and the soft effect parameters in each frame on the basis of the appearance time information determined by the upper claim, on a unified time reference, to construct a plurality of parameter frame sequences available for analysis.

[0162] After obtaining the frame sequence, the system performs the following special analysis in parallel or series for different types of dynamic effects:

[0163] For flicker or pulse effects, the system performs time-frequency analysis on the frame sequence of the parameters, especially the brightness or chroma components. Conventional frequency domain analysis is difficult to capture very short duration effects, while time-frequency analysis, such as an optional wavelet transform implementation, can effectively identify and locate transient flicker events in the signal. This process can output the exact time points of these events, the intensity of the change, i.e. the intensity, and the duration, which are collectively used to generate the color flicker frequency parameter.

[0164] For smooth gradual change effects, the system analyzes the change sequence of the color vector within a preset time window. Considering that short animated films often use nonlinear changes with a "breathing" effect, the embodiment matches the color vector change sequence with a preset nonlinear interpolation function library. The function library contains mathematical representations of various industry-standard easing curves, such as ease-in, ease-out, or ease-in-out curves. The system determines the best matching function type by calculating the fitting error of the change sequence with each function curve in the library, and records the key frame parameters of the function, such as the starting color, ending color, change duration, etc., for generating the color gradient direction and rate parameters.

[0165] When the foreground fill is determined to be a texture fill, to capture its complex motion pattern, the system computes the optical flow field between adjacent frames of the foreground fill region. The optical flow field is a dense vector field that describes the motion direction and speed of each pixel or region in an image. After obtaining the optical flow field, the system decomposes it to resolve the dominant global motion.

[0166] In one specific embodiment, the decomposition process separates the optical flow field into a translation component representing the overall movement, a rotation component representing the rotation, and a scaling component representing the change in distance. These decomposed composite motion parameters are the periodic displacement amounts used to generate the texture scroll.

[0167] Finally, the parameters generated by each of the above-mentioned specialized analysis paths, such as the color flicker frequency, the color gradient direction and rate, and the periodic displacement amount of the texture scroll, are collected and organized by the system into a structured data record, which collectively constitutes the final dynamic change parameters and is output for use by subsequent modules.

[0168] Optionally, the timing analysis further includes:

[0169] analyzing the synchronized audio data of the video on the time axis corresponding to the occurrence time information, to extract prosodic features of speech and / or beat transient points of music;

[0170] performing time axis alignment and synchronization relationship calibration on the dynamic change parameters based on the prosodic features and / or beat transient points;

[0171] outputting the dynamic change parameters including the synchronization relationship.

[0172] Further, the dynamic change parameters extracted only through visual information may have a slight deviation in time stamp from audio events in the video that have strong correlation, such as speech accents and music beats, and the parameters themselves lack direct correlation information with the audio events, which constitutes a technical bottleneck in style reconstruction applications that require high-precision audio-visual synchronization.

[0173] The present embodiment establishes a parallel audio processing stream to analyze audio data synchronized with subtitles to extract acoustic event features with explicit time markers. Subsequently, through a time alignment and correlation step, these acoustic event features are calibrated with previously extracted visual dynamic parameters, thereby encoding cross-modal synchronization information into the final parameter set. In this way, by introducing audio as a time reference, the positioning accuracy of the dynamic change parameters on the time axis is improved, and the data dimension of the parameter set is enriched to include cross-modal correlation information that can be used to guide high-fidelity, rhythm-synchronized rendering.

[0174] In one specific embodiment, the multi-modal temporal analysis process comprises the following steps:

[0175] Firstly, the system separates and decodes the corresponding synchronized audio data segment from the video file according to the occurrence time information determined in the above claims. The data segment can be a digital audio signal in pulse code modulation format.

[0176] Secondly, the system analyzes the synchronized audio data to extract acoustic event features. The analysis process can contain two parallel processing paths:

[0177] For speech content, the system performs prosody feature extraction. In one embodiment, the fundamental frequency contour of the audio signal is calculated by employing a fundamental frequency detection algorithm, such as the YIN algorithm, and the energy contour is obtained by using a short-time energy calculation method. Subsequently, the accent or intonation break points in the speech are located by performing peak detection or slope discontinuity detection on the fundamental frequency contour and the energy contour, and the timestamps and feature values of these points are taken as prosody features.

[0178] For music content, the system performs beat transient point extraction. In one embodiment, this is achieved by performing a short-time Fourier transform (STFT) on the audio signal to obtain a spectrogram, and calculating the spectral flux based on the spectrogram. The local peak points in the spectral flux sequence correspond to the beats or instrument onset points in the music, and the timestamps of these beat transient points are determined by a peak picking algorithm.

[0179] Thirdly, the system performs time axis alignment and synchronization relationship calibration on the generated dynamic change parameters based on the extracted prosody features and beat transient points.

[0180] In one embodiment, the process places the timestamps of all visual dynamic events, such as the onset points of a single flash, and the timestamps of all acoustic events on a common high-precision time axis. The system employs a time proximity search algorithm to find the acoustic event that is closest in time to each visual dynamic event within a preset time window, for example, plus or minus 100 milliseconds. If found, it is determined that there is a synchronization relationship between the two, and an association identifier containing the timestamps and event types of both is generated.

[0181] Finally, the system expands the data structure of the dynamic change parameters according to the calibration results and integrates the determined synchronization relationships into it. For example, a new data field is added to a parameter entry describing a flash event to store the associated acoustic event type and precise timestamp. The final output is the dynamic change parameters that include the synchronization relationships and have enhanced information dimensions.

[0182] Optionally, the implicit style embedding vector representing the abstract aesthetic features thereof comprises:

[0183] Multi-scale feature maps are extracted within a spatial range aligned with the subtitle region, and spatial pooling is performed on each scale feature to generate a scale feature aggregation result.

[0184] Based on the convergence results of the scale features, channel statistics are calculated to generate a first style statistical vector;

[0185] Based on the convergence results of the scale features, the correlation statistics between channels are calculated to generate a second style statistical vector; the second style statistical vector is composed of the vectorized results of the correlation statistics.

[0186] The scale feature aggregation result is subjected to content decoupling processing to generate content-decoupled feature vectors; the content decoupling processing includes instance normalization and frequency domain low-pass filtering and / or multi-scale smoothing convolution.

[0187] Within the consecutive time slices corresponding to the occurrence time information, the first style statistical vector, the second style statistical vector, and the content-de-embedded feature vector are temporally aggregated to generate an implicit style embedding vector.

[0188] The channel statistics include channel mean, channel variance, and channel skewness; the inter-channel correlation statistics include correlation matrix and / or Gram matrix.

[0189] Furthermore, in an optional embodiment, to address the problem of how to generate an implicit style embedding vector that can robustly represent the abstract aesthetic features of subtitles while effectively decoupling from specific text content and video background, this application provides a preferred implementation method based on deep feature statistics.

[0190] This implementation draws upon and develops the core idea of ​​neural network style transfer: the style information of an image is primarily contained in the statistical distribution of its deep feature maps and the correlation between feature channels, rather than the spatial arrangement of the feature maps themselves. By extracting and combining these statistics that are insensitive to spatial content, a powerful style representation can be constructed.

[0191] In this way, a style fingerprint, or implicit style embedding vector, is generated that is rich in information dimensions, insensitive to content changes, and highly sensitive to style changes, providing a high-quality and highly stable style conditional input for subsequent style rendering in generative networks.

[0192] Specifically, the process of generating implicit style embedding vectors that represent their abstract aesthetic features includes the following steps in one embodiment:

[0193] The process begins with multi-level feature extraction on the subtitle region image. The system employs a pre-trained deep convolutional neural network, such as a VGG network, as a fixed feature extractor. The subtitle region image is input into the network, and outputs are extracted from multiple different depth convolutional layers of the network. These outputs are multi-scale feature maps, which represent the features of the image at different levels of abstraction, from low-level textures to high-level shapes. Subsequently, to eliminate the influence of spatial position information and focus on the style itself, the system performs a spatial pooling operation, such as global average pooling, on each multi-scale feature map. This operation compresses the two-dimensional spatial information of each feature map, generating a scale feature aggregation result that only retains the channel dimension information.

[0194] Based on the scale feature aggregation result, the system calculates three different types of feature vectors in parallel:

[0195] First, the system calculates a first style statistical vector. This process is achieved by calculating channel statistics of the scale feature aggregation result. In one embodiment, the channel statistics include the calculation of the channel mean, channel variance, and channel skew of all activation values in each feature channel.

[0196] Among them, the mean and variance can effectively capture the color distribution and basic texture information of the subtitle, while the skew can describe the asymmetry of its distribution. Concatenating these statistics of all channels together constitutes the first style statistical vector.

[0197] Second, the system calculates a second style statistical vector. This process is achieved by calculating inter-channel correlation statistics. In one embodiment, the statistics can be a correlation matrix or a Gram matrix. The calculation process of the matrix is to perform an inner product operation on the activation values of any two feature channels in the scale feature aggregation result, thereby obtaining a value that describes the synchronization of the response of the pair of channels. By traversing all channel pairs, a complete matrix can be constructed. This matrix captures which features, such as a certain color and a certain brush texture, tend to appear at the same time, and is a powerful tool for representing complex artistic styles. The system then vectorizes the two-dimensional matrix, i.e., unfolds it into a one-dimensional vector in a predetermined order, to form the second style statistical vector.

[0198] Third, the system generates a content-agnostic feature vector. To further eliminate the interference of specific literal glyphs (i.e., content) on the style representation, the system performs a content decoupling process on the scale feature aggregation results. In one embodiment, the process includes instance normalization, which normalizes each channel of each sample independently, effectively removing content-affected image contrast information, etc. The process can also include frequency domain low-pass filtering, i.e., transforming the feature data into the frequency domain and filtering out high-frequency components usually associated with content edges, or using a multi-scale smoothing convolution to blur the specific details of the content. The result obtained after these processes is a content-agnostic feature vector, which retains more style information related to the macroscopic morphology of the font.

[0199] Finally, since the subtitle style can dynamically evolve over time, the system performs a temporal aggregation operation on the sequence of the three types of vectors (the first style statistical vector, the second style statistical vector, and the content-agnostic feature vector) generated in parallel above within a continuous time slice defined by the appearance time information. This aggregation process fuses multiple vectors in the time dimension into a single implicit style embedding vector that can represent the core abstract aesthetic features of the entire subtitle segment.

[0200] Optionally, the temporal aggregation includes:

[0201] computing the first style statistical vector, the second style statistical vector, and the element-wise temporal average within the continuous time slice to generate a static style vector representing a static underlying style;

[0202] subtracting the static style vector from the vector at each time point within the continuous time slice to generate a residual vector sequence representing a dynamic superimposed style;

[0203] using a self-attention mechanism-based time series encoder to process the residual vector sequence, calculating an attention weight for each time point in the residual vector sequence, the attention weight representing the dynamic significance of the time point;

[0204] performing weighted summation on the residual vector sequence based on the attention weight to generate a dynamic style vector representing a dynamic superimposed style; and,

[0205] concatenating or adding the static style vector and the dynamic style vector to generate the final implicit style embedding vector.

[0206] Further, the visual style of a short episode usually presents a hierarchical structure in the time dimension, i.e. a relatively stable "base style" and one or more "dynamic effects" with very short duration superimposed. Conventional temporal aggregation methods, such as averaging or max-pooling over the whole sequence, are difficult to preserve both kinds of temporal features with different properties, and often lead to the transient dynamic effects being smoothed or diluted.

[0207] In this embodiment, a decoupling and weighting strategy is adopted. First, the time-series feature signal is explicitly decomposed into a time-invariant static component and a time-varying dynamic component by mathematical methods. Then, for the dynamic component, a self-attention mechanism is used to identify and amplify the time points with the largest amount of information and the most significant. In this way, this method can generate a structured implicit style embedding vector, different parts of which encode the static base style and the core dynamic effects of the subtitles, respectively, so that the overall time evolution characteristics of the subtitles can be more complete and accurate.

[0208] In one specific embodiment, the process of time-series aggregation includes the following steps:

[0209] The input of this process is the three vector sequences generated in the previous embodiment, i.e. the first style statistic vector sequence, the second style statistic vector sequence and the content-agnostic feature vector sequence, sampled within consecutive time slices. For ease of description, the three vectors at each time point can be concatenated or combined into a unified input vector.

[0210] First, the system calculates the element-wise time average of the input vector sequence. Specifically, for each dimension in the vector, the system calculates the arithmetic average of the numerical values at all time points. The average values of all dimensions are combined together to form a single vector with the same dimension as the original vector, which is defined as the static style vector representing the static base style.

[0211] Second, the system subtracts the static style vector calculated in the previous step from the input vector at each time point within the consecutive time slices. The purpose of this operation is to strip off the constant part of the signal, so as to obtain a residual vector sequence that only includes dynamic change information. Each vector in this sequence represents the deviation and direction of its corresponding time point relative to the average style.

[0212] Subsequently, the residual vector sequence is input to a time series encoder based on a self-attention mechanism for processing. In an embodiment, the encoder can be implemented using an encoder layer in a Transformer network. The core function of this mechanism is to calculate the correlation between the vector at each time point in the sequence and all other time points. Based on these correlations, the encoder calculates and outputs an attention weight for each time point in the residual vector sequence. The weight is a numerical value whose magnitude is proportional to the significance or uniqueness of the dynamic information contained in the time point. For example, a residual vector corresponding to a moment of dramatic flashing of the subtitles will be assigned a higher attention weight due to its large variation.

[0213] After obtaining the attention weight of each time point, the system performs a weighted sum on the residual vector sequence based on these weights. That is, each residual vector in the sequence is multiplied by its corresponding attention weight, and then all the weighted vectors are added to obtain a single aggregated vector. This vector effectively represents the most core dynamic superposition style in the entire time period because it focuses on dynamic information with high significance, and is defined as a dynamic style vector.

[0214] Finally, the system combines the static style vector and the dynamic style vector to generate the final implicit style embedding vector. In an embodiment, the combination can be concatenation, i.e., connecting the two vectors end to end to form a longer vector, or addition if the dimensions are the same. Through this step, the final implicit style embedding vector realizes the separate encoding of static and dynamic style information in structure.

[0215] Optionally, generating target rendering parameters adapted to the target language subtitle data based on the source style data and the text features of the target language subtitle data comprises:

[0216] Pre-rendering the target language subtitle data to obtain target text features representing the morphology of the target language subtitle data, the target text features including at least one of the number of characters, the aspect ratio of the rendering bounding box, and the pixel density;

[0217] Inputting the explicit style parameters and the target text features into a parameter mapping model to calculate and generate adjusted explicit style parameters adapted to the geometric morphology of the target language subtitle data;

[0218] Inputting the implicit style embedding vector as a style condition and the target language subtitle data as a content condition into a generative style rendering network to generate generative rendering parameters including font shaping and texture details; and

[0219] fusing the adjusted explicit style parameters with the generative rendering parameters to form the target rendering parameters.

[0220] To solve the visual inconsistency and style distortion problem caused by language morphology differences when applying the extracted source style data to target subtitles in different languages, the application adopts a hybrid strategy combining analytical mapping and generative rendering. For explicit style parameters that have clear physical or geometric meanings, a deterministic mapping model is used to make precise adjustments based on the morphological characteristics of the target text. For implicit style embedding vectors that represent the overall artistic sense, the system uses them as style conditions to drive a generative network to "recreate" the style.

[0221] This method takes into account the accuracy and artisticity of style transfer, ensuring reasonable adaptation of basic attributes such as stroke width and shadow offset, and creatively generating complex font forms and texture details that perfectly integrate with new text content, ultimately obtaining a set of customized target rendering parameters that are faithful to the original work.

[0222] In one specific embodiment, the process of generating target rendering parameters adapted to the target language subtitle data includes the following steps:

[0223] The process begins with a quantitative analysis of the morphology of the target language subtitle data itself. To achieve this, the system first performs a pre-rendering of the target language subtitle data in memory, i.e., using a standard font to render it onto an invisible canvas. Then, the system analyzes this pre-rendered image to calculate a set of target text features.

[0224] In one embodiment, the target text features can include: the number of characters directly counted; the aspect ratio of the rendering bounding box calculated from the pre-rendered image; and the pixel density calculated from the proportion of non-background pixels within the bounding box, which can be used to represent the visual weight or complexity of the text.

[0225] After obtaining the target text features, the system starts two parallel processing paths:

[0226] The first path handles explicit style parameters. The system takes the explicit style parameters obtained from upstream and the just-calculated target text features as inputs, and sends them to a parameter mapping model.

[0227] The model can be, in one embodiment, a light-weight multi-layer perceptron (MLP) neural network. For example, the multi-layer perceptron can include several fully connected layers, each layer configured with tens to hundreds of neurons, with the total number of model parameters in the order of tens to hundreds of thousands, and the model is trained with hundreds to thousands of iterations on a paired dataset with mean squared error loss function to optimize the weights. The model learns the internal rules of how to adjust the style parameters according to the differences between source and target text features through pre-training on a large amount of “source style-target style” paired data. At runtime, the model calculates and outputs a set of adjusted explicit style parameters, for example, a reduced stroke width value that is more suitable for the current long English text.

[0228] The second path processes the implicit style embedding vector. The system feeds the implicit style embedding vector into a generative style rendering network as a style condition to guide style generation. Meanwhile, the target language subtitle data, for example, represented as its character sequence or base glyph image, is also input into the network as a content condition.

[0229] The generative network, in one embodiment, can employ a generative adversarial network (GAN) or the decoder part of a variational autoencoder (VAE), which is designed to receive both content and style inputs and output low-level parameters that constitute complex visual effects. These output generative rendering parameters can include, for example, a set of Bezier curve control points for modifying the outline of a base glyph to achieve unique font shaping, or a procedural noise map to define texture details. As an example, the structure of the network can be based on a deep neural network containing multiple transpose convolutional layers and adaptive instance normalization layers, with a total number of parameters in the order of millions to tens of millions, and its training process can employ a target function that combines adversarial loss and perceptual loss to optimize on a large amount of unlabeled subtitle image corpus.

[0230] Finally, to form a complete set of rendering instructions for downstream rendering engines, the system fuses the outputs of the two paths. Specifically, the system integrates the adjusted explicit style parameters and generative rendering parameters, such as font shaping control points and texture data, into a unified data structure, which constitutes the final target rendering parameters.

[0231] Optionally, the style rendering of the target language subtitle data according to the target rendering parameters comprises:

[0232] Based on the font shaping and texture details included in the generative rendering parameters, and in combination with the foreground color parameter in the adjusted explicit style parameters, a foreground fill layer image of the target language subtitle data is generated;

[0233] generate a stroke layer image and a soft effect layer image respectively at the outer periphery of the foreground fill layer image or at a designated offset position based on the stroke parameter and the soft effect parameter in the adjusted explicit style parameters; and

[0234] perform layer composition on the foreground fill layer image, the stroke layer image and the soft effect layer image in a preset vertical order to form a final style subtitle image.

[0235] To convert the complex target rendering parameters generated in the previous embodiment into a final pixel image, the system adopts an implementation based on layered rendering and composition. This approach aims to solve the rendering challenges brought about by the mixed data structure of the target rendering parameters. The principle is to mimic the rendering pipeline of professional graphics software, treating different visual components of the subtitle as independent image layers for processing, and finally performing composition. This method can clearly and orderly process mixed-type rendering parameters, ensuring the correct occlusion and blending relationship between visual elements, thereby generating a high-quality style subtitle image without visual errors.

[0236] In this embodiment, the process of style rendering of target language subtitle data according to target rendering parameters is specifically as follows.

[0237] The rendering process begins with the generation of a foreground fill layer image. The system first initializes a digital canvas with the same resolution as the target video, with all pixels being completely transparent. To define the precise glyphs of the subtitle, the system first obtains the basic vector outline corresponding to the target language subtitle data from a standard glyph library, such as a TrueType or OpenType font file. This outline is composed of a series of Bezier curves. Then, the system reads the font shaping information included in the generative rendering parameters, which can be specifically a list of differential vectors or offsets applied to the control points of the Bezier curves. The system realizes geometric transformation of the glyphs by applying these offsets to the corresponding control points of the basic outline. After obtaining the final vector outline, the system uses it as a mask for the drawing area.

[0238] Next, the system fills the region inside the mask. If the foreground color parameter in the adjusted explicit style parameters is specified as a solid color or a gradient color, the corresponding color filling is directly performed. If the parameter includes a texture descriptor vector, it indicates that procedural texture generation is needed. In this case, the system can call a fragment shader running on the graphics processor. The texture details in the generative rendering parameters, such as a noise seed, multiple scale parameters, and a color lookup table, are passed as input uniform variables to the shader. The shader uses these inputs to calculate a unique color value for each pixel inside the mask, thus procedurally generating a complex texture effect. The output of this step is a bitmap image including the filled-in content of the main body with the background transparent, i.e., the foreground fill layer image.

[0239] Secondly, the system generates the accompanying special effect layer. This process is based on the stroke parameter and the soft effect parameter in the adjusted explicit style parameters. To generate the stroke layer image, the system extracts the Alpha Channel of the foreground fill layer image to obtain its precise contour. Subsequently, according to the pixel width defined in the stroke parameter, a circular structuring element is used to perform the dilation operation in image morphology on the contour to generate a smooth and equally wide expanded contour region, which is then filled in with the color defined in the stroke parameter. To generate the soft effect layer image, the system also bases on the foreground contour, performs displacement and color filling according to the pixel offset defined in the soft effect parameter, and then applies a two-dimensional Gaussian convolution. The standard deviation of the Gaussian kernel is precisely controlled by the blur radius component in the soft effect parameter, thus achieving a controllable blurring effect.

[0240] Finally, the system performs layer composition. The foreground fill layer image, the stroke layer image, and the soft effect layer image generated in the previous steps are superimposed in the preset vertical order.

[0241] In a specific embodiment, the composition order is that the soft effect layer is at the bottom layer, the stroke layer is in the middle, and the foreground fill layer is at the top layer. The superimposition between the layers uses the standard Alpha Blending algorithm.

[0242] Specifically, for each pixel, the final composition color value is the weighted average of the color values of the upper layer and the lower layer, and the weight is determined by the transparency value of the pixel in the upper layer. This composition process can be efficiently performed on the graphics processor, and the final output is a single complete style subtitle image including all visual styles and background transparent.

[0243] Those skilled in the art can realize that the units and algorithm steps of each example described in combination with the embodiments disclosed in the present application can be realized in electronic hardware or a combination of computer software and electronic hardware. Whether the functions are realized in hardware or software depends on specific applications and design constraints of the technical solutions. Those skilled in the art can use different methods to realize the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

Claims

1. A short drama subtitle translation system based on artificial intelligence, characterized in that, include: The subtitle recognition module is used to analyze the input short video, determine the appearance time information of the subtitles and the position coordinates of the subtitles in the screen, extract the subtitle area, and recognize the original subtitle text. The visual style of the subtitle area is analyzed, and source style data representing the visual style is extracted; The erasure module is used to perform filling processing in the subtitle area to generate a sequence of screen frames that do not contain the original subtitles; The translation module is used to translate the original subtitle text into target language subtitle data; The output module is used to generate target rendering parameters adapted to the target language subtitle data based on the text features of the source style data and the target language subtitle data; Based on the target rendering parameters, the target language subtitle data is style-rendered, and the style-rendered target language subtitles are superimposed onto the frame sequence without the original subtitles. The extraction of source style data characterizing the visual style includes: using image processing algorithms to analyze the color, outline, and texture of the subtitle area to obtain explicit style parameters, which include foreground color parameters, stroke parameters, soft effect parameters, transparency parameters, and dynamic change parameters. The subtitle region is encoded using a style coding network to generate an implicit style embedding vector that represents its abstract aesthetic features. The generation of target rendering parameters adapted to the target language subtitle data based on the text features of the source style data and the target language subtitle data includes: The target language subtitle data is pre-rendered to obtain target text features that characterize the form of the target language subtitle data. The target text features include at least one of the following: number of characters, aspect ratio of the rendering bounding box, and pixel density. The explicit style parameters and the target text features are input into the parameter mapping model to calculate and generate adjusted explicit style parameters that are adapted to the geometric shape of the target language subtitle data. The implicit style embedding vector is used as a style condition, and the target language subtitle data is used as a content condition. Both are input into a generative style rendering network to generate generative rendering parameters, including font shaping and texture details; and... The adjusted explicit style parameters are combined with the generative rendering parameters to form the target rendering parameters.

2. The short drama subtitle translation system based on artificial intelligence according to claim 1, characterized in that, The process of obtaining explicit style parameters includes: The subtitle region is segmented using a multi-class semantic segmentation network to generate a layered component mask that at least distinguishes between the foreground fill region, the hard outline region, and the soft effect region. Based on the layered component mask, the foreground fill color, stroke color, and soft effect color of the subtitle area are distinguished, and the corresponding RGB values, pixel width, and pixel offset are calculated respectively to generate the foreground color parameters, stroke parameters, and soft effect parameters. Based on the edge pixels of the subtitle area, perform α mask segmentation processing to separate and obtain the foreground color value and transparency value, and generate the transparency parameter; On the video timeline corresponding to the appearance time information of the subtitles, a time-series analysis is performed on the foreground color parameters, outline parameters, and soft effect parameters to generate dynamic change parameters representing the color flashing frequency, color gradient direction and rate, and texture scrolling periodic displacement.

3. The short drama subtitle translation system based on artificial intelligence according to claim 2, characterized in that, The generation of the foreground color parameters, stroke parameters, and soft effect parameters includes: For the mask of the foreground filling region, the foreground filling is classified as solid color filling, gradient filling or texture filling based on color distribution entropy and / or texture complexity analysis, and a texture descriptor vector is generated by a pre-trained feature extraction network when filling with texture. For the masks of the hard stroke area and the soft effect area, calculate the RGB values ​​of the main color respectively, and perform gradient analysis and / or transparency decay curve fitting on the edge contour to generate the stroke width and blur radius. The layered component mask serves as a region definition for foreground color parameters, stroke parameters, soft effect parameters, and transparency parameters; The texture descriptor vector serves as the texture fill description parameter for the foreground color parameter; the primary color RGB value, stroke width, and blur radius serve as components of the foreground color parameter, stroke parameter, and soft effect parameter, respectively, and are used as inputs for the temporal analysis.

4. The short drama subtitle translation system based on artificial intelligence according to claim 3, characterized in that, The time series analysis includes: On a unified time base aligned with the occurrence time information, a frame-by-frame sequence is constructed for the foreground color parameters, outline parameters, and soft effect parameters; Perform time-frequency analysis on the frame-by-frame sequence to identify and parameterize the time point, intensity, and duration of transient flicker events in subtitle brightness or chroma, and generate color flicker frequency; Within a preset time window, the color vector change sequence is fitted and matched with a nonlinear interpolation function library, including easing curves, to determine the best matching function type and its key frame parameters, and to generate the color gradient direction and rate. When the foreground fill is determined to be a texture fill, the optical flow field is calculated between adjacent frames based on the foreground fill region and the optical flow field is decomposed to obtain composite motion parameters including translation, rotation and scaling components, and to generate the periodic displacement of the texture scroll. The color flashing frequency, the color gradient direction and rate, and the periodic displacement of the texture scrolling are combined into the dynamic change parameters and output.

5. The short drama subtitle translation system based on artificial intelligence according to claim 4, characterized in that, The time series analysis also includes: Analyze the synchronized audio data of the video on the timeline corresponding to the occurrence time information to extract the prosodic features of the speech and / or the tempo transients of the music; Based on the rhythmic features and / or tempo transients, time axis alignment and synchronization relationship calibration are performed on the dynamic change parameters. The output includes the dynamic change parameters of the synchronization relationship.

6. The short drama subtitle translation system based on artificial intelligence according to claim 1, characterized in that, The implicit style embedding vectors that generate the abstract aesthetic features include: Multi-scale feature maps are extracted within a spatial range aligned with the subtitle region, and spatial pooling is performed on each scale feature to generate a scale feature aggregation result. Based on the convergence results of the scale features, channel statistics are calculated to generate a first style statistical vector; Based on the convergence results of the scale features, the correlation statistics between channels are calculated to generate a second style statistical vector; the second style statistical vector is composed of the vectorized results of the correlation statistics. The scale feature aggregation result is subjected to content decoupling processing to generate content-decoupled feature vectors; the content decoupling processing includes instance normalization and frequency domain low-pass filtering and / or multi-scale smoothing convolution. Within the consecutive time slices corresponding to the occurrence time information, the first style statistical vector, the second style statistical vector, and the content-de-embedded feature vector are temporally aggregated to generate an implicit style embedding vector. The channel statistics include channel mean, channel variance, and channel skewness; the inter-channel correlation statistics include correlation matrix and / or Gram matrix.

7. The short drama subtitle translation system based on artificial intelligence according to claim 6, characterized in that, The time-series aggregation includes: Calculate the first style statistical vector, the second style statistical vector, and the element-wise time average within the continuous time slice to generate a static style vector representing the static basic style; Subtract the static style vector from the vector at each time point within the continuous time slice to generate a residual vector sequence representing the dynamic overlay style. The residual vector sequence is processed by a time series encoder based on a self-attention mechanism, and an attention weight is calculated for each time point in the residual vector sequence. The attention weight represents the dynamic saliency of that time point. The residual vector sequence is weighted and summed based on the attention weights to generate a dynamic style vector representing the dynamic overlay style; and... The static style vector is concatenated or added to the dynamic style vector to generate the final implicit style embedding vector.

8. The short drama subtitle translation system based on artificial intelligence according to claim 1, characterized in that, The step of performing style rendering on the target language subtitle data according to the target rendering parameters includes: Based on the font shaping and texture details included in the generative rendering parameters, and combined with the foreground color parameter in the adjusted explicit style parameters, a foreground fill layer image of the target language subtitle data is generated. Based on the stroke parameters and soft effect parameters in the adjusted explicit style parameters, a stroke layer image and a soft effect layer image are generated respectively at the outer contour of the foreground fill layer image or at a specified offset position; and, The foreground fill layer image, the outline layer image, and the soft effects layer image are composited in a preset vertical order to form the final style caption image.

Images