Short drama subtitle translation system based on artificial intelligence

Abstract

The invention provides a short play subtitle translation system based on artificial intelligence, and relates to the technical field of artificial intelligence, and the system comprises a subtitle recognition module, an erasing module, a translation module and an output module, and can automatically recognize time information, position coordinates and visual style parameters of subtitles in a short play video. And generating a picture sequence without original subtitles in combination with the erasing processing, and translating the recognized original subtitle text into target language subtitles. The system generates target rendering parameters based on explicit style parameters and implicit style embedding vectors, realizes high restoration of fonts, strokes, shadows, gradient, transparency, textures and dynamic special effects, and performs adaptive adjustment according to target subtitle text features. Therefore, the visual consistency and culture adaptability of translated subtitles are improved in a short drama scene with complicated subtitle styles, frequent dynamic changes and obvious cross-culture differences, the audience impression is improved, and the manual post-processing workload is reduced.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, in particular to a short play subtitle translation system based on artificial intelligence. BACKGROUND

[0002] With the rapid popularization of short videos and streaming platforms, short plays have rapidly become popular due to their compact rhythm, concise plot, and high information density, and have become an important content type for cross-cultural communication. Compared with traditional films and television works, short plays often have more exaggerated and diversified subtitle designs in order to attract the attention of the audience in a short time, such as high-contrast outlines, shadow effects, gradient colors, semi-transparent processing, and even embedding subtitles directly into the surface of objects in the scene to create an immersive and visually striking effect. These expressive visual elements, while enhancing the viewing experience, have brought additional challenges to subtitle detection and recognition.

[0003] In the prior art, subtitle translation usually involves the following processes: first, extracting the original subtitle text through optical character recognition (OCR); second, generating target language subtitles using machine translation; and finally, performing style rendering and superimposition. However, traditional OCR algorithms often significantly reduce recognition accuracy when dealing with short play subtitles due to factors such as complex outlines of outlines and shadows, low contrast caused by gradient and transparency, and motion blur introduced by dynamic effects. When subtitles are highly integrated with background content or attached to the surface of scene objects in three-dimensional perspective form, the recognition difficulty is multiplied. This not only affects the accurate extraction of the original subtitle text, but also poses hidden dangers for subsequent translation and style restoration.

[0004] In addition, due to differences in character structure, character quantity, writing direction, and cultural expression habits between different languages, simply applying the translation result to the original subtitle style often leads to crowded subtitles, poor readability, and even a sense of discomfort with the picture. Misplacement or lack of cultural connotations can also weaken the audience's understanding and sense of immersion. Existing methods are difficult to ensure semantic accuracy while considering visual consistency and cross-cultural adaptability.

[0005] Therefore, how to improve the visual consistency and cultural adaptability of translated subtitles in the process of short play subtitle translation has become a technical problem that needs to be solved. SUMMARY

[0006] To overcome the shortcomings of the prior art, the present application provides a short play subtitle translation system based on artificial intelligence, comprising: a subtitle recognition module, configured to analyze an input short play video, determine the appearance time information of the subtitles and the position coordinates of the subtitles in the picture, extract the subtitle area, and recognize the original subtitle text; analyze the visual style of the subtitle area and extract source style data representing the visual style; an erasing module configured to perform a filling process on the subtitle region to generate a picture frame sequence without original subtitles; a translation module configured to translate the original subtitle text to obtain target language subtitle data; an output module configured to generate target rendering parameters adapted to the target language subtitle data based on text features of the source style data and the target language subtitle data, perform style rendering on the target language subtitle data according to the target rendering parameters, and superimpose the target language subtitle after the style rendering onto the picture frame sequence without original subtitles.

[0007] Optionally, the source style data representing the visual style comprises: explicit style parameters are obtained by analyzing colors, contours, and textures of the subtitle region using an image processing algorithm, the explicit style parameters including foreground color parameters, outline parameters, soft special effect parameters, transparency parameters, and dynamic change parameters; implicit style embedding vectors representing abstract aesthetic features of the subtitle region are generated by encoding the subtitle region using a style encoding network.

[0008] Optionally, the explicit style parameters are obtained by: a multi-class semantic segmentation network is used to segment the subtitle region to generate a hierarchical component mask distinguishing at least a foreground filling region, a hard outline region, and a soft special effect region; based on the hierarchical component mask, foreground filling colors, outline colors, and soft special effect colors of the subtitle region are distinguished, and corresponding RGB values, pixel widths, and pixel offsets are calculated respectively to generate the foreground color parameters, the outline parameters, and the soft special effect parameters; based on edge pixels of the subtitle region, an alpha mask segmentation process is performed to separate and obtain the foreground color values and the transparency values to generate the transparency parameters; on a video timeline corresponding to the appearance time information of the subtitle, a time sequence analysis is performed on the foreground color parameters, the outline parameters, and the soft special effect parameters to generate dynamic change parameters representing color flicker frequency, color gradient direction and rate, and texture scrolling periodic displacement amount.

[0009] Optionally, the foreground color parameters, the outline parameters, and the soft special effect parameters are generated by: for the mask of the foreground filling region, a classification of pure color filling, gradient filling, or texture filling is performed on the foreground filling based on color distribution entropy and / or texture complexity analysis, and when it is texture filling, a texture descriptor vector is generated by a pre-trained feature extraction network; The main color RGB value is calculated for the mask of the hard outline region and the soft special effect region respectively, and the gradient analysis and / or the transparency decay curve fitting are performed on the edge contour to generate the outline width and the blur radius; The layered component mask is used as the region definition of the foreground color parameter, the outline parameter, the soft special effect parameter and the transparency parameter. The texture descriptor vector is used as the texture filling description parameter of the foreground color parameter, and the main color RGB value, the outline width and the blur radius are used as the components of the foreground color parameter, the outline parameter and the soft special effect parameter respectively and as the input of the timing analysis.

[0010] Optionally, the timing analysis includes: A frame-by-frame sequence is constructed for the foreground color parameter, the outline parameter and the soft special effect parameter on a unified time reference aligned with the appearance time information. The time-frequency analysis is performed on the frame-by-frame sequence to identify and parameterize the time point, intensity and duration of the transient flicker event of the subtitle brightness or chrominance, and the color flicker frequency is generated. Within a preset time window, the color vector change sequence is fitted and matched with a nonlinear interpolation function library including a fade curve to determine the best matching function type and its key frame parameters, and the color gradient direction and rate are generated. When the foreground filling is determined to be texture filling, the optical flow field is calculated based on the foreground filling region between adjacent frames and the optical flow field is decomposed to obtain the composite motion parameters including translation, rotation and scaling components, and the periodic displacement amount of texture scrolling is generated. The color flicker frequency, the color gradient direction and rate, and the periodic displacement amount of texture scrolling are collected as the dynamic change parameters and output.

[0011] Optionally, the timing analysis further includes: The synchronous audio data of the video is analyzed on the time axis corresponding to the appearance time information to extract the prosody features of the voice and / or the beat transient points of the music. Based on the prosody features and / or the beat transient points, the timing axis alignment and the synchronization relationship calibration are performed on the dynamic change parameters. The dynamic change parameters including the synchronization relationship are output.

[0012] Optionally, the generation of the implicit style embedding vector representing the abstract aesthetic features thereof includes: The multi-scale feature maps are extracted within the spatial range aligned with the subtitle region, and the spatial pooling is performed on the features of each scale to generate the scale feature convergence result. The channel statistics are calculated based on the scale feature convergence result to generate the first style statistical vector. Calculate inter-channel correlation statistics based on the scale feature aggregation result, and generate a second style statistical vector; the second style statistical vector is composed of the vectorization result of the correlation statistics; Perform content decoupling processing on the scale feature aggregation result to generate a content-free feature vector; the content decoupling processing includes instance normalization, and low-pass filtering and / or multi-scale smoothing convolution in the frequency domain; Perform time series aggregation on the first style statistical vector, the second style statistical vector, and the content-free feature vector in the continuous time slice corresponding to the occurrence time information to generate an implicit style embedding vector; The channel statistics include channel mean, channel variance, and channel skewness; the inter-channel correlation statistics include a correlation matrix and / or a Gram matrix.

[0013] Optionally, the time series aggregation includes: Calculate the first style statistical vector, the second style statistical vector, and the element-wise time average in the continuous time slice to generate a static style vector representing a static base style; Subtract the static style vector from the vector of each time point in the continuous time slice to generate a residual vector sequence representing a dynamic superimposed style; Use a time series encoder based on a self-attention mechanism to process the residual vector sequence, and calculate an attention weight for each time point in the residual vector sequence, the attention weight representing the dynamic significance of the time point; Weighted sum the residual vector sequence based on the attention weight to generate a dynamic style vector representing a dynamic superimposed style; and, Concatenate or add the static style vector and the dynamic style vector to generate the final implicit style embedding vector.

[0014] Optionally, the generating target rendering parameters adapted to the target language subtitle data based on the text features of the source style data and the target language subtitle data includes: Pre-render the target language subtitle data to obtain target text features representing the morphology of the target language subtitle data, the target text features including at least one of character quantity, rendering aspect ratio, and pixel density; Input the explicit style parameters and the target text features into a parameter mapping model to calculate and generate adjusted explicit style parameters adapted to the geometric morphology of the target language subtitle data; inputting the implicit style embedding vector as a style condition and the target language subtitle data as content condition into a generative style rendering network to generate a generative rendering parameter including font shaping and texture details; and fusing the adjusted explicit style parameter and the generative rendering parameter to form the target rendering parameter.

[0015] Optionally, the style rendering of the target language subtitle data according to the target rendering parameter comprises: generating a foreground fill layer image of the target language subtitle data based on font shaping and texture details included in the generative rendering parameter and in combination with a foreground color parameter in the adjusted explicit style parameter; generating a stroke layer image and a soft special effect layer image based on a stroke parameter and a soft special effect parameter in the adjusted explicit style parameter at the contour periphery or the specified offset position of the foreground fill layer image, respectively; and performing layer composition on the foreground fill layer image, the stroke layer image and the soft special effect layer image in a preset vertical order to form a final style subtitle image.

[0016] Compared with the prior art, the present application automatically identifies and extracts a subtitle region in a short drama video, accurately obtains appearance time, position and visual style parameters of the subtitle, generates a clean picture in combination with erasing processing, and then translates original subtitle text into target language subtitles. The system generates a target rendering parameter based on an explicit style parameter and an implicit style embedding vector, realizes high restoration and dynamic adaptation of font, stroke, shadow, gradient, transparency, texture and other effects, and can automatically adjust to different subtitle text length and format. Compared with the prior art, the present application can significantly improve visual consistency and cultural adaptability of translated subtitles under complex subtitle style, dynamic special effects and cross-cultural differences in a short drama, improve the viewing experience and reduce manual post-processing. BRIEF DESCRIPTION OF DRAWINGS

[0017] Figure 1 a flowchart of a short drama subtitle translation system based on artificial intelligence provided by the present application; Figure 2 a flowchart of a method for extracting source style data representing the visual style provided by the present application; Figure 3 a flowchart of a method for obtaining an explicit style parameter provided by an embodiment of the present application; Figure 4 a flowchart of a method for generating the foreground color parameter, the stroke parameter and the soft special effect parameter provided by an embodiment of the present application. DETAILED DESCRIPTION

[0018] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, rather than all the embodiments of the present application.

[0019] Referring to Figure 1 As shown in FIG. 1, a schematic diagram of a short play subtitle translation system based on artificial intelligence provided by an embodiment of the present application includes: A subtitle recognition module is configured to analyze an input short play video, determine appearance time information of a subtitle and position coordinates of the subtitle in a picture, extract a subtitle region, and recognize original subtitle text; analyze a visual style of the subtitle region, and extract source style data representing the visual style. An erasing module is configured to perform filling processing on the subtitle region to generate a picture frame sequence without original subtitles. A translation module is configured to translate the original subtitle text to obtain target language subtitle data. An output module is configured to generate target rendering parameters adapted to the target language subtitle data based on text features of the source style data and the target language subtitle data; perform style rendering on the target language subtitle data according to the target rendering parameters, and superimpose the target language subtitle after the style rendering on the picture frame sequence without the original subtitles.

[0020] In an optional embodiment, the subtitle recognition module can include a video frame analysis unit, a subtitle region detection unit, a text recognition unit, and a style analysis unit, which are connected through a data bus or a high-speed interface and are executed by a processor to complete a processing flow according to a preset program.

[0021] First, the video frame analysis unit receives an input short play video data stream, decodes it into a continuous picture frame sequence in chronological order, and synchronously analyzes audio and video time codes to establish a correspondence between frames and a time axis. The analyzed video frames can be cached in a lossless or low compression ratio format to prevent subsequent processing from introducing distortion.

[0022] The subtitle region detection unit identifies a region that may contain a subtitle in each frame of the obtained picture based on a pre-trained subtitle detection model or a candidate region extraction method based on image processing. This process can combine traditional visual methods such as color threshold segmentation, edge detection, and connected component analysis to quickly exclude irrelevant regions; for suspected subtitle regions, a convolutional neural network can be further used for region classification confirmation. Through continuous frame comparison, the position coordinates of the subtitle in the picture and the appearance and disappearance time of the subtitle on the time axis can be determined, thereby generating the appearance time information of the subtitle.

[0023] The text recognition unit performs image enhancement processing on the located subtitle region, such as increasing contrast, removing background noise, sharpening edges, etc., to improve the subsequent recognition accuracy. Subsequently, the processed subtitle region image is converted into text data using an optical character recognition (OCR) engine. To adapt to the diversified fonts and decorative effects commonly seen in short dramas, the OCR engine can load a recognition sub-model trained for non-standard fonts and special characters to enhance the recognition robustness under interference factors such as outlines, shadows, gradients, etc. The final output is the original subtitle text, which is associated and stored with the corresponding time information.

[0024] The style analysis unit extracts features from the visual presentation of the subtitle region after identifying it. This process can include sampling the color histogram of the subtitle region, extracting the contour shape, calculating the outline width and color, determining whether there is a shadow and its offset direction and transparency, analyzing the filling method (solid color, gradient or texture), etc. If necessary, a feature encoding network can be used to convert the region image into a multi-dimensional feature vector to represent its overall visual style features. These source style data can be used as a reference for the subsequent rendering process to ensure that the translated subtitle maintains consistency with the original subtitle in visual effects.

[0025] In practical applications, this subtitle recognition module can adapt to the characteristics of diversified and dynamic subtitles in short dramas, and can stably extract the time information, position coordinates, original text and visual style data of the subtitles even if the subtitles have complex special effects or are embedded on the surface of scene objects, thereby providing reliable input data for subsequent translation and rendering.

[0026] In some embodiments, the feature encoding network can be constructed based on a convolutional neural network, a visual Transformer or a combination thereof. The input can be the aligned subtitle region image block, and the image block size can be, for example, 64x64 pixels, 128x32 pixels or other commonly used sizes in the art, in order to balance detail capture and computational efficiency. The image block can include three RGB channels, and an Alpha transparency channel can be added as a fourth channel when needed. The network output can be a one-dimensional or multi-dimensional vector, such as a floating-point number vector with a length of 128, 256 or 512, which is used to represent the overall style features of the subtitle region. Those skilled in the art can adjust the input size, channel number and output vector dimension according to different computing resources and task requirements without changing the core idea of the present application.

[0027] In some embodiments, the OCR sub-model can be trained in a supervised learning manner. The training data can come from public subtitle image datasets, manually annotated subtitle frames in short video clips, or multi-style subtitle samples automatically generated by a synthesis engine. To ensure the generalization ability of the model, various style perturbations such as stroke thickness variation, color gradient, transparency variation, dynamic blur, etc. can be introduced in the training to simulate the complex visual characteristics of short video subtitles. The loss function of the model can include character recognition accuracy loss and position coordinate regression loss, both of which are weighted to optimize the comprehensive performance of the model in text recognition and positioning. The training process can be performed on a computing platform with GPU acceleration and can be repeated to verify the convergence stability.

[0028] For the above-mentioned erasing module: In some embodiments, the erasing module includes a subtitle area filling unit and a background consistency repairing unit, both of which can be executed by the same processor or a dedicated image processing chip. The function of this module is to remove the located subtitle area from the picture without trace and generate a picture frame sequence that is visually natural and does not contain the original subtitle, providing a clean background for subsequent superimposed translation subtitles.

[0029] First, the subtitle area filling unit receives the subtitle position coordinate information and the corresponding frame image output from the subtitle recognition module, and marks the subtitle area with a mask. After the mask marking is completed, different initial filling methods can be selected according to the actual scene. For example, in some simpler scenes, the color mean or median of the pixels adjacent to the subtitle area can be used for direct filling to quickly eliminate the subtitle content; in more complex scenes, a method based on texture synthesis can be used to copy and splice the surrounding background texture information to the subtitle area, thereby preliminarily restoring the picture continuity.

[0030] Next, the background consistency repairing unit performs more detailed repair processing on the preliminarily filled area to ensure that the generated picture is highly consistent with the original background in terms of vision. This processing can use image repair algorithms, such as sample filling methods based on block matching or repair models based on deep convolutional neural networks.

[0031] In the implementation of the deep model, an encoder-decoder structure can be used, with the input being a frame image containing a mask and the output being a complete frame after repair. When training the repair model, short video pictures without subtitles can be selected as samples, and virtual occlusion areas can be randomly generated for learning to ensure that the model can restore the real background texture under different scenes and lighting conditions.

[0032] In some optional embodiments, to deal with special subtitle styles in short videos, such as semi-transparent subtitles, gradient subtitles, or subtitles embedded on the surface of scene objects, the erasing module can first use a transparency estimation method to separate the subtitle foreground from the background, reducing the loss of background information.

[0033] For example, the transparency mask of the subtitle pixels can be predicted by analyzing the Alpha channel or using an image layer separation network, and more original background texture can be preserved in the background consistency repair stage.

[0034] Through the above steps, the erasing module not only removes the original subtitles in the short drama, but also maintains the structural integrity and natural details of the picture, avoiding obvious repair marks, thereby providing high-quality background frames for subsequent subtitle rendering and superimposition.

[0035] In some embodiments, the input and output data formats of the erasing module can be explicitly as follows: the input includes original video frame images with subtitle region masks, which can be stored in RGB or RGBA format, where the Alpha channel is used to identify the transparency of the subtitle pixels; the subtitle region mask can be a binary or grayscale image, which is used to distinguish between pixels that need to be filled and pixels that need to be preserved. The output is the repaired complete frame image, which maintains the same resolution and color space as the input frame to seamlessly replace the original frame.

[0036] In terms of implementation and training of the deep repair model, a publicly reproducible neural network structure can be used, such as U-Net or an image repair network based on attention mechanism. The training data can come from scene frames without subtitles in the short drama, and the effect of subtitle masking can be simulated by randomly generating virtual occlusion regions of irregular shapes. The shape, size, and position of the occlusion region can be diversified to enhance the model's generalization ability. The training target is to minimize the difference between the repaired frame and the original frame in terms of pixel value, texture feature, and perceptual feature.

[0037] For example, the input size of the model, such as 256x256 pixels or 512x512 pixels, the number of channels (RGB or RGBA), the batch size, such as 8 or 16 frames per batch, and the optimizer parameters, such as learning rate, weight decay coefficient, etc. At the same time, the loss curve and the repair effect of the validation set can be recorded during the training process, and the model weight file can be exported after reaching the convergence standard.

[0038] For the above translation module: In some embodiments, the translation module can be composed of a text preprocessing unit, a translation engine unit, and a post-processing unit. The units pass the processing results through the memory data channel in sequence. After receiving the original subtitle text output from the subtitle recognition module, the text preprocessing unit will perform cleaning and standardization operations, such as removing line breaks, unnecessary spaces, meaningless symbols, and repeated characters caused by recognition errors, while preserving necessary punctuation marks to preserve the natural sentence breaks of the subtitle semantics.

[0039] In addition, in multilingual or mixed-language short plays, this unit can also call a language detection model to automatically determine the source language category of each subtitle segment, so as to select the appropriate translation direction and model parameters for the translation engine.

[0040] The translation engine unit can be implemented based on a neural machine translation architecture, such as a model using an encoder-decoder network combined with an attention mechanism. The encoder maps the source language subtitle text into a semantic vector representation, and the decoder then gradually generates the target language subtitle text based on this representation.

[0041] To improve the naturalness and colloquialism of the translation of short play dialogues, a movie and TV subtitle corpus can be introduced during the training of the translation model, and a training data set can be constructed by combining parallel corpora and alignment tools.

[0042] Exemplarily, the training data can come from publicly available movie and TV subtitle data, multilingual subtitle resources similar to the short play type, and samples annotated by human translation. The data can be preprocessed, such as sentence splitting, annotation, denoising, and word form restoration, before use to reduce the impact of source language noise on translation accuracy.

[0043] The training process of the model can be executed in multiple rounds of iteration. The parameters are optimized by minimizing the difference between the predicted translation and the reference translation, and model checkpoints can be saved at different training rounds to verify convergence and reproducibility.

[0044] The post-processing unit for the translation formats and adapts the target language subtitles output by the translation engine, including adjusting the sentence break position, aligning the time codes, matching the subtitle display length limit, and replacing literal translations that do not conform to the target language habits. In some cases, this unit will also call a context consistency check algorithm to conduct a context review of the translations of consecutive subtitles to ensure the unity of character appellations, key terms, and emotional tones. For special expressions, onomatopoeias, or culture-specific elements commonly found in short plays, the post-processing unit can perform replacements or annotations based on a custom dictionary to ensure that the audience can understand the subtitle meaning without spoiling the viewing experience.

[0045] In a specific example, assume the input subtitle is the Chinese dialogue "你怎么来了?" (How come you are here?). The text preprocessing unit first removes the redundant symbols and identifies it as simplified Chinese. The translation engine selects a Chinese-to-English model based on this language category. The encoder encodes the Chinese sentence into a semantic vector, and the decoder generates the translation "How come you are here?". The post-processing unit determines that this sentence appears when friends meet according to the scene context, replaces "How come" with a more natural colloquial expression "What brings you here?", and aligns the translation with the time information of the original subtitle to generate the final target language subtitle data that can be rendered.

[0046] The translation module design ensures accurate conveying of the subtitle semantics and preserves the language style of the short drama, ensuring that the target language subtitles generated in different scenes are in line with language habits and have an aesthetic appearance.

[0047] In some embodiments, the output module obtains source style data, target language subtitle data, appearance time information, and position coordinates. The output module can include a parameter generation unit, a style rendering unit, and a superimposed synthesis unit, each of which processes the data of the same subtitle in turn.

[0048] The parameter generation unit extracts text features from the target language subtitle data, including the number of characters, the average word length, the punctuation density, the writing direction, the rendering boundary box height-width ratio, and the number of candidate line breaks. The parameter generation unit maps the text features with the source style data to obtain each sub-item of the target rendering parameters. Each sub-item includes font family and backup font, font size, font weight, kerning, line spacing, alignment, foreground color, transparency, outline width and color, shadow offset and blur radius, subtitle block margin and safety zone constraints, and enabled time period on the time axis. The parameter generation unit adjusts the font size, line breaks, and kerning and line spacing of the subtitles that exceed the width-height ratio or character number threshold according to the preset priority; when the writing direction of the target language and the source language is different, switch the layout direction and alignment; when the foreground and background contrast is lower than the threshold, adjust the foreground color, outline width, and shadow offset in turn until the contrast threshold is met. The parameter generation unit outputs complete target rendering parameters.

[0049] The style rendering unit receives the target rendering parameters and the target language subtitle data and generates a superimposable subtitle block image. The style rendering unit completes font shaping and line breaks based on the font family, font size, font weight, and writing direction to obtain glyph sequences and glyph displacements. The style rendering unit generates a foreground fill layer based on the foreground color and transparency, a contour line layer based on the contour line width and color, and a background effect layer based on the shadow offset and blur radius. The above layers are synthesized in a fixed stacking order to obtain the subtitle block image. During rendering, the style rendering unit uses the same color space and bit depth as the video frame and performs anti-aliasing processing on the glyph edges.

[0050] For parameters that need to be updated within the appearance time information, the style rendering unit reads the corresponding values in the target rendering parameters by frame and performs incremental redrawing; if a frame parameter does not change, the last frame subtitle block is reused to reduce computational complexity. The style rendering unit outputs a sequence of subtitle block images consistent with the size of the video frame.

[0051] The superimposition synthesis unit receives the subtitle block image sequence and position coordinates, and places the subtitle block to the specified position of the corresponding video frame. The position coordinates are defined in pixels relative to the origin of the video frame or the origin of the safe area. When detecting that the subtitle block overlaps with a high-priority area in the picture, the superimposition synthesis unit fine-tunes the position according to the safe area and the forbidden area rules, and preferentially moves in the horizontal direction, and then moves in the vertical direction, with the offset not exceeding a preset upper limit. The superimposition adopts a premultiplied alpha blending mode, and maintains the same color management as the video frame. After the superimposition is completed, the superimposition synthesis unit outputs a frame sequence consistent with the input frame number, and the frame sequence contains the target language subtitles that have been rendered and superimposed.

[0052] In one embodiment, the input is a subtitle in a time-of-occurrence information of a landscape video, and the number of characters exceeds a single-line threshold. The parameter generation unit sets the number of line breaks to two lines, reduces the font size, and increases the line spacing, and selects a foreground color and an outline color similar to the source style. The style rendering unit generates three types of layers of foreground, outline, and background effects after completing font shaping, and synthesizes a subtitle block image. The superimposition synthesis unit places according to the bottom safe area, and detects overlap with the watermark area and performs a slight offset in the upper direction, and outputs a final frame sequence consistent with the resolution of the video frame.

[0053] Optionally, referring to Figure 2 A flowchart of a method for extracting source style data representing the visual style provided by the present application includes steps S101-S102, wherein: S101: An image processing algorithm is used to analyze the color, outline, and texture of the subtitle area, to obtain explicit style parameters, including foreground color parameters, outline parameters, soft special effect parameters, transparency parameters, and dynamic change parameters; S102: A style encoding network is used to encode the subtitle area to generate an implicit style embedding vector representing the abstract aesthetic features thereof.

[0054] In one optional embodiment, to solve the technical problem that conventional technologies lack a structured description system that can simultaneously consider specific, quantifiable features and abstract, overall aesthetics when analyzing the style of a subtitle, the extraction of source style data by the subtitle recognition module provided by the present application uses a binary composite representation method.

[0055] The present embodiment decomposes the visual style of a subtitle into two dimensions: one part is an attribute that can be obtained by precise calculation and has a clear physical or geometric meaning, and the other part is an artistic feature that can only be perceived and abstractly modeled by a deep learning model. In this way, by combining the information of the two dimensions, a comprehensive and three-dimensional description of the style of the subtitle can be achieved, providing a complete and robust data basis for subsequent high-fidelity style reconstruction.

[0056] Specifically, the process of extracting source style data includes the following two parallel processing paths: First, obtain explicit style parameters.

[0057] This path aims to analyze and quantify those specific, measurable visual attributes in the subtitle region. The explicit style parameters are a structured set of parameters that include foreground color parameters, outline parameters, soft effect parameters, transparency parameters, and dynamic change parameters.

[0058] In this stage of the embodiment, the system performs a preliminary analysis of the color, contour, and texture of the subtitle region through image processing algorithms to identify and separate the respective values or descriptors of these parameters. For example, the foreground color parameters can be used to record the dominant color value of the subtitle fill region, the outline parameters can be used to record the width and color of the contour line, and subsequent embodiments will further elaborate on the specific calculation process of generating these parameters.

[0059] Second, generate an implicit style embedding vector.

[0060] This path aims to capture the overall, abstract aesthetic features of the subtitle. The abstract aesthetic features referred to here are technically a high-dimensional mathematical representation that is insensitive to the specific text content of the subtitle image but highly sensitive to the visual presentation method, such as the curvature and sharpness of the font design, the contrast and harmony of color matching, and the integration method of special effect elements.

[0061] To generate this vector, the system uses a pre-trained style encoding network.

[0062] Specifically, after preprocessing such as size normalization, the image data of the subtitle region is input into the style encoding network. The network can be constructed based on a deep convolutional neural network architecture, and through its successive convolutional and pooling layers, it gradually extracts higher-level, more abstract visual patterns from pixel-level details. The final output of the network is a fixed-length numerical vector, such as a 256-dimensional floating-point vector. The position and direction of this vector in the feature space represent the abstract aesthetic style of the input subtitle. When training this style encoding network, supervised learning or self-supervised learning can be used, for example, using a dataset of subtitle image data labeled with different style categories (such as "cartoon style", "technology style", etc.) to ensure that the network can learn a style representation with discriminative power.

[0063] Through the processing of the above two paths, the system finally outputs source style data that is rich in information and composed of explicit style parameters and implicit style embedding vectors.

[0064] Optionally, see Figure 3A flowchart of a method for obtaining explicit style parameters according to an embodiment of the present application includes steps S201-S204, wherein: S201: segmenting the subtitle region using a multi-class semantic segmentation network to generate a hierarchical component mask that at least distinguishes a foreground fill region, a hard outline region, and a soft special effect region; S202: based on the hierarchical component mask, distinguishing foreground fill color, outline color, and soft special effect color of the subtitle region, and respectively calculating corresponding RGB value, pixel width, and pixel offset, to generate the foreground color parameter, outline parameter, and soft special effect parameter; S203: based on the edge pixels of the subtitle region, performing alpha mask segmentation processing to separate and obtain the foreground color value and transparency value, to generate the transparency parameter; S204: on the video timeline corresponding to the appearance time information of the subtitle, performing time series analysis on the foreground color parameter, outline parameter, and soft special effect parameter to generate dynamic change parameters representing color flicker frequency, color gradient direction and rate, and texture scrolling periodic displacement amount.

[0065] In short drama subtitles, multiple visual elements such as foreground text, outlines, shadows, or highlights are closely coupled or even overlapped in space, making it difficult for traditional image analysis methods to separate them for independent and accurate parameter measurement. For example, at the edge of a text with an outline, the color of the pixel is a mixture of the foreground and outline colors, and direct measurement will produce serious bias.

[0066] The present embodiment adopts a deconstructive analysis strategy of "segmentation first, then measurement". It does not directly measure on the original, mixed subtitle image, but first uses a deep learning model to "dissect" each visual component of the subtitle at the pixel level, separating it into independent regions, and then calculating parameters for each pure region. The beneficial effect is that it fundamentally avoids measurement interference caused by visual element overlap, so that the extraction of each style parameter is based on clear, unambiguous pixel regions, thereby improving the accuracy and reliability of the parameters.

[0067] Specifically, the process of obtaining explicit style parameters, in one embodiment, includes the following consecutive processing steps: First step: generating a hierarchical component mask based on multi-class semantic segmentation The core of this step is to process the subtitle region image with a multi-class semantic segmentation network. This network is a specially trained deep learning model, whose architecture can be based on an encoder-decoder structure such as U-Net to achieve accurate pixel-level classification. In the training phase, a labeled dataset containing a large number of diverse style subtitles is needed, where each pixel is manually or semi-automatically labeled as a different class.

[0068] At runtime, the network receives the subtitle region image as input and outputs a set of layered component masks. This set of masks is a plurality of binary images, each corresponding to a visual component. For example, one mask only identifies pixels belonging to the foreground fill region, another mask only identifies pixels belonging to the hard stroke region, and a third mask identifies pixels belonging to the soft effect region. In this way, the originally mixed subtitle image is clearly decomposed into multiple independent layers.

[0069] Second step: Calculate static style parameters based on masks After obtaining the layered component masks, the system can calculate various parameters based on these accurate region limits.

[0070] Generate foreground color parameters, stroke parameters, and soft effect parameters: The system overlays the layered component masks on the original subtitle region image as a "template".

[0071] To calculate the foreground color parameters, the system only extracts and analyzes the pixels identified by the foreground fill region mask, calculates the average color value or dominant color value of these pixels as the RGB color value.

[0072] Similarly, the system extracts the pixels of the hard stroke region and the soft effect region respectively, and calculates their color values. At the same time, through image morphology or distance transformation, etc. Method, analyze the geometric shape of these regions to calculate the pixel width and pixel offset relative to the foreground fill region. These calculation results together constitute the foreground color parameters, stroke parameters and soft effect parameters. Among them, the pixel width may be the thickness of the stroke, and the pixel offset may be the displacement of the shadow.

[0073] To accurately calculate the transparency of the subtitle, the system performs alpha matting processing. This process first determines which pixels are absolute foreground, which are absolute background, and which are unknown regions located at the edge of the two based on the layered component masks. Subsequently, an alpha matting algorithm focuses on analyzing the pixels in the unknown region to calculate the true foreground color value and the corresponding transparency value of each pixel. Finally, by statistically analyzing these transparency values, such as taking the average, the transparency parameter can be generated.

[0074] Step 3: Dynamic change parameter generation through time series analysis To capture the dynamic effects of the subtitles, the above-mentioned static parameter calculation process based on masks will be repeatedly executed on multiple key frames on the video timeline corresponding to the appearance time information of the subtitles. This will generate a sequence of values that change over time for each static parameter, such as foreground color, stroke width, etc.

[0075] Subsequently, a time series analysis unit will process these parameter sequences to extract and quantify their dynamic patterns. For example, by analyzing the periodic fluctuations of the color brightness parameter sequence, the color flicker frequency can be determined; by analyzing the persistent directional movement of color values in the color space, the color gradient direction and rate can be determined; by analyzing the positional translation of the subtitle area between consecutive frames, the periodic displacement amount of the texture scrolling can be determined. These quantified dynamic patterns are collected together to form the dynamic change parameter.

[0076] Optionally, referring to Figure 4 A flowchart of a method for generating the foreground color parameter, stroke parameter, and soft special effect parameter provided by the embodiments of the present application includes steps S301-S304, wherein: S301: For the mask of the foreground fill area, classify the foreground fill as pure color fill, gradient fill, or texture fill based on color distribution entropy and / or texture complexity analysis, and when it is texture fill, generate a texture descriptor vector through a pre-trained feature extraction network; S302: For the masks of the hard stroke area and the soft special effect area, respectively calculate the primary color RGB value, and perform gradient analysis and / or transparency decay curve fitting on the edge contour to generate the stroke width and blur radius; S303: The layered component mask serves as the area limitation for the foreground color parameter, stroke parameter, soft special effect parameter, and transparency parameter; S304: The texture descriptor vector serves as the texture fill description parameter of the foreground color parameter; the primary color RGB value, stroke width, and blur radius respectively serve as the components of the foreground color parameter, stroke parameter, and soft special effect parameter, and as the input of the time series analysis.

[0077] Furthermore, the visual style of short video subtitles is not only diverse, but also extremely rich in details. For example, the fill content of the subtitles can be complex images or textures, and the edge effect can be a clear hard stroke or a blurred soft glow. The preliminary parameter generation in the previous embodiment still needs more refined analysis methods to accurately classify and quantify these complex visual details.

[0078] The embodiment introduces a classification-driven deep analysis strategy, that is, before parameter quantization, the type of visual component is intelligently classified first, and then according to the classification result, the most suitable deep analysis algorithm is called to extract the feature. Its beneficial effect is that the parameter extraction process is more targeted and adaptive, which can generate a parameter set with richer information dimension and more accurate description, thereby providing the possibility of reproducing highly complex texture filling and delicate edge effects.

[0079] Specifically, the deep generation process first analyzes the mask of the foreground filling area generated by the previous embodiment. The system first needs to judge the filling type of the area, for which one or more analysis methods can be used.

[0080] One method is color distribution entropy-based analysis. The system calculates the information entropy of all pixel colors in the mask. If the entropy value is very low, it is determined to be solid color filling. If the entropy value is relatively high and the color distribution shows regular changes, it is determined to be gradient filling. If the entropy value is very high and the color distribution has no obvious rules, it can be determined to be texture filling.

[0081] Another method is texture complexity-based analysis. The system can use Gabor filters or local binary pattern operators to process the area in the mask, and judge its complexity according to the response strength of the texture features, thereby classifying. When the filling is classified as texture filling, since its visual information cannot be described by simple color values, the system will call a pre-trained feature extraction network to generate a texture descriptor vector.

[0082] In a specific embodiment, the network can be a deep convolutional neural network pre-trained on a large image dataset, such as VGG-19. The system inputs the image block of the foreground filling area into the network and extracts the activation output of the intermediate convolutional layer. After processing, the numerical vector formed by these activation outputs is the texture descriptor vector as a mathematical "fingerprint".

[0083] While analyzing the foreground filling, the system also processes the masks of the hard edge region and the soft special effect region to calculate more detailed geometric and appearance parameters. The system calculates the dominant color of the pixels in these two mask regions as their respective primary color RGB values, and analyzes the edge profile to distinguish between hard edge and soft special effect.

[0084] For example, the system can calculate the image gradient extending outward from the edge of the filling area. The edge gradient curve of the hard edge is very steep, while the edge gradient of the soft special effect is relatively flat. By analyzing the width and slope of the gradient, the edge width and blur radius can be calculated.

[0085] Optionally, the system can also analyze the change of pixel transparency value from the edge of the fill region outward, and fit a preset function curve, such as a Gaussian function, to this transparency decay data, and the parameters of the function can also be used to determine the blur radius.

[0086] To ensure the data integrity and consistency of the subsequent processing flow, the embodiment finally performs structured assignment to the extracted values. The hierarchical component mask itself is designated as the region limit of the foreground color parameter, the stroke parameter, the soft effect parameter and the transparency parameter. The texture descriptor vector calculated in the foregoing steps is formally assigned as a special component of the foreground color parameter, i.e. the texture fill description parameter. Similarly, the calculated values of the main color RGB value, the stroke width and the blur radius are respectively assigned as the corresponding components of the foreground color parameter, the stroke parameter and the soft effect parameter. Finally, this complete parameter set that has been deeply analyzed and assigned in detail will be passed down as the input data of the next time sequence analysis.

[0087] Optionally, the time sequence analysis includes: constructing a frame-by-frame sequence for the foreground color parameter, the stroke parameter and the soft effect parameter on a unified time reference aligned with the appearance time information; performing time-frequency analysis on the frame-by-frame sequence to identify and parameterize the time point, intensity and duration of the transient flicker event of the subtitle brightness or chrominance, and generating a color flicker frequency; within a preset time window, fitting and matching the color vector change sequence with a nonlinear interpolation function library including a fade curve, determining the best matching function type and its key frame parameters, and generating a color gradient direction and rate; when the foreground fill is determined to be a texture fill, calculating an optical flow field between adjacent frames based on the foreground fill region and decomposing the optical flow field to obtain a composite motion parameter including translation, rotation and scaling components, and generating a periodic displacement amount of texture scrolling; collecting the color flicker frequency, the color gradient direction and rate, and the periodic displacement amount of texture scrolling into the dynamic change parameter and outputting.

[0088] Further, the dynamic effects of short film subtitles, such as flickering, gradient and texture scrolling, are often instantaneous, nonlinear and complex in motion pattern. Conventional analysis methods based on simple interframe differences are difficult to accurately capture and quantify the artistic details of these dynamic effects, such as the "fade-in and fade-out" animation rhythm, sudden pulse-like flickering, or texture motion with rotation and scaling.

[0089] Instead of using a single general algorithm to analyze all dynamic effects, the present embodiment calls for the most suitable analysis model in each field to process different types of dynamic changes (periodic / transient changes, continuous gradual changes, complex motion), and finally collects the results of each special analysis.

[0090] In this way, through the fine classification and special processing of dynamic effects, a set of dynamic change parameters with richer dimensions and more accurate descriptions can be extracted, thereby making it possible to faithfully reproduce the animation rhythm and visual rhythm of the original subtitles.

[0091] Specifically, the process of the timing analysis includes the following processing steps in one embodiment: The timing analysis process begins with data preparation. The system first extracts or interpolates the values of the foreground color parameters, the stroke parameters, and the soft effect parameters in each frame on the basis of the appearance time information determined by the upper claim, on a unified time reference, to construct a plurality of parameter frame sequences available for analysis.

[0092] After obtaining the frame sequences, the system performs the following special analysis in parallel or series for different types of dynamic effects: For flicker or pulse effects, the system performs time-frequency analysis on the frame sequence of the parameters, especially the brightness or chroma components. Conventional frequency domain analysis is difficult to capture very short duration effects, while time-frequency analysis, such as an optional wavelet transform implementation, can effectively identify and locate transient flicker events in the signal. This process can output the exact time points of these events, the intensity of the change, i.e. the strength, and the duration, which are collectively used to generate the color flicker frequency parameter.

[0093] For smooth gradual change effects, the system analyzes the change sequence of the color vector within a preset time window. Considering that short animated films often use nonlinear changes with a "breathing" effect, the present embodiment matches the color vector change sequence with a preset nonlinear interpolation function library. The function library contains mathematical representations of various industry-standard easing curves, such as ease-in, ease-out, or ease-in-out curves. The system determines the best matching function type by calculating the fitting error of the change sequence with each function curve in the library, and records the key frame parameters of the function, such as the starting color, the ending color, the change duration, etc., which are used to generate the color gradient direction and rate parameters.

[0094] When the foreground fill is determined to be a texture fill, to capture its complex motion pattern, the system computes the optical flow field between adjacent frames of the foreground fill region. The optical flow field is a dense vector field that describes the direction and speed of motion of each pixel or region in an image. After obtaining the optical flow field, the system decomposes it to resolve the dominant global motion.

[0095] In one specific embodiment, the decomposition process separates the optical flow field into a translation component representing the overall movement, a rotation component representing the rotation, and a scaling component representing the change in distance. These decomposed motion parameters, i.e., the periodic displacement used to generate the texture scroll, are then used to generate the texture scroll.

[0096] Finally, the parameters generated by each of the above-mentioned specialized analysis paths, such as the color flicker frequency, the color gradient direction and rate, and the periodic displacement of the texture scroll, are collected and organized by the system into a structured data record, which collectively constitutes the final dynamic change parameters and is output for use by subsequent modules.

[0097] Optionally, the timing analysis further includes: analyzing the synchronized audio data of the video on the time axis corresponding to the occurrence time information, to extract the prosody features of the speech and / or the beat transient points of the music; performing time axis alignment and synchronization relationship calibration on the dynamic change parameters based on the prosody features and / or the beat transient points; outputting the dynamic change parameters including the synchronization relationship.

[0098] Further, the dynamic change parameters extracted only through visual information may have a slight deviation in time stamp from audio events in the video that have strong correlation, such as speech accents and music beats, and the parameters themselves lack direct correlation information with the audio events, which constitutes a technical bottleneck in style reconstruction applications that require high-precision audio-visual synchronization.

[0099] The present embodiment establishes a parallel audio processing flow to analyze the audio data synchronized with the subtitles to extract acoustic event features with clear time markers. Subsequently, through a time alignment and correlation step, these acoustic event features are calibrated with the previously extracted visual dynamic parameters, thereby encoding cross-modal synchronization information into the final parameter set. In this way, by introducing audio as a time reference, the positioning accuracy of the dynamic change parameters on the time axis is improved, and the data dimension of the parameter set is enriched, making it contain cross-modal correlation information that can be used to guide high-fidelity, rhythm-synchronized rendering.

[0100] In one specific embodiment, the multi-modal timing analysis process includes the following steps: First, the system separates and decodes the corresponding synchronized audio data segment from the video file according to the occurrence time information determined in the above claims. The data segment can be a digital audio signal in pulse code modulation format.

[0101] Second, the system analyzes the synchronized audio data to extract acoustic event features. The analysis process can include two parallel processing paths: For speech content, the system performs prosody feature extraction. In one embodiment, the fundamental frequency contour of the audio signal is calculated by employing a fundamental frequency detection algorithm, such as the YIN algorithm, and the energy contour is obtained by using a short-time energy calculation method. Subsequently, the accent or intonation break points in the speech are located by performing peak detection or slope discontinuity detection on the fundamental frequency contour and the energy contour, and the timestamps and feature values of these points are taken as prosody features.

[0102] For music content, the system performs beat transient point extraction. In one embodiment, this is achieved by performing a short-time Fourier transform (STFT) on the audio signal to obtain a spectrogram, and calculating the spectral flux based on the spectrogram. The local peak points in the spectral flux sequence correspond to the beats or instrument onset points in the music, and the timestamps of these beat transient points are determined by a peak picking algorithm.

[0103] Third, the system performs time axis alignment and synchronization relationship calibration for the generated dynamic change parameters based on the extracted prosody features and beat transient points.

[0104] In one embodiment, the process places the timestamps of all visual dynamic events, such as the start points of a single flash, and the timestamps of all acoustic events on a common high-precision time axis. The system employs a time proximity search algorithm to find the acoustic event that is closest in time to each visual dynamic event within a preset time window, for example, plus or minus 100 milliseconds. If found, it is determined that there is a synchronization relationship between the two, and an association identifier containing the timestamps and event types of both is generated.

[0105] Finally, the system expands the data structure of the dynamic change parameters according to the calibration results and integrates the determined synchronization relationships into it. For example, a new data field is added to a parameter entry describing a flash event to store the associated acoustic event type and precise timestamp. The final output is the dynamic change parameters that include the synchronization relationships and have enhanced information dimensions.

[0106] Optionally, the implicit style embedding vector representing the abstract aesthetic features thereof includes: extracting multi-scale feature maps within the spatial range aligned with the subtitle region, and performing spatial pooling on the scale features to generate scale feature convergence results; calculating channel statistics based on the scale feature aggregation result, and generating a first style statistical vector; calculating inter-channel correlation statistics based on the scale feature aggregation result, and generating a second style statistical vector; the second style statistical vector is composed of the vectorization result of the correlation statistics; performing content decoupling processing on the scale feature aggregation result, and generating a content-free feature vector; the content decoupling processing includes instance normalization, and low-pass filtering and / or multi-scale smoothing convolution in the frequency domain; performing time-series aggregation on the first style statistical vector, the second style statistical vector, and the content-free feature vector in a continuous time slice corresponding to the occurrence time information, and generating an implicit style embedding vector; The channel statistics include channel mean, channel variance, and channel skewness; and the inter-channel correlation statistics include a correlation matrix and / or a Gram matrix.

[0107] In addition, in an optional embodiment, to solve the problem of how to generate an implicit style embedding vector that can robustly represent the abstract aesthetic features of the subtitles and effectively decouple the specific text content and video background, the application provides a preferred implementation based on deep feature statistics.

[0108] This embodiment develops the core idea in the field of neural network style transfer, that is, the style information of an image is mainly contained in the statistical distribution of its deep feature map and the correlation between feature channels, rather than the spatial arrangement of the feature map itself. By extracting and combining these statistics that are not sensitive to spatial content, a powerful style representation can be constructed.

[0109] In this way, a style fingerprint, that is, an implicit style embedding vector, is generated, which is rich in information, not sensitive to content changes, and highly sensitive to style changes, providing a high-quality and stable style condition input for subsequent generative network style rendering.

[0110] Specifically, the process of generating an implicit style embedding vector representing the abstract aesthetic features thereof includes the following processing steps in an embodiment: The process begins with multi-level feature extraction on the subtitle region image. The system employs a pre-trained deep convolutional neural network, such as a VGG network, as a fixed feature extractor. The subtitle region image is input into the network, and outputs are extracted from multiple different depth convolutional layers of the network, which are multi-scale feature maps representing the image's features at different levels of abstraction from low-level textures to high-level shapes. Subsequently, to eliminate the influence of spatial position information and focus on the style itself, the system performs a spatial pooling operation, such as global average pooling, on each multi-scale feature map, which compresses the two-dimensional spatial information of each feature map to generate a scale feature aggregation result that only retains the channel dimension information.

[0111] Based on the scale feature aggregation result, the system calculates three different types of feature vectors in parallel: First, the system calculates a first style statistical vector. This process is achieved by calculating channel statistics of the scale feature aggregation result. In one embodiment, the channel statistics include the calculation of the channel mean, channel variance, and channel skew of all activation values in each feature channel.

[0112] Among them, the mean and variance can effectively capture the color distribution and basic texture information of the subtitle, and the skew can describe the asymmetry of its distribution. Concatenating these statistics of all channels together constitutes the first style statistical vector.

[0113] Second, the system calculates a second style statistical vector. This process is achieved by calculating inter-channel correlation statistics. In one embodiment, the statistics can be a correlation matrix or a Gram matrix. The calculation process of the matrix is to perform an inner product operation on the activation values of any two feature channels in the scale feature aggregation result, thereby obtaining a value describing the synchronization of the response of the pair of channels. By traversing all channel pairs, a complete matrix can be constructed. This matrix captures which features, such as a certain color and a certain brush texture, tend to appear at the same time, and is a powerful tool for representing complex artistic styles. The system then vectorizes the two-dimensional matrix, i.e., unfolds it into a one-dimensional vector in a predetermined order, to form the second style statistical vector.

[0114] Third, the system generates a content-agnostic feature vector. To further eliminate the interference of specific literal glyphs (i.e., content) on the style representation, the system performs a content decoupling process on the scale feature aggregation results. In one embodiment, the process includes instance normalization, which normalizes each channel of each sample independently, effectively removing content-affected image contrast information, etc. The process can also include frequency domain low-pass filtering, i.e., transforming the feature data into the frequency domain and filtering out high-frequency components that are usually associated with content edges, or using a multi-scale smoothing convolution to blur the specific details of the content. The resulting content-agnostic feature vector retains more style information related to the macroscopic morphology of the font.

[0115] Finally, since the subtitle style can dynamically evolve over time, the system performs a temporal aggregation operation on the sequence of the three types of vectors (the first style statistical vector, the second style statistical vector, and the content-agnostic feature vector) generated in parallel above within a continuous time slice defined by the appearance time information. This aggregation process fuses multiple vectors in the time dimension into a single implicit style embedding vector that can represent the core abstract aesthetic features of the entire subtitle segment.

[0116] Optionally, the temporal aggregation includes: computing the first style statistical vector, the second style statistical vector, and the element-wise temporal average within the continuous time slice to generate a static style vector representing a static underlying style; subtracting the static style vector from the vector at each time point within the continuous time slice to generate a sequence of residual vectors representing a dynamic superimposed style; processing the sequence of residual vectors using a time series encoder based on a self-attention mechanism to calculate an attention weight for each time point in the sequence of residual vectors, the attention weight representing the dynamic significance of the time point; performing weighted summation on the sequence of residual vectors based on the attention weights to generate a dynamic style vector representing the dynamic superimposed style; and, concatenating or adding the static style vector and the dynamic style vector to generate the final implicit style embedding vector.

[0117] Further, the visual style of a short drama subtitle usually presents a hierarchical structure in the time dimension, i.e., a relatively stable "underlying style" and one or more "dynamic effects" with very short durations superimposed. Conventional temporal aggregation methods, such as averaging or max-pooling over the entire sequence, are difficult to preserve both types of time features with different properties, often resulting in transient dynamic effect information being smoothed or diluted.

[0118] In this embodiment, a strategy combining decoupling and weighting is adopted. First, the time-series feature signal is explicitly decomposed into a time-invariant static component and a time-varying dynamic component by mathematical methods. Then, for the dynamic component, a self-attention mechanism is used to identify and amplify the time points with the largest amount of information and the most significant. In this way, the method can generate a structured implicit style embedding vector, different parts of which respectively encode the static basic style and the core dynamic effects of the subtitles, so as to more completely and accurately model the overall time evolution characteristics of the subtitles.

[0119] In a specific embodiment, the process of time-series aggregation includes the following steps: The input of this process is the three vector sequences sampled within the continuous time slice, i.e., the first style statistic vector sequence, the second style statistic vector sequence, and the content-agnostic feature vector sequence, generated in the previous embodiment. For ease of description, the three vectors at each time point can be spliced or combined into a unified input vector.

[0120] First, the system calculates the element-wise time average of the input vector sequence. Specifically, for each dimension in the vector, the system calculates the arithmetic average of the values of that dimension at all time points. The average values of all dimensions are combined together to form a single vector with the same dimensions as the original vector, which is defined as the static style vector representing the static basic style.

[0121] Second, the system subtracts the static style vector calculated in the previous step from the input vector at each time point within the continuous time slice. The purpose of this operation is to strip off the constant part of the signal, thereby obtaining a residual vector sequence that only includes dynamic change information. Each vector in this sequence represents the deviation and direction of its corresponding time point relative to the average style.

[0122] Subsequently, the residual vector sequence is input to a time series encoder based on a self-attention mechanism for processing. In an embodiment, the encoder can be implemented using an encoder layer in a Transformer network. The core function of this mechanism is to calculate the correlation between the vector at each time point in the sequence and all other time point vectors. Based on these correlation degrees, the encoder calculates and outputs an attention weight for each time point in the residual vector sequence. The weight is a numerical value whose magnitude is proportional to the significance or uniqueness of the dynamic information contained in the time point. For example, a residual vector corresponding to a moment of intense flashing of the subtitles will be assigned a higher attention weight due to its large change amplitude.

[0123] After obtaining the attention weights at each time point, the system performs a weighted sum of the sequence of residual vectors based on these weights. That is, each residual vector in the sequence is multiplied by its corresponding attention weight, and then all the weighted vectors are added to obtain a single aggregated vector. This vector, which effectively represents the most core dynamic overlay style in the entire time period by focusing on those dynamic information with high salience, is defined as the dynamic style vector.

[0124] Finally, the system combines the static style vector and the dynamic style vector to generate the final implicit style embedding vector. In one embodiment, the combination can be concatenation, i.e., connecting the two vectors end to end to form a longer vector; or, if the dimensions are the same, it can also be addition. Through this step, the final generated implicit style embedding vector realizes the separate encoding of static and dynamic style information in structure.

[0125] Optionally, the generating target rendering parameters adapted to the target language subtitle data based on the text features of the source style data and the target language subtitle data comprises: pre-rendering the target language subtitle data to obtain target text features representing the morphology of the target language subtitle data, the target text features comprising at least one of the number of characters, the width-height ratio of the rendering bounding box, and the pixel density; inputting the explicit style parameters and the target text features into a parameter mapping model to calculate and generate adjusted explicit style parameters adapted to the geometric morphology of the target language subtitle data; inputting the implicit style embedding vector as a style condition and the target language subtitle data as a content condition into a generative style rendering network to generate generative rendering parameters including font shaping and texture details; and fusing the adjusted explicit style parameters and the generative rendering parameters to form the target rendering parameters.

[0126] To solve the problem of visual incoordination and style distortion caused by language morphology differences when applying the extracted source style data to target subtitles in different languages, the present application adopts a hybrid strategy combining analytical mapping and generative rendering. For explicit style parameters that have explicit physical or geometric meanings, a deterministic mapping model is used to accurately adjust according to the morphological features of the target text. For implicit style embedding vectors representing the overall artistic sense, the system uses them as style conditions to drive a generative network to "recreate" the style.

[0127] The method takes into account both the accuracy and artistic of style transfer, which can not only ensure the reasonable adaptation of basic properties such as stroke width and shadow offset, but also creatively generate complex font forms and texture details that perfectly integrate with new text content, ultimately obtaining a set of customized and faithful-to-original-spirit target rendering parameters.

[0128] In one specific embodiment, the process of generating target rendering parameters adapted to the target language subtitle data includes the following steps: The process begins with a quantitative analysis of the morphology of the target language subtitle data itself. To achieve this purpose, the system first performs a pre-rendering of the target language subtitle data in memory, that is, using a standard font to render it onto an invisible canvas. Then, the system analyzes this pre-rendered image to calculate a set of target text features.

[0129] In one embodiment, the target text features can include: the number of characters directly counted; the rendering bounding box aspect ratio calculated from the bounding box of the pre-rendered image; and the pixel density calculated from the proportion of non-background pixels within the bounding box, which can be used to represent the visual weight or complexity of the text.

[0130] After obtaining the target text features, the system starts two parallel processing paths: The first path handles explicit style parameters. The system takes the explicit style parameters obtained from upstream and the target text features just calculated as inputs together, and sends them into a parameter mapping model.

[0131] In one embodiment, the model can be a lightweight multi-layer perceptron (MLP) neural network. For example, the multi-layer perceptron can include several fully connected layers, each layer is configured with tens to hundreds of neurons, the total number of model parameters is in the order of tens of thousands to hundreds of thousands, and the model is optimized by iterating several hundred to several thousand times on a paired data set using a mean square error loss function. The model learns how to adjust the internal rules of the style parameters according to the differences between the source and target text features by pre-training on a large number of "source style-target style" paired data. In runtime, the model calculates and outputs a set of adjusted explicit style parameters, for example, a reduced stroke width value that is more suitable for the current longer English text.

[0132] The second path handles the implicit style embedding vector. The system sends the implicit style embedding vector into a generative style rendering network as a style condition to guide style generation. At the same time, the target language subtitle data, for example, represented as its character sequence or basic glyph graph, is also input into the network as a content condition.

[0133] The generative network, in one embodiment, can employ a decoder part of a generative adversarial network (GAN) or a variational autoencoder (VAE), which is structured to receive both content and style inputs and output underlying parameters that constitute complex visual effects. The output generative rendering parameters can include, for example, a set of Bezier curve control points for modifying the outline of a base glyph to achieve unique font shaping, or a procedural noise map to define texture details. As an example, the network can be structured based on a deep neural network comprising multiple transpose convolutional layers and adaptive instance normalization layers, with a total parameter size in the order of millions to tens of millions, and its training process can employ an objective function combining adversarial loss and perceptual loss, optimized on a large corpus of unannotated subtitle images.

[0134] Finally, to form a complete set of rendering instructions for downstream rendering engines, the system fuses the outputs of the two paths described above. Specifically, the system integrates the adjusted explicit style parameters and the generative rendering parameters, such as font shaping control points and texture data, into a unified data structure, which constitutes the final target rendering parameters.

[0135] Optionally, the rendering of the target language subtitle data in the target style according to the target rendering parameters comprises: generating a foreground fill layer image of the target language subtitle data based on the font shaping and texture details included in the generative rendering parameters, in combination with the foreground color parameter in the adjusted explicit style parameters; generating an outline layer image and a soft effect layer image based on the outline parameter and the soft effect parameter in the adjusted explicit style parameters, at the outline periphery or a designated offset position of the foreground fill layer image, respectively; and performing layer composition on the foreground fill layer image, the outline layer image, and the soft effect layer image in a preset vertical order to form a final style subtitle image.

[0136] To convert the complex target rendering parameters generated in the previous embodiment into a final pixel image, the system employs an implementation based on layered rendering and composition. This approach aims to address the rendering challenges posed by the mixed data structure of the target rendering parameters. The principle is to mimic the rendering pipeline of professional graphics software, treating different visual components of the subtitle as independent image layers for processing, and finally compositing them. This method can clearly and orderly handle mixed types of rendering parameters, ensuring the correct occlusion and blending relationship between visual elements, thus generating a visually error-free, high-quality style subtitle image.

[0137] In this embodiment, the process of style rendering of target language subtitle data according to target rendering parameters is specifically as follows.

[0138] The rendering process starts with the generation of the foreground fill layer image. The system first initializes a digital canvas with the same resolution as the target video, with all pixels being completely transparent. To define the precise glyph of the subtitle, the system first obtains the base vector outline corresponding to the target language subtitle data from a standard glyph library, such as a TrueType or OpenType font file. The outline is composed of a series of Bezier curves. Then, the system reads the font shaping information included in the generative rendering parameters, which can be specifically a list of differential vectors or offsets applied to the control points of the Bezier curves. The system realizes the geometric transformation of the glyph by applying these offsets to the corresponding control points of the base outline. After obtaining the final vector outline, the system uses it as a mask for the drawing area.

[0139] Next, the system fills the area within the mask. If the foreground color parameter in the adjusted explicit style parameters is specified as a solid color or a gradient color, the corresponding color filling is directly performed. If the parameter includes a texture descriptor vector, it indicates that programmatic texture generation is needed. At this time, the system can call a fragment shader running on a graphics processor. The texture details in the generative rendering parameters, such as a noise seed, a plurality of scale parameters, and a color lookup table, will be passed as input uniform variables to the shader. The shader uses these inputs to calculate a unique color value for each pixel within the mask, thereby programmatically generating complex texture effects. The output of this step is a bitmap image that includes the main body fill content of the subtitle and has a transparent background, i.e., the foreground fill layer image.

[0140] Secondly, the system generates the accompanying special effect layer. This process is based on the stroke parameter and the soft special effect parameter in the adjusted explicit style parameters. To generate the stroke layer image, the system extracts the Alpha Channel of the foreground fill layer image to obtain its precise outline. Then, according to the pixel width defined in the stroke parameter, a circular structuring element is used to perform the dilation operation in image morphology on the outline to generate a smooth and equal-width expanded outline region, which is then filled with the color defined in the stroke parameter. To generate the soft special effect layer image, the system also bases on the foreground outline and performs displacement and color filling according to the pixel offset defined in the soft special effect parameter, and then applies a two-dimensional Gaussian convolution. The standard deviation of the Gaussian convolution kernel is precisely controlled by the blur radius component in the soft special effect parameter, thereby achieving a controllable blur effect.

[0141] Finally, the system performs layer composition. The foreground fill layer image, the stroke layer image, and the soft special effect layer image generated independently in the previous steps are superimposed in the preset vertical order.

[0142] In one specific embodiment, the synthesis sequence is that the soft special effect layer is at the bottom layer, the outline layer is in the middle, and the foreground fill layer is at the top layer. The superposition between layers uses a standard Alpha Blending algorithm.

[0143] Specifically, for each pixel, the final synthesized color value is a weighted average of the color values of the upper layer and the lower layer, and the weight is determined by the transparency value of the pixel in the upper layer. This synthesis process can be efficiently executed on a graphics processor, and the final output is a single complete style subtitle image containing all visual styles and background transparency.

[0144] Those skilled in the art can appreciate that the units and algorithm steps of the examples described in combination with the embodiments disclosed in the present application can be implemented in electronic hardware or a combination of computer software and electronic hardware. Whether the functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. A person skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

Claims

1. An AI-based short play script translation system, characterized by, The method comprises: a subtitle recognition module for analyzing an input short drama video, determining appearance time information of a subtitle and position coordinates of the subtitle in a picture, extracting a subtitle region, and recognizing original subtitle text; analyzing a visual style of the subtitle region and extracting source style data representing the visual style; an erasing module for performing filling processing on the subtitle region to generate a picture frame sequence without original subtitles; a translation module for translating the original subtitle text to obtain target language subtitle data; an output module for generating target rendering parameters adapted to the target language subtitle data based on text features of the source style data and the target language subtitle data; and according to the target rendering parameters, performing style rendering on the target language subtitle data, and superimposing the style-rendered target language subtitle on the picture frame sequence without original subtitles.

2. The short skit subtitle translation system based on artificial intelligence according to claim 1, wherein, The extraction of the source style data representing the visual style comprises: using an image processing algorithm to analyze the color, contour and texture of the subtitle region to obtain explicit style parameters, including foreground color parameters, outline parameters, soft special effect parameters, transparency parameters and dynamic change parameters; using a style encoding network to encode the subtitle region to generate an implicit style embedding vector representing the abstract aesthetic features thereof. 3.The short play script translation system based on artificial intelligence of claim 2, wherein, The extraction of the explicit style parameters comprises: using a multi-class semantic segmentation network to segment the subtitle region to generate a hierarchical component mask that at least distinguishes foreground filling regions, hard outline regions and soft special effect regions; based on the hierarchical component mask, distinguishing the foreground filling color, outline color and soft special effect color of the subtitle region, and respectively calculating the corresponding RGB value, pixel width and pixel offset to generate the foreground color parameters, outline parameters and soft special effect parameters; based on the edge pixels of the subtitle region, performing alpha mask segmentation processing to separate and obtain the foreground color value and transparency value to generate the transparency parameters; performing time series analysis on the foreground color parameters, outline parameters and soft special effect parameters on a video timeline corresponding to the appearance time information of the subtitle to generate dynamic change parameters representing color flicker frequency, color gradient direction and rate, and texture scrolling periodic displacement amount.

4. The short skit subtitle translation system based on artificial intelligence according to claim 3, wherein, The generation of the foreground color parameters, outline parameters and soft special effect parameters comprises: for the mask of the foreground filling region, based on color distribution entropy and / or texture complexity analysis, classifying the foreground filling as solid color filling, gradient filling or texture filling, and when it is texture filling, generating a texture descriptor vector through a pre-trained feature extraction network; for the masks of the hard outline region and the soft special effect region, respectively calculating the primary color RGB value, and performing gradient analysis and / or transparency decay curve fitting on the edge contour to generate the outline width and blur radius; the hierarchical component mask serves as a region limitation for the foreground color parameters, outline parameters, soft special effect parameters and transparency parameters. The texture descriptor vector is used as a texture filling description parameter of the foreground color parameter; the main color RGB value, the stroke width, and the blur radius are used as components of the foreground color parameter, the stroke parameter, and the soft special effect parameter respectively, and are used as inputs of the timing analysis.

5. The short skit subtitle translation system based on artificial intelligence according to claim 4, wherein, The timing analysis includes: A frame-by-frame sequence is constructed for the foreground color parameter, the stroke parameter, and the soft special effect parameter on a unified time reference aligned with the appearance time information; A time-frequency analysis is performed on the frame-by-frame sequence to identify and parameterize the time point, intensity, and duration of the transient flicker event of the subtitle brightness or chrominance, and a color flicker frequency is generated; Within a preset time window, the color vector change sequence is fitted and matched with a nonlinear interpolation function library including a fade curve to determine the best matching function type and its key frame parameters, and a color gradient direction and rate are generated; When the foreground filling is determined to be texture filling, a flow field is calculated based on the foreground filling area between adjacent frames and the flow field is decomposed to obtain composite motion parameters including translation, rotation, and scaling components, and a periodic displacement amount of texture scrolling is generated; The color flicker frequency, the color gradient direction and rate, and the periodic displacement amount of texture scrolling are collected as the dynamic change parameters and output.

6. The short skit subtitle translation system based on artificial intelligence according to claim 5, wherein, The timing analysis further includes: The synchronous audio data of the video is analyzed on a time axis corresponding to the appearance time information to extract the prosody features of the voice and / or the beat transient points of the music; Based on the prosody features and / or the beat transient points, a time axis alignment and a synchronization relationship calibration are performed on the dynamic change parameters; The dynamic change parameters including the synchronization relationship are output.

7. The short skit subtitle translation system based on artificial intelligence according to claim 2, wherein, The generation of the implicit style embedding vector representing the abstract aesthetic features thereof includes: Multi-scale feature maps are extracted within a spatial range aligned with the subtitle area, and spatial pooling is performed on each scale feature to generate a scale feature convergence result; Channel statistics are calculated based on the scale feature convergence result to generate a first style statistical vector; Inter-channel correlation statistics are calculated based on the scale feature convergence result to generate a second style statistical vector; the second style statistical vector is composed of the vectorization result of the correlation statistics; Content decoupling processing is performed on the scale feature convergence result to generate a content-free feature vector; the content decoupling processing includes instance normalization, and frequency domain low-pass filtering and / or multi-scale smoothing convolution; The first style statistical vector, the second style statistical vector, and the content-free feature vector are time-aggregated within a continuous time slice corresponding to the appearance time information to generate an implicit style embedding vector; The channel statistics include channel mean, channel variance, and channel skewness; the inter-channel correlation statistics include a correlation matrix and / or a Gram matrix.

8. The short skit subtitle translation system based on artificial intelligence according to claim 7, wherein, The time aggregation includes: The first style statistical vector, the second style statistical vector, and the element-wise time average within the continuous time slice are calculated to generate a static style vector representing a static base style; The vector at each time point within the continuous time slice is subtracted from the static style vector to generate a residual vector sequence representing a dynamic superimposed style; The residual vector sequence is processed by using a time sequence encoder based on a self-attention mechanism, and an attention weight is calculated for each time point in the residual vector sequence, the attention weight representing dynamic significance of the time point; The residual vector sequence is weighted and summed based on the attention weight to generate a dynamic style vector representing a dynamic superimposed style; and The static style vector and the dynamic style vector are spliced or added to generate a final implicit style embedding vector. 9.The short skit subtitle translation system based on artificial intelligence of claim 2, wherein, The target rendering parameters adapted to the target language subtitle data are generated based on the source style data and the text features of the target language subtitle data, including: The target language subtitle data is pre-rendered to obtain target text features representing the morphology of the target language subtitle data, the target text features including at least one of the number of characters, the width-height ratio of the rendering bounding box, and the pixel density; The explicit style parameters and the target text features are input into a parameter mapping model to calculate and generate adjusted explicit style parameters adapted to the geometric morphology of the target language subtitle data; The implicit style embedding vector is input into a generative style rendering network as a style condition, and the target language subtitle data is input into the generative style rendering network as a content condition to generate generative rendering parameters including font shaping and texture details; and The adjusted explicit style parameters and the generative rendering parameters are fused to form the target rendering parameters. 10.The short play script translation system based on artificial intelligence of claim 9, wherein, The style rendering of the target language subtitle data according to the target rendering parameters includes: Based on the font shaping and texture details included in the generative rendering parameters, and combined with the foreground color parameter in the adjusted explicit style parameters, a foreground fill layer image of the target language subtitle data is generated; Based on the stroke parameter and the soft special effect parameter in the adjusted explicit style parameters, a stroke layer image and a soft special effect layer image are respectively generated at the contour periphery or the specified offset position of the foreground fill layer image; and The foreground fill layer image, the stroke layer image, and the soft special effect layer image are layer-composited in a preset vertical order to form a final style subtitle image.

Images