[0088] The present invention will be described in detail below with reference to the drawings and specific embodiments.
[0089] The main core solution of the present invention is based on the data structure of the subtitle broadcasting system, using a random number generator to simulate the actual application environment, automatically constructing a graphic subtitle object, and then using a subtitle rendering engine to perform multithreaded automatic static rendering and dynamic dynamics on the subtitles Special effects rendering, while recording the usage of system resources, to monitor the flaws in the subtitle rendering logic design, and to ensure the safety, real-time and stability of the subtitle broadcast system.
[0090] The technical content involved in the present invention is as follows:
[0091] Random generator model of static rendering parameters for TV subtitles
[0092] Random generator model of dynamic special effects rendering parameters for TV subtitles
[0093] TV subtitle rendering engine model
[0094] TV subtitle single-driven playout model
[0095] Practical adjustment of random parameter model
[0096] Distributed, multi-threaded subtitle rendering and broadcast based on unattended
[0097] Rendering and playout engine based on continuous testing
[0098] 1. TV subtitle parameter random generator model
[0099] Due to the diversity and complexity of TV subtitle applications, the data structure of a subtitle broadcast server is very complicated, and the main body data includes static rendering data and dynamic special effects parameters.
[0100] 1.1 The static rendering parameter model of the subtitle object:
[0101] In terms of static rendering parameters, the following parameters are included:
[0102] Scene types: including ordinary subtitle scenes, scrolling subtitle scenes, interstitial subtitle scenes, and lyrics subtitle scenes. Each subtitle scene contains multiple different types of subtitle objects;
[0103] Subtitle objects: including general text subtitles, graphic element subtitles, three-dimensional objects, vocal subtitles, animated subtitles, storyboard subtitles. Each type of subtitle object includes the geometric properties of the subtitle, two-dimensional rendering parameters, and three-dimensional rendering parameters.
[0104] The geometric properties of the subtitle object: including two-dimensional space size, TrueType font library, font size of characters, logical width, rotation angle, three-dimensional space properties.
[0105] The two-dimensional rendering parameters of the subtitle object: the surface, inner edge, outer edge, side edge, relief, projection, filling, inner glow, outer glow, inner shadow, outer shadow, gloss property, and stroke property of the subtitle object. The rendering source for each rendering includes monochrome, gradient color, and image texture filling.
[0106] The three-dimensional rendering parameters of the subtitle object: the vertex color of the three-dimensional object, plane texture, bump mapping, reflection texture, material properties, and three-dimensional lighting model.
[0107] figure 1 It is the general data structure of the subtitle object. It can be seen from this figure that a complete subtitle list is a linked list composed of several subtitle broadcast tasks; a subtitle broadcast task is a linked list composed of several subtitle scenes; a subtitle scene is a linked list composed of several subtitle objects Linked list.
[0108] figure 2 It is a static rendering model of a subtitle object. It can be seen that a typical subtitle system has a very complex data structure.
[0109] 1.2 Dynamic rendering parameter model of subtitle object:
[0110] In terms of dynamic rendering parameters, the following parameters are included:
[0111] Types of dynamic stunts: such as water pottery, space transformation stunts, wave stunts, flame stunts, path stunts, packaging stunts.
[0112] Specific parameters of different types of special effects.
[0113] Combination stunts.
[0114] image 3 It is a dynamic rendering model of a subtitle object.
[0115] 1.3 Subtitle object parameter random number model
[0116] According to the above parameter model and the value range of each parameter, a random number generator is used to automatically generate the subtitle object. Figure 4 The random number model for the subtitle object parameter.
[0117] The random number generator is a well-known technology, which is used in many practical applications, such as game design, information simulation, spread spectrum communication, information encryption, and system testing. The main purpose of using the random number generator here is: first, unattended automatic rendering; second, the generated subtitle objects can cover as much as possible all the situations when the user actually uses the subtitle broadcast server.
[0118] 2. TV subtitle rendering engine model
[0119] Figure 5 Render engine model for TV subtitles. A brief description of this engine is as follows:
[0120] The first step is to enumerate all the TrueType fonts in the system, and for each TrueType font, enumerate all the valid characters in this font.
[0121] The second step is to enumerate all the primitive classes that can be processed in the system.
[0122] The third step is to start a rendering.
[0123] In a rendering process, if the current subtitle type is a character, then:
[0124] (1) Traverse each TrueType font library;
[0125] (2) For each TrueType font, traverse each valid character in the font;
[0126] (3) For each character, cyclically process each font size according to the preset minimum font size, maximum font size, and font size step size;
[0127] (4) For each font size, according to the preset minimum width, maximum width, and width step length, loop processing each width;
[0128] (5) For each width, random static rendering parameters and dynamic rendering parameters are generated according to the random number generator in the subtitle object rendering engine;
[0129] (6) Create a subtitle object, and assign the above static and dynamic rendering parameters to this subtitle object;
[0130] (7) Generate a subtitle project file from this subtitle object according to the organization form of the subtitle project file;
[0131] (8) Pass this subtitle object to the rendering engine of the broadcast system for static rendering and dynamic broadcast;
[0132] (9) During the rendering process, record the usage of CPU, GPU, physical memory, and virtual memory;
[0133] (10) Error handling.
[0134] Determine the type of error: downtime, CPU, GPU, and memory usage exceeds the preset threshold.
[0135] It is generally considered that downtime is the most serious problem. If downtime occurs, you need to start the subtitle production system, open the originally saved subtitle project file, and perform tracking and debugging.
[0136] If the usage of CPU, GPU, and memory exceeds the preset threshold, further judgments are needed, such as processing of recursive functions, recording of memory usage applications and releases, and triangulation of GPUs.
[0137] If the current subtitle type is primitive, then:
[0138] (1) Traverse each type of graphic element;
[0139] (2) For each graphic element, process each height in a loop according to the preset minimum height, maximum height, and height step;
[0140] (3) For each height, according to the preset minimum width, maximum width, and width step length, loop processing each width;
[0141] (4) For each width, generate random static rendering parameters and dynamic rendering parameters according to the random number generator in the subtitle object rendering engine;
[0142] (5) Create a subtitle object, and assign the above static and dynamic rendering parameters to this subtitle object;
[0143] (6) Generate a subtitle project file from this subtitle object according to the organization form of the subtitle project file;
[0144] (7) Pass this subtitle object to the rendering engine of the broadcast system for static rendering and dynamic broadcast;
[0145] (8) During the rendering process, record the usage of CPU, GPU, physical memory, and virtual memory;
[0146] (9) Error handling. Same as above.
[0147] The fourth step is to loop for the next rendering.
[0148] 3. TV subtitle single-driven broadcast model
[0149] The final use mode of the subtitle broadcast equipment is to generate a subtitle broadcast list based on a program series list and a pre-defined subtitle template, and then broadcast one by one according to the absolute time. Therefore, we also need to define a "broadcast model driven by TV subtitles", in which subtitle objects can be generated by the above random number model, and the broadcast time can also be randomly generated, but it must be based on certain pre-sequence rules .
[0150] TV subtitles single-driven broadcast model such as Figure 6 Shown. The description of this model is as follows:
[0151] (1) A program list is randomly generated, and the broadcast time of each subtitle entry can be randomly generated, but the rules before and after need to be followed;
[0152] (2) Randomly generate a subtitle broadcast sequence for each broadcast item;
[0153] (3) For each subtitle broadcast sequence, generate a subtitle object or an object group according to the random number model;
[0154] (4) These subtitle objects are generated into a subtitle project file according to the organization form of the subtitle project file;
[0155] (5) Pass this subtitle object to the rendering engine of the broadcast system for static rendering and dynamic broadcast;
[0156] (6) During the rendering process, record the usage of CPU, GPU, physical memory, and virtual memory;
[0157] (7) Perform error handling;
[0158] (8) Process the next program list cyclically.
[0159] Since a subtitle playlist is composed of multiple subtitle objects, there are complex logical relationships in time and space between these subtitle objects, and the information covered is much richer, so the way to deal with errors is also complicated. Many. There will be a separate error handling mechanism for this rendering model, which is introduced as follows:
[0160] Downtime error handling: It is generally considered that downtime is the most serious problem. If there is a downtime, you need to start the subtitle broadcast system, open the original saved subtitle project file, and perform tracking and debugging.
[0161] Real-time processing of subtitles single-broadcasting: In the information broadcasted by a subtitle single-entry, there will be real-time information, recording the number of frames broadcast per second, and judging whether it is real-time by this value. There are four situations that will cause the subtitles to be broadcast in real time: First, the spatial relationship of the subtitle objects exceeds the processing capacity, as if there are too many layers at a time, or the space of related graphics elements is too large, or the rendering parameters are too complicated. In this case, the specific algorithm needs to be optimized, and the limitations and risks of the product application are proposed; second, the GPU acceleration capability is not enough, mainly refers to the three-dimensional acceleration function of the graphics card, in this case, you need to change the graphics according to the specific situation Accelerator; Third, the processing power of the CPU-based rendering algorithm is not enough. At this time, the rendering algorithm needs to be optimized. For example, the pre-rendering method is used to generate animations from the broadcast content. Fourth, the subtitle broadcast channel card (input and output card) appears Data is blocked, which leads to jitter in the broadcast effect. At this time, it is necessary to track and debug the compatibility of the channel card and the input and output mechanism.
[0162] Subtitle single broadcast time relationship processing: each subtitle broadcast list will be given a trigger time attribute accurate to the frame, and the corresponding broadcast thread will trigger the broadcast of the subtitle according to this time. In the broadcast information of a subtitle list entry, the actual start time of the subtitle broadcast list will be accurately recorded. When the error is handled, the two can be compared to find the problem. And by comparing the broadcast time of the sequence composed of multiple broadcast orders, it is confirmed whether there is a cumulative error problem. Because in the actual work process of the broadcast server, a certain range of error is allowed in the broadcast time of a single subtitle (the specific range value depends on the requirements of different programs), but the cumulative error is absolutely not allowed.
[0163] The processing of the spatial relationship of the single subtitles: mainly confirm whether there is an overlap in the physical space of different subtitles at the same time (or within the same time period). In the actual work of the subtitle broadcast server, a subtitle list is a basic broadcast unit, and the broadcast sequence is broadcasted one by one in the order of the subtitle list, namely: subtitle list 1 in-->subtitle list 1 out-- >Subtitle list 2 in-->Subtitle list 2 out...But sometimes it is necessary to make an emergency interruption. The inserted subtitle list cannot be overlapped with the subtitle list currently being broadcast in the physical location, otherwise the final broadcast effect will be There is coverage. Therefore, in each subtitle broadcast information, it is necessary to record the subtitle information of the same time period and the space information of each subtitle finally broadcast, so as to perform error processing.
[0164] 4. Practical adjustment of random parameter model
[0165] The above rendering models are completely generated based on random number generators. In theory, these rendering models can traverse all the logic branches of static rendering and dynamic special effects algorithms to the greatest extent, thereby discovering design defects to the greatest extent. In practical applications, the probability of using some parameter values is very small. For example, there are only about 3000 Chinese characters commonly used in TV subtitles, while there are more than 40,000 actual characters in a TrueType; for another example, for one character , The value range of the radius of bordering and shadowing is 0 to 500 pixels, but in fact the most used value is concentrated in 5 to 20. In this case, if we are completely based on the random number generator, the parameters obtained are completely out of control. In this case, the many renderings we perform are actually meaningless.
[0166] In this case, the random number model needs to be corrected so that the automatically generated parameters are more in line with the actual application. The main transformation includes the following three aspects:
[0167] First, establish a database of commonly used Chinese characters, words, and sentences (dictionaries, dictionaries, sentence dictionaries), which can be organized simply by using Windows standard TEXT text files. When rendering, instead of traversing all valid characters in a TrueType font, it traverses all text information in a certain text file.
[0168]Second, because the computer's random number generator generates a pseudo-random number sequence, it is necessary to use a variable as the seed of the random number before each rendering. No matter which programming language is used, it is a pseudo-random number sequence returned by the computer's random number generator. This set of sequences is actually a set of values calculated according to the recursive formula. When the sequence is long enough, this set of values Approximately meet a uniform distribution. But if the initial value (called the seed) of the calculated pseudo-random sequence is the same, the calculated pseudo-random sequence is exactly the same, so in order to make the randomness of each rendering parameter more reasonable, so as to cover the program of the rendering engine more Logical branching requires a variable to be used as a seed for random numbers before each rendering. There are many ways to specify the random number seed, and using the computer clock is the simplest and most effective.
[0169] Therefore, in the actual rendering model, we use the computer's clock TickCount.
[0170] DWORD dwTickCount=GetCurrentSys temTickCount();
[0171] srand(dwTickCount);
[0172] Third, determine an empirical value range for the value range of most parameters, so that more than 90% of the value generated by the random number generator falls within this empirical value range. The specific algorithm is as follows:
[0173] Assumption 1: The value range of a parameter is [M, N].
[0174] Assumption 2: In actual use, the effective value of this parameter falls within the interval of [S1, S2], M
[0175] Hypothesis 3: Transform the random number generator model so that the percentage of random numbers generated in the range of [M, N] is P falls within the range of [S1, S2]. P is a threshold. In actual use, 90% is generally used.
[0176] Based on the above assumptions, the model of the random number generator is as follows:
[0177] The first step is to reduce the interval [M, N] to the interval [0, (N-M)], and then generate a random number K in the interval [0, (N-M)];
[0178] The second step is to determine the mapping algorithm formula according to the range of K value, and recalculate the required random parameters.
[0179] If K is in the interval [0, P*(N-M)], map the value of K from the interval [0, P*(N-M)] to the interval [S1, S2], the mapping algorithm formula is as follows:
[0180] I = S 1 + ( S 2 - S 1 ) × K P × ( N - M )
[0181] If K is within the interval of [P*(NM), (1+P)/2*(NM)], change the value of K from the interval [P*(NM), (1+P)/2*(NM)] Mapping to the interval [M, S1], the mapping algorithm formula is as follows:
[0182] I = M + ( S 1 - M ) × ( 2 × K ( 1 - P ) × ( N - M ) - 2 × P 1 - P )
[0183] If K is within the interval of [(1+P)/2*(NM), (NM)], map the value of K from the interval [(1+P)/2*(NM), (NM)] to the interval [ Within S2, N], the mapping algorithm formula is as follows:
[0184] I = S 2 + ( N - S 2 ) × ( 2 × K ( 1 - P ) × ( N - M ) - 1 + P 1 - P )
[0185] The I obtained after the calculation is the random number needed at the end.
[0186] The adjustment of the automatic parameter generation model through the above three aspects ensures that the subtitle rendering parameters are closer to the actual application environment, and the effectiveness of this method is guaranteed.
[0187] 5. Based on unattended distributed, multi-threaded subtitle rendering and broadcast
[0188] Even with the above transformation of the random number generator model, in actual work, the rendering of subtitles is very time-consuming. Take character rendering as an example, assuming that the actual available TrueType font library in the current Windows operating system is 100 , To render 2000 common Chinese characters in each font, the font size of each character ranges from 8 to 200, and the width ranges from 10% to 300%. The number of rendering times is:
[0189] 100×2000×(200-8)×(300-10)=11136000000 times
[0190] Assuming that the average time of each rendering is 50 milliseconds, then the time to complete the rendering is: 5.56×10 11 Milliseconds, approximately equal to 155,000 hours, approximately 6400 days.
[0191] In view of practical considerations, the present invention proposes the idea of distributed, multi-threaded subtitle rendering: use a cluster of rendering workstations, and deploy different rendering parameters on each workstation through a preset method, for example, these 100 font libraries They are installed on different workstations, and a different data dictionary is specified on each workstation. For one workstation, multiple rendering threads can also be started to render different types of subtitle content. Then, by deploying a workstation, centrally monitor the rendering status of each workstation.
[0192] Figure 7 Shows an unattended distributed, multi-threaded subtitle rendering and playout engine. Deploy cluster rendering workstations according to the equipment, and formulate the rendering tasks of each workstation before rendering. The monitoring and error handling terminal is used to collect the rendering status of each rendering workstation, collect error information and process it.
[0193] 6. Subtitle rendering and playout engine based on continuous testing
[0194] As stated in the previous section, the rendering of subtitles is a time-consuming task. The ultimate goal of the method mentioned in the present invention is to find the flaws in the design of the subtitle broadcasting equipment. Then, when the error occurs, once the system exits, the work to be done is to design logic based on some data left at the scene of the error. Perform analysis and finally correct this defect. Then when the defect is corrected, you need to restart the rendering of the subtitles. If you still render from the beginning at this time, some work must be repeated and meaningless. In view of this, the present invention also proposes a subtitle rendering and playout engine based on continuous measurement of breakpoints, the main function of which is to perform rendering following the parameters of the last rendering exit. In this way, the efficiency of subtitle rendering can be improved, and some useless work can be avoided. The main design idea is: when the rendering parameters are generated, before handing over to the rendering engine, first use a text log or binary file to record these rendering parameters, and then restore the parameters based on these files when restarting the next time. And perform subsequent rendering tasks according to the restored subtitle parameters.
[0195] The method of the present invention is not limited to the examples described in the specific implementation manners. Those skilled in the art can obtain other implementation manners based on the technical solutions of the present invention, which also belong to the technical innovation scope of the present invention.
