In-picture text translation replacement method and device

Abstract

The application provides a method and device for translating and replacing text in a picture, which cross-frame associates text candidate regions in each frame of an original video frame sequence to generate track assets, that is, to change single-frame candidate boxes into assets that can be reused in continuous frames, then determines target modes corresponding to each track in the track assets based on the track assets and a set of text candidate regions, and processes the tracks based on processing manners corresponding to the target modes to obtain target assets of the tracks; and then generates video frames after translation and replacement based on the target assets of the tracks and the track assets, so as to efficiently and high-quality generate video frames after translation and replacement to meet requirements for film and television level delivery.

Description

Technical Field

[0001] This invention relates to the field of video processing technology, and in particular to a method and apparatus for translating and replacing text within a video frame. Background Technology

[0002] In the process of exporting, importing, and distributing film and television content overseas, the text within the video often carries key information, such as: people chatting on their phones, road signs, scoreboards, program packaging cards, and opening and closing credits.

[0003] In existing technologies, the translation of key information typically involves manual methods such as annotating text areas frame by frame, manually repairing the background, and then having artists add text and adjust colors. This process is costly, time-consuming, and its quality depends heavily on experience. Furthermore, any changes to the text require repeated rework. Therefore, existing technologies have developed automated tools for translating key information, which usually rely on "single-frame detection + erasure + text addition." However, in scenarios involving occlusion, motion blur, compressed noise, and screen refresh, these tools are prone to issues such as drifting, flickering, edge artifacts, screen compositing artifacts, and translation overflow, making it difficult to meet the requirements for film-quality delivery. Summary of the Invention

[0004] In view of this, the present invention provides a method and apparatus for translating and replacing text within a video image, which efficiently and with high quality generates translated and replaced videos to meet film and television-grade delivery requirements.

[0005] The first aspect of this invention provides a method for translating and replacing text within a screen, comprising:

[0006] Obtain the original video frame sequence and determine the set of text candidate regions in each frame of the original video frame sequence;

[0007] The text candidate regions in each frame of the original video frame sequence are correlated across frames to generate trajectory assets;

[0008] Based on the trajectory asset and the set of text candidate regions, determine the target pattern corresponding to each trajectory in the trajectory asset;

[0009] The trajectory is processed based on the processing method corresponding to the target pattern to obtain the target asset of the trajectory;

[0010] Based on the target asset and the trajectory asset, a translated and replaced video frame is generated. Optionally, determining the set of candidate text regions in each frame of the original video frame sequence includes:

[0011] Perform shot boundary detection on each frame of the original video frame sequence to obtain a set of shot segments;

[0012] For each shot in the shot clip set, determine the set of keyframes for that shot clip;

[0013] For each keyframe in the set of keyframes, text detection is performed to obtain candidate text regions.

[0014] Optionally, the step of cross-frame association of text candidate regions in each frame of the original video frame sequence to generate trajectory assets includes:

[0015] Associate the same text candidate region in consecutive frames of the original video frame sequence as a trajectory;

[0016] For each trajectory, image feature information of each frame of the trajectory is extracted, and for each frame of the trajectory, the mapping parameters of the text candidate region in the frame are determined based on the image feature information of that frame.

[0017] If the boundaries of the text candidate regions in N consecutive frames of the trajectory meet the first preset condition, the trajectory is taken as an approximate planar trajectory, and a surface coordinate system is constructed based on the approximate planar trajectory.

[0018] Optionally, the step of constructing the surface coordinate system based on the approximate planar trajectory includes:

[0019] The frames in the approximate planar trajectory that satisfy the second preset condition are used as reference frames;

[0020] Based on the text candidate regions in the reference frame, a reference boundary is generated, and a rectangular plane is established based on the reference boundary;

[0021] Each frame in the trajectory is mapped onto the rectangular plane to obtain the surface coordinate system.

[0022] Optionally, determining the target pattern corresponding to each trajectory in the trajectory asset based on the trajectory asset and the set of text candidate regions includes:

[0023] For each trajectory in the trajectory asset, a list of text events is generated based on the mapping parameters of the text candidate regions in each frame of the trajectory and the surface coordinate system;

[0024] Based on the trajectory assets, the list of text events, and the set of text candidate regions, the target pattern corresponding to each trajectory is determined.

[0025] Optionally, for each trajectory in the trajectory asset, based on the mapping parameters of the text candidate regions in each frame of the trajectory and the surface coordinate system, a list of text events is generated, including:

[0026] For each trajectory, the frame in the trajectory that satisfies the third preset condition is taken as the target frame;

[0027] For each target frame, the region of interest in the trajectory is mapped to the surface coordinate system using the mapping parameters of the text candidate region in the target frame, to obtain a first surface domain image;

[0028] Text detection is performed on the first surface domain image to obtain a candidate text sequence, and dynamic programming is performed on the candidate text sequence to generate a list of text events for the trajectory.

[0029] Optionally, determining the target pattern corresponding to each trajectory based on the trajectory asset, the list of text events, and the set of text candidate regions includes:

[0030] The feasibility of the atlas path is determined based on approximate planar stability, cross-frame reuse of background texture, estimated proportion of missing base plate areas, and refresh frequency. Specifically, the approximate planar stability is determined by the trajectory asset; the cross-frame reuse of background texture is determined by the proportion of reusable background pixels after generating the first surface domain image; the estimated proportion of missing base plate areas is determined by the proportion of reusable background pixels, pixel occlusion ratio, and under-observed areas; and the refresh frequency is determined by the number of event switches per unit time, the length of stable segments, and the actual change points in the text event list.

[0031] The feasibility of the template path is determined based on the strength of the template structure evidence and the refresh frequency; wherein, the strength of the template structure evidence is determined by the structural information in the set of text candidate regions;

[0032] Based on the feasibility of the atlas path and the feasibility of the template path, the target pattern corresponding to the trajectory is determined.

[0033] Optionally, if the target mode is a map atlas mode, the target asset includes map atlas assets and base plate assets. The step of processing the trajectory based on the processing method corresponding to the target mode to obtain the target asset of the trajectory includes:

[0034] The trajectory is mapped to the surface coordinate system by the mapping parameters of the text candidate regions in each frame of the trajectory, to obtain the second surface domain image;

[0035] The missing pixel set of the second surface domain image is determined, and the missing regions indicated by the missing pixel set are repaired to obtain a surface texture atlas;

[0036] The surface texture atlas is back-projected onto the surface coordinate system to obtain the base plate asset, and an atlas asset is generated based on the surface texture atlas.

[0037] Optionally, if the target pattern is a template pattern, the target asset includes template assets, and the step of processing the trajectory based on the processing method corresponding to the target pattern to obtain the target asset of the trajectory includes:

[0038] Based on the ratio between the screen boundary and the original video frame boundary, the region of interest of the original video frame is mapped to the screen coordinate system.

[0039] The content in the screen coordinate system is decomposed to obtain a text layer, and the text events in the text event list are bound to the corresponding text slots in the text layer.

[0040] Time-varying modulation terms are generated based on the dynamic features of the screen;

[0041] Based on the bound text slots, the time-varying modulation items, and the screen boundaries, a template asset is generated.

[0042] Optionally, generating translated and replaced video frames based on the target asset and the trajectory asset includes:

[0043] Based on the target assets of the trajectory and the list of text events, a rendering recipe is generated;

[0044] Based on the rendering recipe, the list of text events, and the layout constraints, a layout scheme is determined;

[0045] Based on the rendering recipe, the layout scheme, and the mapping parameters of the text candidate regions in each frame of the trajectory, a translated and replaced video frame is generated.

[0046] Optionally, the rendering recipe includes static parameters and time-varying parameters, and generating the rendering recipe based on the target asset of the trajectory and the list of text events includes:

[0047] A text support area is constructed based on the target asset of the trajectory and the list of text events;

[0048] Within the text support region and its neighborhood, residual observation is performed by calculating the residual between the original video frame and the estimated background image to obtain the residual observation result;

[0049] Based on the residual observation results, the parameters are grouped to obtain static parameters and time-varying parameters.

[0050] Optionally, determining the layout scheme based on the rendering recipe, the list of text events, and the layout constraints includes:

[0051] Identify key factual information in the stable text within the aforementioned list of textual events;

[0052] Based on the variant generation strategy, the key factual information, and the stable text in the text event list, a controlled variant is generated;

[0053] For each of the controlled variants, candidate layouts are generated by combining the typographic constraints and the visual readability requirements in the rendering recipe;

[0054] Based on all the candidate layouts, a layout scheme is generated.

[0055] Optionally, generating the translated and replaced video frame based on the rendering recipe, the typography scheme, and the mapping parameters of the text candidate regions in each frame of the trajectory includes:

[0056] Within a standardized coordinate system, layout rendering is performed based on the aforementioned typesetting scheme to obtain the translated text layer; wherein, the standardized coordinate system is either a surface coordinate system or a screen coordinate system.

[0057] Based on the rendering formula, the translated text layer is visually enhanced to obtain the target translated text layer.

[0058] The target translation text layer is projected onto the original video frame using the mapping parameters of the text candidate region in each frame of the trajectory, resulting in the translated and replaced video frame.

[0059] A second aspect of the present invention provides a translation and replacement device for text within a screen, comprising:

[0060] A text candidate region determination unit is used to acquire the original video frame sequence and determine the set of text candidate regions in each frame of the original video frame sequence.

[0061] The trajectory asset determination unit is used to associate the text candidate regions in each frame of the original video frame sequence across frames to generate trajectory assets; wherein, the trajectory assets include at least the mapping parameters and surface coordinate system of the text candidate regions in each frame of each trajectory;

[0062] The target pattern determination unit is used to determine the target pattern corresponding to each trajectory in the trajectory asset based on the trajectory asset and the set of text candidate regions;

[0063] The target asset generation unit is used to process the trajectory based on the processing method corresponding to the target mode to obtain the target asset of the trajectory;

[0064] The video frame generation unit is used to generate translated and replaced video frames based on the target asset and the trajectory asset of the trajectory.

[0065] Optionally, when the text candidate region determination unit determines the set of text candidate regions in each frame of the original video frame sequence, it includes:

[0066] A shot boundary detection unit is used to perform shot boundary detection on each frame of the original video frame sequence to obtain a set of shot segments.

[0067] The keyframe set determination unit is used to determine the keyframe set of each shot in the shot segment set;

[0068] The text candidate region determination subunit is used to perform text detection on each keyframe in the keyframe set to obtain text candidate regions.

[0069] Optionally, the trajectory asset determination unit includes:

[0070] The trajectory determination unit is used to associate the same text candidate region in consecutive frames in the original video frame sequence as a trajectory.

[0071] The mapping parameter determination unit is used to extract image feature information of each frame of the trajectory for each trajectory, and to determine the mapping parameters of the text candidate region in each frame based on the image feature information of that frame for each frame of the trajectory.

[0072] The surface coordinate system construction unit is used to construct a surface coordinate system based on the approximate planar trajectory when the boundaries of the text candidate regions of N consecutive frames in the trajectory meet the first preset condition.

[0073] Optional, surface coordinate system building units include:

[0074] A reference frame determination unit is used to select frames in the approximate planar trajectory that satisfy the second preset condition as reference frames.

[0075] A rectangular plane determination unit is used to generate a reference boundary based on the text candidate region in the reference frame, and to establish a rectangular plane based on the reference boundary;

[0076] A surface coordinate system construction sub-unit is used to map each frame in the trajectory to the rectangular plane to obtain the surface coordinate system.

[0077] Optionally, the target pattern determination unit includes:

[0078] The text list generation unit is used to generate a text event list for each trajectory in the trajectory asset, based on the mapping parameters of the text candidate region in each frame of the trajectory and the surface coordinate system.

[0079] The target pattern determination subunit is used to determine the target pattern corresponding to each trajectory based on the trajectory asset, the text event list, and the text candidate region set.

[0080] Optionally, the text event list generation unit includes:

[0081] The target frame determination unit is used to determine the frame in the trajectory that satisfies the third preset condition as the target frame for each trajectory.

[0082] The first surface domain image generation unit is used to map the region of interest in the trajectory to the surface coordinate system for each target frame by using the mapping parameters of the text candidate region in the target frame, so as to obtain the first surface domain image.

[0083] The text event list generation subunit is used to perform text detection on the first surface domain image, obtain candidate text sequences, and perform dynamic planning on the candidate text sequences to generate a text event list of the trajectory.

[0084] Optionally, the target pattern determination unit includes:

[0085] The atlas path feasibility determination unit is used to determine the atlas path feasibility based on approximate plane stability, background texture cross-frame reuse degree, estimated proportion of missing base plate area, and refresh frequency. The approximate plane stability is determined by the trajectory asset; the background texture cross-frame reuse degree is determined by the proportion of reusable background pixels after generating the first surface domain image; the estimated proportion of missing base plate area is determined by the proportion of reusable background pixels, pixel occlusion ratio, and under-observed areas; and the refresh frequency is determined by the number of event switching times per unit time, stable segment length, and actual change points in the text event list.

[0086] The template path feasibility determination unit is used to determine the template path feasibility based on the template structure evidence strength and the refresh frequency; wherein, the template structure evidence strength is determined by the structural information in the text candidate region set;

[0087] The target pattern determination subunit is used to determine the target pattern corresponding to the trajectory based on the feasibility of the atlas path and the feasibility of the template path.

[0088] Optionally, if the target mode is an atlas mode, the target asset includes atlas assets and base plate assets, and the target asset generation unit includes:

[0089] The second surface domain image generation unit is used to map the entire trajectory to the surface coordinate system through the mapping parameters of the text candidate region in each frame of the trajectory to obtain the second surface domain image.

[0090] The repair unit is used to determine the set of missing pixels in the second surface domain image and repair the missing regions indicated by the set of missing pixels to obtain a surface texture atlas.

[0091] The base plate asset and atlas asset generation unit is used to back-project the surface texture atlas onto the surface coordinate system to obtain the base plate asset, and generate atlas assets based on the surface texture atlas.

[0092] Optionally, if the target mode is a template mode, the target asset includes template assets, and the target asset generation unit includes:

[0093] The screen coordinate system generation unit is used to map the region of interest of the original video frame to the screen coordinate system based on the ratio between the screen boundary and the original video frame boundary.

[0094] The binding unit is used to decompose the content in the screen coordinate system to obtain the text layer, and bind the text events in the text event list to the corresponding text slots in the text layer.

[0095] A time-varying modulation term generation unit is used to generate time-varying modulation terms based on dynamic features of the screen;

[0096] The template asset generation unit is used to generate template assets based on the bound text slots, the time-varying modulation items, and the screen boundaries.

[0097] Optionally, the video frame generation unit includes:

[0098] A rendering recipe generation unit is used to generate a rendering recipe based on the target asset of the trajectory and the list of text events;

[0099] The layout scheme generation unit is used to determine the layout scheme based on the rendering recipe, the text event list, and the layout constraints.

[0100] The video frame generation subunit is used to generate translated and replaced video frames based on the rendering recipe, the layout scheme, and the mapping parameters of the text candidate regions in each frame of the trajectory.

[0101] Optionally, the rendering recipe includes static parameters and time-varying parameters, and the rendering recipe generation unit includes:

[0102] A text support region construction unit is used to construct a text support region based on the target asset of the trajectory and the list of text events;

[0103] The residual observation unit is used to perform residual observation within the text support area and its neighborhood by calculating the residual between the original video frame and the estimated background image, and obtain the residual observation result.

[0104] The parameter grouping unit is used to group parameters based on the residual observation results to obtain static parameters and time-varying parameters.

[0105] Optionally, the typesetting scheme determination unit includes:

[0106] A key fact information identification unit is used to identify key fact information in the stable text of the text event list;

[0107] A controlled variant generation unit is used to generate controlled variants based on a variant generation strategy, the key factual information, and stable text in the text event list.

[0108] A candidate layout generation unit is used to generate a candidate layout for each of the controlled variants, taking into account the layout constraints and the visual readability requirements in the rendering recipe.

[0109] The layout scheme determination sub-unit is used to generate a layout scheme based on all the candidate layouts.

[0110] Optionally, the video frame generation subunit includes:

[0111] The layout rendering subunit is used to perform layout rendering based on the typesetting scheme within a standard coordinate system to obtain the translated text layer; wherein the standard coordinate system is a surface coordinate system or a screen coordinate system.

[0112] The visual enhancement subunit is used to perform visual enhancement processing on the translated text layer based on the rendering recipe to obtain the target translated text layer.

[0113] The projection subunit is used to project the target translation text layer onto the original video frame using the mapping parameters of the text candidate region in each frame of the trajectory, so as to obtain the translated and replaced video frame.

[0114] A third aspect of the present invention provides an electronic device, comprising:

[0115] One or more processors;

[0116] A storage device on which one or more programs are stored;

[0117] When the one or more programs are executed by the one or more processors, the one or more processors implement the text translation and replacement method for the screen as described in any one of the first aspects.

[0118] A fourth aspect of the present invention provides a computer storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the text translation and replacement method for a screen as described in any one of the first aspects.

[0119] As can be seen from the above scheme, the present invention provides a method and apparatus for translating and replacing text within a video frame. The present invention no longer treats text within a video frame as independent single-frame detection boxes, but instead associates the text candidate regions in each frame of the original video frame sequence across frames to generate trajectory assets. That is, single-frame candidate boxes (detection boxes) are transformed into assets that can be reused across consecutive frames. Then, based on the trajectory assets and the set of text candidate regions, the target pattern corresponding to each trajectory in the trajectory assets is determined, and the trajectory is processed according to the processing method corresponding to the target pattern to obtain the target asset of the trajectory. Finally, based on the target asset of the trajectory and the trajectory assets, the translated and replaced video frame is generated. This achieves the goal of generating translated and replaced videos efficiently and with high quality to meet film-level delivery requirements. Attached Figure Description

[0120] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0121] Figure 1 A detailed flowchart of a method for translating and replacing text within a screen, provided in an embodiment of the present invention;

[0122] Figure 2 This is a schematic diagram of a text translation and replacement device within a screen, provided as another embodiment of the present invention. Detailed Implementation

[0123] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0124] The term "comprising" and its variations as used herein are open-ended inclusions, meaning "including but not limited to". The term "based on" means "at least partially based on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". Definitions of other terms will be given in the description below.

[0125] It should be noted that the information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this invention are all information and data authorized by the user or fully authorized by all parties.

[0126] It should be noted that the concepts of "first" and "second" mentioned in this invention are only used to distinguish different devices, modules or units, and are not used to limit the order of functions performed by these devices, modules or units or their interdependencies.

[0127] It should be noted that the terms "a" and "a plurality of" used in this invention are illustrative rather than restrictive. Those skilled in the art should understand that, unless otherwise expressly indicated in the context, they should be understood as "one or more".

[0128] This invention provides a method for translating and replacing text within a screen, such as... Figure 1 As shown, the specific steps include:

[0129] S101. Obtain the original video frame sequence and determine the set of text candidate regions in each frame of the original video frame sequence.

[0130] The candidate text region set (CandidateSet) includes, but is not limited to, candidate text region information and frame numbers, etc., which are not limited here. The candidate text region information includes, but is not limited to, candidate text region numbers and candidate text region types (e.g., candidate boxes, candidate quadrilaterals, etc.), which are not limited here.

[0131] It should be noted that the original video frame sequence can be directly input by the user. Of course, the user can also input the original video and use pre-set video frame extraction tools to extract the video frames from the original video to obtain the original video frame sequence. There is no limitation here.

[0132] Optionally, in another embodiment of the present invention, one implementation of determining the set of candidate text regions in each frame of the original video frame sequence specifically includes the following steps (steps A1 to A3):

[0133] Step A1: Perform shot boundary detection on each frame of the original video frame sequence to obtain a set of shot segments.

[0134] It should be noted that lens boundary detection can be implemented by, but is not limited to, color histogram changes, inter-frame embedding vector changes, or combinations thereof; no limitation is made here.

[0135] Optionally, in another embodiment of the present invention, the lens boundary information in the original video frame sequence can be directly input in advance, so that lens boundary detection is no longer required and step A2 can be executed directly. This is not limited here.

[0136] Step A2: For each shot in the shot clip set, determine the set of keyframes for that shot clip.

[0137] Specifically, for each shot segment, the first frame, middle frame, and last frame can be extracted as basic keyframes, but not limited to. If a sudden change in light flow amplitude or text density is found near the shot boundary, at the corresponding location, more keyframes are obtained, thus obtaining a set of keyframes.

[0138] Optionally, in another embodiment of the present invention, the keyframe set of the original video frame sequence can be directly input in advance, so that keyframe detection is no longer required, and step A3 can be executed directly. This is not limited here.

[0139] Step A3: Perform text detection on each keyframe in the keyframe set to obtain text candidate regions.

[0140] The text detection method can be, but is not limited to, scene text detectors, text segmenters, connected component methods, or combinations thereof; no limitation is made here.

[0141] In the actual application of this invention, after obtaining the candidate text regions, overlapping candidate regions can be merged and suppressed to obtain processed candidate regions, which is not limited here.

[0142] In the practical application of this invention, further information can be extracted from the content in the text candidate area to provide certain pattern type evidence for the subsequent target pattern determination process, including but not limited to:

[0143] 1. Text detection score, character connectivity, and boundary closure;

[0144] 2. Strength of regular rectangular borders, residuals of quadrilateral fitting, and perspective stability;

[0145] 3. UI control boundary response, fixed icon position response, and consistency of layout for separator lines and color blocks;

[0146] 4. Pixel raster, scan lines, compressed noise spectrum, and screen refresh texture response;

[0147] 5. Background texture complexity, candidate area brightness stability, and frequency of suspected text changes.

[0148] In the practical application of this invention, by organizing the above evidence into structured type evidence records and evidence summaries, a preliminary type judgment result is obtained.

[0149] The method for initial type determination can be provided by a rule system, a classification model, or a combination thereof, and should at least distinguish between text attached to physical objects, text on paper, text on screens / UI, and text that is difficult to determine, etc. There are no restrictions here.

[0150] In other words, the CandidateSet in this invention may also include text detection scores, preliminary type judgment results, type evidence records, and evidence summaries.

[0151] Among them, the type evidence record and evidence summary can be directly used as input for the process of determining the target pattern corresponding to the trajectory, and can also be used for subsequent manual review. The text detection score is used to represent the confidence level of this text detection, and there are no restrictions here.

[0152] It should be noted that the keyframe detection method and text detection method used in this invention can employ any existing detection method, and are not limited here. This is because this invention does not focus on "changing the detector or detection method" for detection, but rather organizes the generated detection results into a CandidateSet that can be directly consumed by subsequent trajectory construction and mode selection.

[0153] Optionally, in another embodiment of the present invention, the text candidate regions of the original video frame sequence can be directly input in advance, so that it is no longer necessary to detect the text candidate regions. This is not limited here.

[0154] Optionally, in another embodiment of the present invention, project configurations, such as sampling density, may be provided in advance to assist in keyframe acquisition, which is not limited here.

[0155] S102. Cross-frame association of text candidate regions in each frame of the original video frame sequence to generate trajectory assets.

[0156] The trajectory assets include at least the mapping parameters and surface coordinate system of the text candidate regions in each frame of each trajectory.

[0157] Optionally, in another embodiment of the present invention, one implementation of step S102 specifically includes the following steps (steps B1 to B3):

[0158] Step B1: Associate the same text candidate region in consecutive frames of the original video frame sequence as a trajectory.

[0159] In the actual application of this invention, the correlation score between consecutive frames can be further calculated to determine whether the same text candidate region in two consecutive frames is associated as a trajectory.

[0160] Specifically, factors such as, but not limited to, the following can be considered:

[0161] 1. Geometric overlap of candidate boxes or candidate quadrilaterals ;

[0162] 2. Similarity of texture, border, and text appearance of candidate text regions. ;

[0163] 3. Motion continuity obtained based on optical flow, center point displacement, or motion trend. ;

[0164] 4. Geometric consistency of whether candidate text regions can be stably mapped to the same plane. ;

[0165] 5. Adverse effects caused by characters, props, or foreground objects obstructing the view. .

[0166] In the practical application of this invention, the correlation score can be calculated using, but is not limited to, the following formula. :

[0167] ;

[0168] Among them, the weights w1, w2, w3, w4, and w5 are pre-set weights that can be pre-configured according to business scenarios, sample statistical results, or engineering experience, and can be adapted and adjusted in different types of videos. No limitation is made here.

[0169] In the practical application of this invention, Hungarian matching, greedy matching, or a combination thereof can be used to select the consecutive frames with the highest association score within each time window and associate them as a trajectory, and maintain the trajectory state machine: entry, stability, brief loss, occlusion, and termination.

[0170] Optionally, in another embodiment of the present invention, an association score threshold can also be set. The association score must also meet the association score threshold in order to be associated as a trajectory. This is not limited here.

[0171] Optionally, in another embodiment of the present invention, a time interval can be set for consecutive frames, and the correlation score is no longer calculated for two consecutive frames that exceed the preset time interval; this is not limited here.

[0172] Optionally, in another embodiment of the present invention, if a trajectory is briefly lost within a preset time interval, but subsequent candidates meet the association threshold and geometric consistency requirements again, then "backlink after brief loss" is allowed.

[0173] Step B2: For each trajectory, extract the image feature information of each frame of the trajectory, and for each frame of the trajectory, determine the mapping parameters of the text candidate region in that frame based on the image feature information of that frame.

[0174] Image feature information includes, but is not limited to, border corner points, background texture feature points, and intersections of edge lines, etc., which are not limited here.

[0175] In practical applications of this invention, mapping parameters for candidate text regions in a frame can be determined based on the image feature information of that frame, but not limited to least squares, RANSAC, or a combination thereof. No restrictions are specified here, but it should be noted that... These are estimated values, not preset values.

[0176] Step B3: If the boundaries of the text candidate regions in N consecutive frames of the trajectory meet the first preset condition, the trajectory is taken as an approximate planar trajectory, and a surface coordinate system is constructed based on the approximate planar trajectory.

[0177] It should be noted that an approximate plane refers to the candidate text regions within the trajectory that can be considered as being attached to the same plane. The first preset condition can be, but is not limited to, presenting a stable quadrilateral, having approximately coplanar boundary lines, and having a reprojection error below an error threshold; these conditions are not specified here.

[0178] Optionally, in another embodiment of the present invention, one implementation of constructing a surface coordinate system specifically includes the following steps (steps B31 to B33):

[0179] Step B31: Use the frame that satisfies the second preset condition in the approximate planar trajectory as the reference frame.

[0180] The second preset conditions include, but are not limited to, whether the clarity meets the clarity threshold, whether the occlusion degree meets the occlusion degree threshold, and whether the boundary is complete, etc., which are not limited here.

[0181] Specifically, any frame can be used as a reference frame; the frame with a sharpness greater than the sharpness threshold, an occlusion less than the occlusion threshold, and a complete boundary can be used as a reference frame.

[0182] Step B32: Generate a reference boundary based on the text candidate region in the reference frame, and establish a rectangular plane based on the reference boundary.

[0183] In the practical application of this invention, the reference boundary can be obtained by, but is not limited to, candidate quadrilaterals, border corner points, edge line fitting results, etc., and is not limited here.

[0184] In the practical application of this invention, a rectangular plane with the same length and width ratio as the reference boundary can be established, which is not limited here.

[0185] Step B33: Map each frame in the trajectory to a rectangular plane to obtain the surface coordinate system.

[0186] Specifically, the mapping relationship between the reference frame and other frames within the trajectory to the rectangular plane is calculated so that the text areas of each frame within the trajectory can be aligned to the same coordinate plane, thus obtaining the surface coordinate system.

[0187] Optionally, in another embodiment of the present invention, if a certain trajectory cannot stably determine the reference boundary, or if the mapping error is too large for multiple consecutive frames, the trajectory is not considered to have successfully established a surface coordinate system. Instead, the cause code is retained and the process is switched to the template path or the backtracking path.

[0188] Optionally, in another embodiment of the present invention, in order to suppress jitter, the mapping parameters of each frame are temporally smoothed. The smoothing can be achieved by sliding window weighted averaging, spline smoothing or small-scale joint optimization; wherein, the weight can be determined comprehensively based on the sharpness of the frame, the proportion of effective feature points and the reprojection error.

[0189] Optionally, in another embodiment of the present invention, in addition to the mapping parameters and surface coordinate system of the text candidate region in each frame of each trajectory, the trajectory asset may also include trajectory number, frame interval of the trajectory, surface coordinate system definition, text mask, background mask, foreground occlusion mask, quality curve, trajectory comprehensive credibility, evidence summary, etc., which are not limited here.

[0190] Among them, the surface coordinate system definition is used to record the definition of the reference frame, reference boundary and rectangular plane; the quality curve is used to record the frame-by-frame changes of sharpness score, occlusion ratio and fitting residual; the sharpness score can be obtained by Laplacian energy, edge gradient energy or a combination thereof; the trajectory comprehensive reliability can be determined by detection stability, cross-frame correlation stability, mapping fitting stability, occlusion ratio and sharpness, without limitation here.

[0191] S103. Based on the trajectory asset and the set of text candidate regions, determine the target pattern corresponding to each trajectory in the trajectory asset.

[0192] The target mode can be divided into three types: atlas mode, template mode, and fallback mode. Atlas mode is suitable for text affixed to realistic textured surfaces, such as store signs, posters, and title cards; template mode is suitable for structured UI text such as mobile phone screens, scoreboards, and program packaging; and fallback mode can be executed when neither of the two main paths can form a deliverable intermediate asset.

[0193] It should be noted that this invention does not simply label each trajectory and decide which mode to follow based on experience. Instead, it further analyzes the structured results (trajectory assets and text candidate region set) generated in the aforementioned steps to determine the correct processing branch for each trajectory in the trajectory asset, i.e., the target mode corresponding to the trajectory.

[0194] Optionally, in another embodiment of the present invention, one implementation of step S103 includes the following steps (steps C1 to C2):

[0195] Step C1: For each trajectory in the trajectory asset, generate a list of text events based on the mapping parameters of the text candidate regions and the surface coordinate system in each frame of the trajectory.

[0196] Optionally, in another embodiment of the present invention, one implementation of step C1 specifically includes the following steps (steps C11 to C13):

[0197] Step C11: For each trajectory, take the frame in the trajectory that meets the third preset condition as the target frame.

[0198] The third preset condition includes, but is not limited to, whether the frame is the entry frame, termination frame, local maximum sharpness frame, whether the appearance difference between adjacent frames is greater than the difference threshold, or a sampled frame with a fixed step size interval. For example, the entry frame, termination frame, local maximum sharpness frame, frame with the appearance difference between adjacent frames greater than the difference threshold, or sampled frame with a fixed step size interval can be used as the target frame.

[0199] Step C12: For each target frame, the region of interest in the trajectory is mapped to the surface coordinate system through the mapping parameters of the text candidate region in the target frame to obtain the first surface domain image.

[0200] The region of interest (ROI) in the trajectory is the region of interest (ROI) in the original video frame within the frame range covered by the trajectory.

[0201] Step C13: Perform text detection on the first surface domain image to obtain candidate text sequences, and perform dynamic programming on the candidate text sequences to generate a list of text events for the trajectory.

[0202] Optionally, in the actual application of the present invention, after performing text detection (such as optical character recognition (OCR)) on the first surface domain image to obtain multiple candidate texts and corresponding confidence scores for each frame, character normalization, symbol standardization, number format unification, and noise character filtering can also be performed, which is not limited here.

[0203] Optionally, in another embodiment of the present invention, in order to avoid misjudgment caused by relying solely on OCR confidence, the following are considered simultaneously during temporal decoding: the recognition confidence of the current candidate text itself, the magnitude of changes between candidate texts in previous and subsequent frames, and whether the evidence of image changes is consistent with the text changes. This is not limited here.

[0204] Among them, evidence of image changes can be used The adjacent frames are aligned to the surface coordinate system; then, an evidence area for observing background changes is constructed, avoiding suspected stroke areas and foreground occlusion areas as much as possible; finally, the brightness difference, structural difference, or texture difference between adjacent frames is calculated within this area to determine whether the image has undergone a real change. It should be noted that the purpose of this design is not to directly output text results, but to focus the evaluation on the background and structural areas that better reflect changes in screen switching, page turning, and scoring.

[0205] In the practical application of this invention, not only can candidate text sequences be dynamically planned, but the shortest path solution method can also be used to obtain the optimal stable text timeline on the entire trajectory.

[0206] If the optimal stable text remains unchanged for several consecutive frames and the evidence of image change remains low, a stable interval is formed; if the optimal stable text changes and the evidence of image change is high for several consecutive frames, it is determined to be a real change point.

[0207] Optionally, in another embodiment of the present invention, if a real change point is detected, the original trajectory is not directly deleted, but the original trajectory is split into parent and child event segments that share the primary key of the trajectory number (track_id) but have different text event numbers (event_id), so that the same spatial text can be divided in time but can still be traced in space.

[0208] Optionally, in another embodiment of the present invention, if the stable text of the trajectory does not meet the minimum stable segment length configuration, the stable text is discarded.

[0209] In the practical application of this invention, the text event list (TextEventList) may include, but is not limited to, trajectory number, text event number, trajectory start time, trajectory end time, stable text, change type, trajectory splitting reason, etc., wherein the change type is divided into OCR jitter, real change, page turning change, points change, etc., which are not limited here.

[0210] This invention uses TrackAsset and TextEventList to jointly determine stable text, real change points, and surface coordinates through multi-frame observations within the trajectory, thus distinguishing it from existing technologies that only complete editing on a certain reference frame and then propagate the results to other frames.

[0211] Step C2: Based on the trajectory assets, the list of text events, and the set of text candidate regions, determine the target pattern corresponding to each trajectory.

[0212] It should be noted that this invention does not simply label each trajectory, but rather calculates the feasibility of the atlas path and the template path for the same trajectory, and allows re-evaluation at each text event boundary. In this way, target mode selection is no longer a one-time action, but a decision that can change over time.

[0213] Optionally, in another embodiment of the present invention, one implementation of step C2 specifically includes the following steps (steps C21 to C23):

[0214] Step C21: Determine the feasibility of the atlas path based on the approximate planar stability, the degree of cross-frame reuse of background texture, the estimated proportion of missing areas on the base plate, and the refresh rate.

[0215] Among them, the approximate planar stability can be determined by the trajectory asset, specifically by the stability of the reference boundary defined in the surface coordinate system of the trajectory asset. The reprojection residual, the proportion of effective feature points, and the proportion of continuously visible frames are jointly determined, and no restrictions are imposed here.

[0216] The degree of background texture reuse across frames can be determined by the proportion of reusable background pixels after the generation of the first surface domain image, and is not limited here.

[0217] Estimated missing area ratio of the base plate (MissingRate) est The ratio of reusable background pixels, the pixel occlusion ratio, and the area of ​​insufficient observation can be determined, and no limit is imposed here.

[0218] The refresh rate (RefreshFreq) can be determined by the number of event switching times per unit time in the text event list, the length of the stable segment, and the actual change point; no restrictions are imposed here.

[0219] In the practical application of this invention, the feasibility of the atlas path can be calculated using the following formula. :

[0220] ;

[0221] Among them, weight , , , This can be pre-set and modified by experts and technicians, and is not limited here. Specifically: the feasibility of the atlas path increases with the increase of approximate planar stability and the degree of cross-frame reuse of background textures, and decreases with the increase of the estimated proportion of missing areas of the base plate and the refresh rate.

[0222] Step C22: Determine the feasibility of the template path based on the strength of the template structure evidence and the refresh frequency.

[0223] The strength of the template structure evidence is determined by the structural information in the set of candidate text regions. The structural information includes, but is not limited to, the consistency score of the screen border (UIFrame), the consistency score of the UI layout (UILayout), and the consistency score of the regular rectangle (RectConsistency), which are not limited here.

[0224] In the practical application of this invention, the feasibility of the template path can be calculated using the following formula. :

[0225] ;

[0226] Among them, weight , , , The settings can be pre-configured and modified by experts and technicians, and are not limited here. Specifically, the feasibility of the template path increases with the strength of the template structure evidence and the refresh frequency.

[0227] Step C23: Determine the target pattern corresponding to the trajectory based on the feasibility of the atlas path and the feasibility of the template path.

[0228] In the practical application of this invention, two feasibility thresholds can be set, but are not limited to: when the feasibility of the atlas path is greater than the atlas path feasibility threshold, the target mode corresponding to the trajectory is determined to be the atlas mode; when the feasibility of the template path is greater than the template path feasibility threshold, the target mode corresponding to the trajectory is determined to be the template mode; if the feasibility of the atlas path is not greater than the atlas path feasibility threshold and the feasibility of the template path is not greater than the template path feasibility threshold, the target mode corresponding to the trajectory is determined to be the backtrack mode.

[0229] Optionally, in another embodiment of the invention, it can also be combined with and This is used to determine the target pattern corresponding to the trajectory. For example, if the template path feasibility is greater than the template path feasibility threshold, and If the refresh rate is greater than the refresh rate threshold, then template mode is preferred; if the atlas path feasibility is greater than the atlas path feasibility threshold and If the ratio is less than the missing percentage threshold, then the atlas mode is preferred; otherwise, the rollback mode is selected.

[0230] Optionally, in another embodiment of the present invention, the above score is recalculated every time an event_id boundary is encountered, because the same trajectory may be a real texture text in the first half and may become a mobile phone screen refresh or severe occlusion in the second half.

[0231] It should be noted that the rollback mode in this invention does not mean "returning to the previous computing node to recalculate," but rather refers to directly entering the conservative delivery branch when the main replacement link does not meet the deliverable conditions. For example, it outputs an executable delivery plan (e.g., retaining the original text + translator's notes, transcribing the text, and using bubble annotations), and provides a reason code and evidence summary, such as:

[0232] 1. TRACK_UNSTABLE: Trajectory association conflict or unstable surface fitting;

[0233] 2. PLATE_INCOMPLETE: The missing area of ​​the base plate is too large;

[0234] 3. UI_SKELETON_LOW: Insufficient evidence of screen / UI skeleton;

[0235] 4. LAYOUT_INFEASIBLE: No feasible solution is expected for the layout;

[0236] 5. OCR_READBACK_FAIL: Unstable historical readback performance;

[0237] 6. TEXTURE_MISMATCH: High risk of inconsistent texture.

[0238] S104. Process the trajectory based on the processing method corresponding to the target pattern to obtain the target asset of the trajectory.

[0239] In practical applications of this invention, for scenarios involving realistic texture text, the invention uses an atlas mode to generate atlas assets and base assets; for structured UI scenarios such as screens, scoreboards, and packaging, the invention uses a template mode to generate template assets and only redraws text nodes. This invention's off-processing helps reduce the risk of compositing artifacts and structural misalignment in screen-like scenarios.

[0240] Optionally, in another embodiment of the present invention, if the target mode is an atlas mode and the target asset includes atlas assets and base plate assets, one implementation of step S104 specifically includes the following steps (steps D1 to D3):

[0241] Step D1: Map the entire trajectory to the surface coordinate system using the mapping parameters of the text candidate regions in each frame of the trajectory to obtain the second surface domain image.

[0242] Step D2: Determine the set of missing pixels in the second surface domain image, and repair the missing areas indicated by the set of missing pixels to obtain the surface texture atlas.

[0243] In the practical application of this invention, the missing pixel set of the second surface domain image can be determined by taking the observed values ​​in the second surface domain that meet the observation conditions as valid observed pixels.

[0244] Here, the observed values ​​are the valid image information about pixel u obtained from different frames in the surface domain. Valid image information typically includes the pixel's color value, depth information, texture features, etc., which are not limited here.

[0245] The observation conditions include, but are not limited to, whether pixel u is in the mask of the occluded part of the foreground region, whether pixel u is located in the support region of the suspected text layer, and whether the sharpness and reprojection residual of the frame to which pixel u belongs meet the corresponding thresholds.

[0246] Specifically, pixels that simultaneously meet the following criteria are considered valid observation pixels: not in the mask of the occluded part of the foreground area, not located in the support area of ​​the suspected text layer, and whose sharpness and reprojection residual of the frame meet the corresponding threshold.

[0247] In the practical application of this invention, after determining the effective observation pixels, sampling weights are also assigned to the effective observation pixels. The sampling weights can be determined comprehensively based on the sharpness, occlusion, noise level and reprojection stability of the frame.

[0248] First, a weighted median or weighted average is calculated based on the sampling weights to obtain candidate colors; then, outlier observations that deviate too much from the median are removed. If the number of valid observations for pixel u is less than the threshold Nmin, or the remaining observations after outlier detection are still insufficient, or the observation variance is higher than the threshold, then u is marked as a missing pixel and written to MissingMap(u)=1.

[0249] In the practical application of this invention, for the missing areas indicated by the MissingMap, texture completion or generative repair is performed only within those areas to obtain a surface texture atlas.

[0250] Alternatively, in another embodiment of the present invention, during the repair process, the texture direction, brightness continuity and boundary smoothness of adjacent non-missing pixels can be constrained, and the complete observable area is not regenerated.

[0251] Step D3: Back-project the surface texture atlas onto the surface coordinate system to obtain the base plate asset, and generate the atlas asset based on the surface texture atlas.

[0252] Among them, the base plate asset serves as the initialization for the subsequent "base plate-text layer bidirectional inversion" to avoid drift caused by directly fitting texture on a low-quality base plate; at the same time, after the text layer is obtained through inversion, the text layer can be controllably removed from the original video and fed back to update the surface texture atlas, so that the base plate reduces residual text artifacts and improves texture consistency round by round.

[0253] Atlas assets are used to record surface texture atlases and their correspondence with surface coordinate systems and original video frames. Specifically, surface texture atlases, missing region markers, pixel confidence levels, surface coordinate system definitions, and mapping relationships with original video frames can be encapsulated into atlas assets for direct reference in subsequent rendering recipe inversion, back projection compositing, and self-checking processes. No restrictions are imposed here.

[0254] In the practical application of this invention, the final output atlas assets and baseplate assets may include, but are not limited to, atlas numbers, atlas images, missing region markers, pixel confidence scores, baseplate frame references, and overall baseplate confidence scores, which are not limited here. The overall baseplate confidence score can be obtained by combining the MissingMap proportion, pixel confidence score distribution, and backprojection consistency, which are not limited here. Pixel confidence scores can be obtained through Bayesian convolutional neural networks, Monte Carlo sampling, etc., which are not limited here. The confidence score distribution per pixel can be generated based on the number of effective observations and observation consistency; that is, the more times the same location is reliably observed and the more consistent the observations are, the higher its confidence score, which is not limited here.

[0255] Optionally, in another embodiment of the present invention, if the target mode is a template mode and the target asset includes template assets, one implementation of step S104 specifically includes the following steps (steps E1 to E4):

[0256] Step E1: Based on the ratio between the screen boundary and the original video frame boundary, map the region of interest of the original video frame to the screen coordinate system.

[0257] In the actual application of this invention, the screen boundary can be estimated by the screen border, the regular rectangular boundary, the fixed UI dividing line, and the perspective consistency, and is not limited here.

[0258] It should be noted that in practical applications, it is not required to directly detect four ideal corner points; when the corner points are incomplete, they can be supplemented by edge line fitting and historical frame stabilization results, which is not limited here.

[0259] The region of interest (ROI) in the original video frame can be obtained either manually specified by the user or automatically detected by an algorithm. Automatic detection methods include, but are not limited to, combining threshold segmentation, edge detection, motion detection (such as optical flow), and deep learning methods.

[0260] Step E2: Decompose the content in the screen coordinate system to obtain the text layer, and bind the text events in the text event list to the corresponding text slots in the text layer.

[0261] Specifically, within a unified screen coordinate system, the layout can be broken down into a background layer, a control layer, and a text layer, based on color blocks, boundaries, fixed control templates, or UI layout rules. No specific limitations are imposed here. The background layer consists of a stable background color, gradient, and non-interactive texture over a long period; the control layer comprises stable primitives such as button boundaries, separator lines, fixed icon positions, speech bubbles, and table frames; and the text layer consists of text nodes that refresh with event changes and can be redrawn individually.

[0262] The background layer and control layer can be obtained through time median plot, color block clustering, edge response and connectivity stability; the text layer includes multiple text slots text_slots[], which can be given by the connectivity of the text region, OCR box clustering and slot boundary rules, and are not limited here.

[0263] In another embodiment of the present invention, for each text event, the text event is bound to the corresponding text_slots[] based on the text slot and the time range, spatial overlap, and order of text changes. The present invention ensures that "only the text node is modified, not the control layer" in high-frequency refresh scenarios by binding the text events in the text event list to the corresponding text slots in the text layer.

[0264] Step E3: Generate time-varying modulation terms based on the dynamic features of the screen.

[0265] The dynamic characteristics of the screen include, but are not limited to, dynamic characteristics such as compression noise, scan lines or moiré patterns, and brightness fluctuations, which are not limited here.

[0266] It should be noted that the scan line can be obtained by estimating the peak value and phase of the horizontal spectrum, and the brightness fluctuation can be obtained by the global brightness or local slot brightness curve of each frame. There are no restrictions here.

[0267] Step E4: Generate template assets based on the bound text slots, time-varying modulation items, and screen boundaries.

[0268] The template mode in this invention can abstract screen content into a "UI skeleton" (e.g., control boundaries, fixed icon positions, speech bubbles / table boxes, background color blocks, and separator lines), and output it in the form of editable assets (control layer primitives + text layer placeholders + style parameters). When text is refreshed frequently, only the text nodes are redrawn and the control layer is reused, which can significantly reduce the risk of screen compositing artifacts caused by frame-by-frame patching.

[0269] S105. Based on the target asset and trajectory asset, generate translated and replaced video frames.

[0270] Optionally, in another embodiment of the present invention, one implementation of step S105 includes the following steps (steps F1 to F3):

[0271] Step F1: Generate a rendering recipe based on the target assets and text event list of the trajectory.

[0272] The render recipe includes at least static parameters and time-varying parameters.

[0273] It should be noted that this invention writes the text texture in the original video into executable parameters, rather than simply overlaying the translated text directly onto the background or template.

[0274] In practical use of this invention, in atlas mode, observations are aligned based on the base asset; in template mode, observations are aligned based on the control layer and background layer provided by the template asset. Unlike existing schemes based on propagation of a single reference frame, this invention jointly estimates static and time-varying parameters over the entire event sequence, and then simultaneously cleans up the base and text layers through alternating optimization.

[0275] Specifically, an alternating optimization approach can be to first fix the Plate. t (Estimate the current baseboard or template background), then fit the rendering recipe for the current event, and inversely calculate the text layer. Remove from the original video This yields cleaner frame-by-frame background candidates. Cross-frame fusion is then re-performed on these candidates to update the Plate. t Repeat the above process until the change in residual between two adjacent rounds is lower than the convergence threshold, or the maximum number of iterations is reached.

[0276] This invention optimizes and updates the base and text layer formulas alternately, resulting in a cleaner base and more accurate formulas. Generative repair is limited to a small area indicated by the MissingMap, thus helping to reduce texture breaks and edge anomalies caused by large-area repairs.

[0277] Optionally, in another embodiment of the present invention, one implementation of step F1 specifically includes the following steps (steps F11 to F13):

[0278] Step F11: Construct a text support area based on the target assets and text event list of the trajectory.

[0279] In the practical application of this invention, the text support area can be constructed in the following ways, but is not limited to: in atlas mode, based on the text mask information in the trajectory asset, the reference boundary of the surface coordinate system, and the time range of the corresponding text event in the text event list, the unoccluded text areas of multiple frames are mapped and aggregated into the same surface coordinate system to obtain the text support area; in template mode, the text slots bound to the event in the template asset are directly used, and the stable coverage range of the text in the text slots is combined to form the text support area.

[0280] In the practical application of this invention, a parametric renderer can also be constructed in the text support area. The parametric renderer includes at least executable items such as fill, stroke, shadow, glow, grain and blur, which are not limited here.

[0281] Step F12: Within the text support region and its neighborhood, residual observation is performed by calculating the residual between the original video frame and the estimated background image to obtain the residual observation results.

[0282] In the atlas mode, the specific method for residual observation can be to compare the aligned original video frame with the current base or template background estimate. The difference is mainly statistically analyzed in the text support area or its expanded neighborhood, avoiding the area indicated by the foreground occlusion mask. In the template mode, the scan lines, compressed noise and brightness fluctuations recorded in the template asset can be used as the initial observation of time-varying features, which is not limited here.

[0283] Step F13: Group the parameters based on the residual observation results to obtain static parameters and time-varying parameters.

[0284] Specifically, a set of candidate texture quantities is first extracted from the residual observations, such as fill color, stroke thickness, shadow offset, basic luminous radius, brightness gain, reflection intensity, blur direction, scan line phase, and grain amplitude, etc., without limitation here.

[0285] The candidate texture values ​​are further categorized based on their fluctuations across most unoccluded frames within the same event. Specifically, if a candidate texture value changes little within the event range and remains largely consistent across different frames, it is classified into the static parameter group, such as fill primary color, stroke thickness, shadow offset, and basic luminous radius. If a candidate texture value changes continuously over time, and this change can be continuously observed by brightness residual, spectral phase, or local blur direction changes, it is classified into the time-varying parameter group, such as brightness gain, reflection intensity, blur direction, scan line phase, and grain amplitude.

[0286] Optionally, in another embodiment of the present invention, after completing the parameter grouping, the current base plate or template background can be fixed first to estimate the static parameters; then, the time-varying parameters can be fitted in the form of a low-dimensional time curve, for example, using a piecewise linear curve, spline curve, or other low-dimensional time base to describe the changes in brightness gain, reflection intensity, and scan line phase over time. Furthermore, to avoid parameter jitter, continuity and amplitude limitations can be imposed on the time curve.

[0287] In the practical application of this invention, the rendering recipe may also include recipe reliability. The recipe reliability can preferably be determined by convergence, residual consistency, and parameter stability, but this is not limited here. Specifically, convergence can be judged by whether the change in residual between two adjacent rounds continues to decrease. For example, if the improvement after multiple iterations is already very small, then the current solution is considered to have basically converged, but this is not limited here.

[0288] Step F2: Determine the layout scheme based on the rendering recipe, text event list, and layout constraints.

[0289] Among them, the layout constraints include, but are not limited to, whether the layout completely falls within the boundary of the target area or text slot corresponding to the current event, whether it enters the main safe area, whether the font size is lower than the minimum font size, and whether the number of lines exceeds the preset upper limit. These are not limited here.

[0290] Optionally, in another embodiment of the present invention, one implementation of step F2 specifically includes the following steps (steps F21 to F24):

[0291] Step F21: Identify key factual information in the stable text of the text event list.

[0292] Key factual information includes, but is not limited to, names of people, place names, numbers, units of measurement, scores, and project terminology, etc., which are not limited here.

[0293] It should be noted that the key factual information in this invention is not allowed to be rewritten in the subsequent generation of controlled variants. The key factual information can be written into the lock set, which is not limited here.

[0294] Step F22: Generate controlled variants based on the variant generation strategy, key factual information, and stable text in the text event list.

[0295] The variant generation strategies include, but are not limited to, generating translation candidates from multiple machines, terminology substitution and standard abbreviation rules, shorter synonym templates, and localized format conversion of numbers and units. These are not limited here.

[0296] Step F23: For each controlled variant, generate candidate layouts by combining the typographic constraints and the visual readability requirements in the rendering recipe.

[0297] In the practical application of this invention, initial candidate templates can be generated through pre-screening conditions. The pre-screening conditions include, but are not limited to, joint search font size, line break mode, character spacing, stroke width, alignment mode, and top and bottom white space, to form initial candidate layouts.

[0298] Then, if the initial candidate layouts meet the layout constraints and the visual readability requirements in the rendering recipe, the final candidate template is determined. The visual readability requirements in the rendering recipe include, but are not limited to, minimum contrast and stroke readability compatibility.

[0299] For example, the initial candidate layout that meets the following conditions will be used as the final candidate layout: it must fall completely within the boundary of the target area or text slot corresponding to the current event, not enter the main safe area, have a font size no smaller than the minimum font size, have a number of lines not exceeding the preset upper limit, and meet the minimum contrast and stroke readability requirements given by RenderRecipe.

[0300] In atlas mode, the target region boundary is selected from the reference boundary in the surface coordinate system of the trajectory asset or the target boundary obtained by the stable envelope of multiple text masks; no specific limitation is made here. In template mode, the text slot boundary already bound in the template resource is selected, and the allowed inner margin, maximum number of lines, and font size range of that slot are inherited. In this invention, the region boundary used for subsequent typesetting searches has a clear source, rather than being reassigned out of thin air.

[0301] Optionally, in another embodiment of the present invention, if none of the initial candidate layouts meet the layout constraints and the visual readability requirements in the rendering recipe, the reason code is output according to the constraint that failed first.

[0302] Optionally, in another embodiment of the present invention, corresponding degradation suggestions may also be output, such as enabling shorter variants, transliteration, retaining the original text + translator's notes, or entering fallback mode.

[0303] Step F24: Generate a layout scheme based on all candidate layouts.

[0304] In the practical application of this invention, a feasibility score can be calculated for each candidate layout to ultimately determine the layout scheme. The specific template feasibility score can be based on the remaining width and height margin of the target area, the main safe area margin, the contrast margin after compatibility with RenderRecipe constraints, line break balance, and the center offset of the text within the target area. It should be noted that a higher template feasibility score indicates that the candidate layout is more reliable without sacrificing readability.

[0305] Optionally, in another embodiment of the present invention, the layout plan may include, but is not limited to, layout plan number, target text (translated text), selected controlled variant, font size, line break mode, character spacing, stroke width, target area boundary, layout feasibility score, and failure reason code, which are not limited here.

[0306] Step F3: Based on the rendering recipe, layout scheme, and mapping parameters of the text candidate regions in each frame of the trajectory, generate the translated and replaced video frames.

[0307] Optionally, in another embodiment of the present invention, one implementation of step F3 specifically includes the following steps (steps F31 to F33):

[0308] Step F31: Render the text based on the typesetting scheme within the standard coordinate system to obtain the translated text layer.

[0309] The standard coordinate system is either the surface coordinate system or the screen coordinate system.

[0310] In the practical application of this invention, one embodiment of step F31 can be as follows: First, determine the origin position, coordinate axis direction (e.g., in screen coordinate systems, the origin is often the upper left corner, the X-axis is to the right, and the Y-axis is downward), and coordinate units (pixels, logical units, etc.) of the standard coordinate system. Then, based on the target text and layout information (e.g., font size, line break mode, character spacing, stroke width, target area boundary, etc.), calculate the bounding box size of the text layer to ensure that the text is displayed completely within the coordinate system without overflow or truncation. If there are multiple lines of text, calculate the starting coordinates of each line to ensure that the spacing between lines meets the set line spacing requirements. Simultaneously, consider automatic line wrapping logic; when text exceeds the width of the coordinate system, wrap appropriately and adjust the coordinates of subsequent lines. Finally, render the calculated text content within the standard coordinate system according to the determined coordinates and layout style to obtain the translated text layer. .

[0311] Step F32: Perform visual enhancement processing on the translated text layer based on the rendering recipe to obtain the target translated text layer.

[0312] Specifically, according to the rendering formula, the translated text layer is overlaid with fill, outline, shadow, glow, grain, time-varying modulation items, etc., which are not limited here.

[0313] Step F33: Project the target translation text layer onto the original video frame using the mapping parameters of the text candidate region in each frame of the trajectory to obtain the translated and replaced video frame.

[0314] In the practical application of this invention, when there are situations where the character's hands, face, or prop edges cross the text area, it is necessary to overlay a foreground occlusion layer.

[0315] Specifically, the difference between the original video and the "base / template background + rendered text" is first calculated within the surface domain. Then, an occlusion mask and its transparency are generated by combining the foreground edge and temporal continuity. Next, layer compositing is performed in the order of "background, new text, foreground occlusion layer," without any restrictions. This prevents new text from passing through the foreground when characters or props occlude text.

[0316] Optionally, in another embodiment of the present invention, in order to avoid frame-by-frame changes in color, stroke thickness and contrast, time smoothing can also be performed on key parameters, such as applying exponential smoothing or spline constraints to color gain, stroke width and brightness modulation curves, and requiring that the parameter changes in adjacent frames do not exceed a preset range, which is not limited here.

[0317] Optionally, in another embodiment of the present invention, an operation package (ResultPackage) can be generated based on the rendering result frame reference, compositing information, occlusion processing information, the rendering recipe used, and the typography scheme used, where "reference" refers to the pointer or association relationship of data.

[0318] It should be noted that the compositing information includes, but is not limited to, reprojection and compositing order, and the occlusion processing information records whether a foreground occlusion layer is superimposed and the corresponding mask reference. Among them, reprojection refers to mapping the translated text layer in the normal coordinate system back to the original video frame according to the mapping parameters of the current frame, so as to complete the layered compositing with the background layer, control layer and foreground occlusion layer.

[0319] Optionally, in another embodiment of the present invention, the ResultPackage can also be read back, self-checked, and risk assessed; if successful, delivery and audit materials are output; if unsuccessful, it is automatically retried in a fixed tier, and after multiple failures, it enters a rollback mode and outputs retry records, rollback reason codes, evidence summaries, and suggested manual intervention points.

[0320] Specifically, several keyframes are first extracted from each event_id and OCR readback is performed. The readback results are then compared with the target text to obtain the matching hit rate. It should be noted that the hit rate here is not a general similarity score, but can be a character-level hit rate, a word-level hit rate, or a combination thereof. If the readback fails, error cause labels are preferably generated based on the failure mode, such as: MISS_GLYPH: missing character shape or stroke; LINEBREAK_WRONG: line break error; LOW_CONTRAST: insufficient contrast; OVERFLOW_CLIP: region overflow or cropping; TEMPORAL_DRIFT: temporal drift causing inconsistencies in the readback of adjacent frames.

[0321] Optionally, in another embodiment of the present invention, stability and consistency indices can also be calculated in the surface domain or screen domain, such as: flicker index, used to measure the fluctuation of text brightness, edge intensity or stroke width in adjacent frames; edge stitching anomaly, used to measure the degree of discontinuity of text boundary and neighboring background in gradient; template consistency, used to measure the structural difference between screen control layer and template skeleton; texture consistency, used to measure whether noise statistics, scan line modulation and brightness fluctuation are consistent with the original video, which is not limited here.

[0322] Optionally, in another embodiment of the present invention, a comprehensive risk assessment result can be generated based on the above indicators. If any hard threshold is not met, or the comprehensive risk level exceeds the threshold, it is considered a high-risk result.

[0323] Optionally, in another embodiment of the invention, the failure handling actions can be performed in the following order:

[0324] 1. If the layout or controlled variant is changed, the layout scheme is re-determined based on the rendering recipe, text event list, and layout constraints;

[0325] 2. If the recipe is re-evaluated or the outline is enhanced, a new rendering recipe is generated based on the target assets and text event list of the trajectory, and the layout scheme is re-determined after the rendering recipe is updated;

[0326] 3. Switch between gallery mode and template mode;

[0327] 4. If it still fails, it will enter rollback mode and output a conservative delivery plan.

[0328] In the practical application of this invention, the final output can be an audit log and a self-check result, which at least include: the inspection scope number, trajectory number, text event number, pattern result, self-check indicator, error cause, action, final cause code, and related asset references, without limitation here. This invention uniformly writes the pattern selection basis, base / template confidence level, formula parameters, layout result, self-check indicator, and action path into the audit log and self-check result, facilitating subsequent review, evidence presentation, and cross-language reuse.

[0329] The following is an example illustrating the practical application of this invention, taking text within a display such as "shop signs / road signs / posters / title cards" affixed to a real textured surface as an example:

[0330] 1. Corresponding to S101: Extract the first frame, middle frame and last frame as basic keyframes in the shot clip, and encrypt the sampling keyframes near the frames with large optical flow changes; detect text candidate regions on these keyframes, extract text detection scores, rectangular border strength and background texture complexity, and generate CandidateSet.

[0331] 2. Corresponding to S102: Associate the same text candidate region as a trajectory in consecutive frames, and fit and smooth the perspective mapping parameters of each frame. Output mask information and frame-by-frame quality curves; when the trajectory is unstable or there are multiple target conflicts, write the reason code TRACK_UNSTABLE and shorten the trajectory frame interval.

[0332] 3. Corresponding to step C1: Normalize, cluster and temporally decode the OCR candidate sequence in the surface coordinate system, output a list of text events, and distinguish between "identification jitter" and "real refresh / change"; for example, stabilize "CAFE", "CAFF" and "CAFE" in consecutive frames into the same stable text event.

[0333] 4. Corresponding to step C2: Calculate the feasibility of the atlas mode path based on the reference boundary stability and reprojection residual output by S102, the refresh frequency output by step C1, and the reusable proportion of the background pixels after alignment; when the feasibility of the atlas path is higher than that of the template path and the estimated missing proportion is low, determine to prioritize entering the atlas mode.

[0334] 5. Processing method corresponding to the atlas mode (steps D1 to D3): Align the trajectory region to the surface coordinate system, and obtain the atlas asset by "visibility voting + clarity weight + outlier suppression", and generate the MissingMap at the same time; only perform bounded completion on the missing regions marked by the MissingMap to obtain the base plate asset CleanPlate and the base plate confidence Conf_plate.

[0335] 6. Corresponding to step F1: First, determine the text support area based on the text event list and the surface coordinate system. Then, based on the residual of "original image - base plate", classify the low-fluctuation texture quantity into the static parameters, classify the texture quantity that changes continuously with time into the time-varying parameters, and generate the time-varying parameter modulation curve to obtain the rendering recipe. Iterate and update the base plate and rendering recipe in an alternating optimization manner. When the convergence is insufficient, output TEXTURE_MISMATCH or PLATE_INCOMPLETE.

[0336] 7. Corresponding to step F2: First, determine the typographic range by the boundary of the target area in the surface coordinate system, then translate the stable text, lock proper names and numbers, generate multiple controlled variants and solve the typography scheme; if all schemes exceed the sign area, output LAYOUT_OVERFLOW and try shorter variants.

[0337] 8. Corresponding step F3: Render the translated text layer according to the rendering recipe in the surface coordinate system, and use... Project back to the original video; when characters or props obscure text in the foreground, perform layered composition by combining foreground occlusion masks and edge continuity to prevent new text from passing over the foreground.

[0338] 9. Self-test scheme in the corresponding embodiment: Perform OCR readback, blink / stitching / texture consistency and other indicators, and output AuditLog (including key asset reference version, indicators and reason code); if the self-test fails, automatically retry or rollback according to the handling ladder.

[0339] The following is an example illustrating the practical application of this invention, using "mobile phone screen chat bubbles (structured UI)" as an example:

[0340] 1. Corresponding to S101: The mobile phone screen boundary and chat bubble text area were detected in the same shot segment. The type was initially judged as "screen / UI", and scan line response, regular rectangle response and interface layout evidence were written into CandidateSet.

[0341] 2. Corresponding to S102: Associate the chat bubble with the same trajectory in consecutive frames, fit and smooth the mapping parameters for each frame; at the same time, estimate the foreground occlusion area based on the character's hand movements.

[0342] 3. Corresponding to step C1: Perform text stabilization in the surface coordinate system to obtain a list of text events, such as the bubble text remaining stable for a period of time and then being refreshed after a period of time.

[0343] 4. Corresponding to step C2: Calculate the feasibility of the template path based on the evidence of "screen border / grid features / UI layout", event refresh frequency and template slot stability; when the feasibility of the template path is higher than that of the atlas path, select the template mode and generate the corresponding evidence summary.

[0344] 5. Processing method corresponding to the template mode (steps E1 to E4): Decompose the control layer and text layer placeholders in the screen coordinate system to generate template assets. The UI skeleton records the bubble box and fixed icon position, and text_slots[] records the chat bubble text slots.

[0345] 6. Corresponding to step F1: Determine the text support area using the slot boundary of TemplateAsset, extract the observations that change over time, such as scan lines, compressed noise, and brightness fluctuations, from the original screen, write the stable observations into static parameters, write the continuously changing observations into time-varying modulation curves, and form a rendering recipe together with the control layer.

[0346] 7. Corresponding to step F2: Using the template slot boundary as the target area, solve for the feasible region of the target language translation; if there is no solution, first enable the controlled variant, and then output the LAYOUT_INFEASIBLE or FONT_TOO_SMALL reason code.

[0347] 8. Corresponding step F3: Redraw the translated text layer in the screen coordinate system, composite it with the control layer, and project it back to the original video; when a finger or figure outline passes in front of the phone, overlay a foreground occlusion layer.

[0348] 9. Self-test scheme in the corresponding embodiment: If the self-test items such as OCR readback, blink control and template consistency pass, the final output is a trace package, which includes a list of text events, mode selection reason code, template / recipe reference, self-test indicators and rollback delivery scheme.

[0349] This invention does not directly output replacement results when the self-test fails, but instead outputs an "executable handling path" and records the process in a structured manner.

[0350] 1. Triggering conditions: such as OCR_READBACK_FAIL (insufficient readback hit rate), TEXTURE_MISMATCH (abnormal texture consistency), LAYOUT_INFEASIBLE (unsolvable layout), UI_SKELETON_LOW (insufficient template skeleton confidence), etc.

[0351] 2. Automatic processing ladder: Prioritize low-risk adjustments within the same mode (such as changing the layout / controlled variant, enhancing the stroke, or tightening the time-varying curve amplitude); if it still fails, trigger "re-evaluate formula / switch mode"; if it still fails after multiple rounds, enter the rollback mode.

[0352] 3. Rollback Delivery: Output a deliverable solution (retaining the original text + translator's notes, subtitles, bubble annotations, etc.), and record the rollback reason code, evidence summary, and suggested human intervention points in the AuditLog to make delivery risks controllable and responsibility boundaries clear.

[0353] As can be seen from the above scheme, this invention provides a method for translating and replacing text within a video frame. This invention no longer treats text within a frame as independent single-frame detection boxes, but instead associates text candidate regions across frames in the original video frame sequence to generate trajectory assets. That is, single-frame candidate boxes (detection boxes) are transformed into reusable assets for consecutive frames. Then, based on the trajectory assets and the set of text candidate regions, the target pattern corresponding to each trajectory in the trajectory assets is determined, and the trajectory is processed according to the processing method corresponding to the target pattern to obtain the target asset of the trajectory. Finally, based on the target asset and the trajectory assets, the translated and replaced video frame is generated. This achieves the goal of generating translated and replaced videos efficiently and with high quality to meet film-level delivery requirements.

[0354] Another embodiment of the present invention provides a device for translating and replacing text within a screen, such as... Figure 2 As shown, it specifically includes:

[0355] The text candidate region determination unit 201 is used to acquire the original video frame sequence and determine the set of text candidate regions in each frame of the original video frame sequence.

[0356] The trajectory asset determination unit 202 is used to associate the text candidate regions in each frame of the original video frame sequence across frames to generate trajectory assets.

[0357] The trajectory assets include at least the mapping parameters and surface coordinate system of the text candidate regions in each frame of each trajectory.

[0358] The target pattern determination unit 203 is used to determine the target pattern corresponding to each trajectory in the trajectory asset based on the trajectory asset and the set of text candidate regions.

[0359] The target asset generation unit 204 is used to process the trajectory based on the processing method corresponding to the target pattern to obtain the target asset of the trajectory.

[0360] The video frame generation unit 205 is used to generate translated and replaced video frames based on the target asset and trajectory asset.

[0361] For details on the specific operation of the units disclosed in the above embodiments of the present invention, please refer to the corresponding method embodiments, such as... Figure 1 As shown, it will not be elaborated further here.

[0362] As can be seen from the above scheme, the present invention provides a device for translating and replacing text within a video frame. Instead of treating text within the frame as independent single-frame detection boxes, the present invention correlates the text candidate regions in each frame of the original video frame sequence across frames to generate trajectory assets. That is, single-frame candidate boxes (detection boxes) are transformed into assets that can be reused across consecutive frames. Then, based on the trajectory assets and the set of text candidate regions, the target pattern corresponding to each trajectory in the trajectory assets is determined, and the trajectory is processed according to the processing method corresponding to the target pattern to obtain the target asset of the trajectory. Finally, based on the target asset and the trajectory assets, the translated and replaced video frame is generated. This achieves the goal of generating translated and replaced videos efficiently and with high quality to meet film-level delivery requirements.

[0363] The functions described above in this document can be performed at least in part by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: Field-Programmable Gate Array (FPGA), Application-Specific Integrated Circuit (ASIC), Application-Specific Standard Product (ASSP), System on a Chip (SOC), Complex Programmable Logic Device (CPLD), and so on.

[0364] Another embodiment of the present invention provides an electronic device, comprising:

[0365] One or more processors.

[0366] A storage device on which one or more programs are stored.

[0367] When the one or more programs are executed by the one or more processors, the one or more processors implement the text translation and replacement method within the screen as described in the above embodiments.

[0368] Another embodiment of the present invention provides a computer storage medium storing a computer program, wherein when the computer program is executed by a processor, it implements the text translation and replacement method in the screen as described in the above embodiments.

[0369] In the context of this invention, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. Machine-readable media can include, but are not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0370] It should be noted that the computer-readable medium described above in this invention can be a computer-readable signal medium, a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this invention, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this invention, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.

[0371] The aforementioned computer-readable medium may be included in the aforementioned electronic device; or it may exist independently and not assembled into the electronic device.

[0372] Another embodiment of the present invention provides a computer program product, which, when executed, is used to perform the above-described method for translating and replacing text within a screen.

[0373] In particular, according to embodiments of the present invention, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of the present invention include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device, or installed from a storage device, or installed from a ROM. When the computer program is executed by a processing device, it performs the functions defined in the methods of the embodiments of the present invention.

[0374] Although the subject matter has been described using language specific to structural features and / or methodological logic, it should be understood that the subject matter defined in this invention is not necessarily limited to the specific features or actions described above. Rather, the specific features and actions described above are merely exemplary forms for implementing the invention.

[0375] While several specific implementation details are included in the foregoing discussion, these should not be construed as limiting the scope of the invention. Certain features described in the context of individual embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented individually or in any suitable sub-combination in multiple embodiments.

[0376] The above description is merely a preferred embodiment of the present invention and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of the invention is not limited to the specific combination of the above-described technical features, but also includes other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the inventive concept. For example, technical solutions formed by substituting the above-described features with technical features of the present invention (but not limited to) that have similar functions.

Claims

1. A method for translating and replacing text within a screen, characterized in that, include: Obtain the original video frame sequence and determine the set of text candidate regions in each frame of the original video frame sequence; The text candidate regions in each frame of the original video frame sequence are correlated across frames to generate trajectory assets; Based on the trajectory asset and the set of text candidate regions, determine the target pattern corresponding to each trajectory in the trajectory asset; The trajectory is processed based on the processing method corresponding to the target pattern to obtain the target asset of the trajectory; Based on the target asset and the trajectory asset, a translated and replaced video frame is generated.

2. The method for translating and replacing text within a screen according to claim 1, characterized in that, Determining the set of candidate text regions in each frame of the original video frame sequence includes: Perform shot boundary detection on each frame of the original video frame sequence to obtain a set of shot segments; For each shot in the shot clip set, determine the set of keyframes for that shot clip; For each keyframe in the set of keyframes, text detection is performed to obtain candidate text regions.

3. The method for translating and replacing text within a screen according to claim 1, characterized in that, The step of cross-frame association of text candidate regions in each frame of the original video frame sequence to generate trajectory assets includes: Associate the same text candidate region in consecutive frames of the original video frame sequence as a trajectory; For each trajectory, image feature information of each frame of the trajectory is extracted, and for each frame of the trajectory, the mapping parameters of the text candidate region in the frame are determined based on the image feature information of that frame. If the boundaries of the text candidate regions in N consecutive frames of the trajectory meet the first preset condition, the trajectory is taken as an approximate planar trajectory, and a surface coordinate system is constructed based on the approximate planar trajectory.

4. The method for translating and replacing text within a screen according to claim 3, characterized in that, The surface coordinate system constructed based on the approximate planar trajectory includes: The frames in the approximate planar trajectory that satisfy the second preset condition are used as reference frames; Based on the text candidate regions in the reference frame, a reference boundary is generated, and a rectangular plane is established based on the reference boundary; Each frame in the trajectory is mapped onto the rectangular plane to obtain the surface coordinate system.

5. The method for translating and replacing text within a screen according to claim 3, characterized in that, The step of determining the target pattern corresponding to each trajectory in the trajectory asset based on the trajectory asset and the set of text candidate regions includes: For each trajectory in the trajectory asset, a list of text events is generated based on the mapping parameters of the text candidate regions in each frame of the trajectory and the surface coordinate system; Based on the trajectory assets, the list of text events, and the set of text candidate regions, the target pattern corresponding to each trajectory is determined.

6. The method for translating and replacing text within a screen according to claim 5, characterized in that, For each trajectory in the trajectory asset, based on the mapping parameters of the text candidate region in each frame of the trajectory and the surface coordinate system, a list of text events is generated, including: For each trajectory, the frame in the trajectory that satisfies the third preset condition is taken as the target frame; For each target frame, the region of interest in the trajectory is mapped to the surface coordinate system using the mapping parameters of the text candidate region in the target frame, to obtain a first surface domain image; Text detection is performed on the first surface domain image to obtain a candidate text sequence, and dynamic programming is performed on the candidate text sequence to generate a list of text events for the trajectory.

7. The method for translating and replacing text within a screen according to claim 6, characterized in that, The step of determining the target pattern corresponding to each trajectory based on the trajectory asset, the text event list, and the text candidate region set includes: The feasibility of the atlas path is determined based on approximate planar stability, cross-frame reuse of background texture, estimated proportion of missing base plate areas, and refresh frequency. Specifically, the approximate planar stability is determined by the trajectory asset; the cross-frame reuse of background texture is determined by the proportion of reusable background pixels after generating the first surface domain image; the estimated proportion of missing base plate areas is determined by the proportion of reusable background pixels, pixel occlusion ratio, and under-observed areas; and the refresh frequency is determined by the number of event switches per unit time, the length of stable segments, and the actual change points in the text event list. The feasibility of the template path is determined based on the strength of the template structure evidence and the refresh frequency; wherein, the strength of the template structure evidence is determined by the structural information in the set of text candidate regions; Based on the feasibility of the atlas path and the feasibility of the template path, the target pattern corresponding to the trajectory is determined.

8. The method for translating and replacing text within a screen according to claim 5, characterized in that, If the target mode is a map atlas mode, the target asset includes map atlas assets and base plate assets. The process of processing the trajectory based on the processing method corresponding to the target mode to obtain the target asset of the trajectory includes: The trajectory is mapped to the surface coordinate system by the mapping parameters of the text candidate regions in each frame of the trajectory, to obtain the second surface domain image; The missing pixel set of the second surface domain image is determined, and the missing regions indicated by the missing pixel set are repaired to obtain a surface texture atlas; The surface texture atlas is back-projected onto the surface coordinate system to obtain the base plate asset, and an atlas asset is generated based on the surface texture atlas.

9. The method for translating and replacing text within a screen according to claim 5, characterized in that, If the target pattern is a template pattern, and the target asset includes template assets, then processing the trajectory based on the processing method corresponding to the target pattern to obtain the target asset of the trajectory includes: Based on the ratio between the screen boundary and the original video frame boundary, the region of interest of the original video frame is mapped to the screen coordinate system. The content in the screen coordinate system is decomposed to obtain a text layer, and the text events in the text event list are bound to the corresponding text slots in the text layer. Time-varying modulation terms are generated based on the dynamic features of the screen; Based on the bound text slots, the time-varying modulation items, and the screen boundaries, a template asset is generated.

10. The method for translating and replacing text within a screen according to claim 5, characterized in that, The process of generating translated and replaced video frames based on the target asset and the trajectory asset includes: Based on the target assets of the trajectory and the list of text events, a rendering recipe is generated; Based on the rendering recipe, the list of text events, and the layout constraints, a layout scheme is determined; Based on the rendering recipe, the layout scheme, and the mapping parameters of the text candidate regions in each frame of the trajectory, a translated and replaced video frame is generated.

11. The method for translating and replacing text within a screen according to claim 10, characterized in that, The rendering recipe includes static parameters and time-varying parameters. The process of generating the rendering recipe based on the target asset of the trajectory and the list of text events includes: A text support area is constructed based on the target asset of the trajectory and the list of text events; Within the text support region and its neighborhood, residual observation is performed by calculating the residual between the original video frame and the estimated background image to obtain the residual observation result; Based on the residual observation results, the parameters are grouped to obtain static parameters and time-varying parameters.

12. The method for translating and replacing text within a screen according to claim 10, characterized in that, The process of determining the layout scheme based on the rendering recipe, the text event list, and the layout constraints includes: Identify key factual information in the stable text within the aforementioned list of textual events; Based on the variant generation strategy, the key factual information, and the stable text in the text event list, a controlled variant is generated; For each of the controlled variants, candidate layouts are generated by combining the typographic constraints and the visual readability requirements in the rendering recipe; Based on all the candidate layouts, a layout scheme is generated.

13. The method for translating and replacing text within a screen according to claim 10, characterized in that, The process of generating translated and replaced video frames based on the rendering recipe, the typography scheme, and the mapping parameters of the text candidate regions in each frame of the trajectory includes: Within a standardized coordinate system, layout rendering is performed based on the aforementioned typesetting scheme to obtain the translated text layer; wherein, the standardized coordinate system is either a surface coordinate system or a screen coordinate system. Based on the rendering formula, the translated text layer is visually enhanced to obtain the target translated text layer. The target translation text layer is projected onto the original video frame using the mapping parameters of the text candidate region in each frame of the trajectory, resulting in the translated and replaced video frame.

14. A device for translating and replacing text within a screen, characterized in that, include: A text candidate region determination unit is used to acquire the original video frame sequence and determine the set of text candidate regions in each frame of the original video frame sequence. The trajectory asset determination unit is used to associate the text candidate regions in each frame of the original video frame sequence across frames to generate trajectory assets; wherein, the trajectory assets include at least the mapping parameters and surface coordinate system of the text candidate regions in each frame of each trajectory; The target pattern determination unit is used to determine the target pattern corresponding to each trajectory in the trajectory asset based on the trajectory asset and the set of text candidate regions; The target asset generation unit is used to process the trajectory based on the processing method corresponding to the target mode to obtain the target asset of the trajectory; The video frame generation unit is used to generate translated and replaced video frames based on the target asset and the trajectory asset of the trajectory.

Images