Display apparatus and display method for translating subtitles

Abstract

The application discloses a display device and a display method for translating subtitles. In a video picture translation scene, the display device acquires track data of picture text, the track data including a sequence of detection box positions and a sequence of text contents in continuous frame pictures; geometric change features and text content change features of the track data are extracted, a dynamic change type of the picture text is determined based on a preset sequence determination rule; according to a corresponding mapping strategy, the dynamic effect of the picture text is mapped to a translation subtitle across languages, animation configuration parameters are obtained, the translation subtitle carrying the dynamic effect is generated, and a display is controlled to be superimposed and rendered to a corresponding area of the video picture. The application can automatically identify multiple types of text dynamic modes and restore cross-language dynamic effects, and can improve the viewing immersion of the translation subtitle.

Description

Technical Field

[0001] This application relates to the field of display device technology, and in particular to a display device and a method for displaying translated subtitles. Background Technology

[0002] Display devices are intelligent devices capable of presenting a user interface and supporting user interaction. Taking smart TVs as an example, smart TVs are television products based on Internet application technology, equipped with open operating systems and chips, possessing open application platforms, enabling two-way human-computer interaction, and integrating multiple functions such as audio-visual, entertainment, and data to meet diverse and personalized user needs.

[0003] In video content playback and cross-language subtitle translation scenarios, display devices use Optical Character Recognition (OCR) technology to detect, recognize, and track embedded text within video frames. After translating the recognized text, the translated subtitles are overlaid and rendered onto the corresponding area of ​​the screen, achieving cross-language translation and presentation of text on the screen.

[0004] However, the aforementioned methods of presenting translated subtitles cannot adapt to the different dynamic presentation forms of text in video footage, resulting in a disconnect between the translated subtitles and the visual dynamic rhythm of the original text, making it difficult to restore the original visual effects of the video content and the user's immersive viewing experience. Summary of the Invention

[0005] This application provides a display device and a method for displaying translated subtitles to solve the problem that translated subtitles are difficult to restore the original visual effect of video content.

[0006] In a first aspect, this application provides a display device, comprising: The monitor is configured to display video feeds; A controller, coupled to the display, is configured to: In the scenario of translating the video frame, the trajectory data of the text in the video frame is obtained; the trajectory data includes the sequence of detection box positions and the sequence of text content in consecutive frames. Feature extraction is performed on the trajectory data to obtain the geometric change features of the detection boxes corresponding to the detection box position sequence and the text content change features corresponding to the text content sequence; Based on a preset order determination rule, the dynamic change type of the text on the screen is determined according to the text content change characteristics and the detection box geometric change characteristics; Based on the mapping strategy corresponding to the dynamic change type, the dynamic effect of the text on the screen is mapped across languages ​​to the translated subtitles to obtain the animation configuration parameters; The system generates translated subtitles with corresponding dynamic effects based on the animation configuration parameters, and controls the display to overlay and render the translated subtitles onto the corresponding area of ​​the video screen.

[0007] The above technical solution has the following beneficial effects or advantages: It can automatically recognize multiple types of text dynamic effects and complete cross-language restoration, enhancing the immersive experience of watching translated subtitles and solving the problems of being unable to classify the dynamic modes of text on the screen and losing the original animation effects in translated subtitles; moreover, it requires no additional model training, has extremely low computational overhead, and can adapt to the edge computing power constraints of display devices.

[0008] In some embodiments of this application, the preset order determination rule includes a first-stage determination rule, a second-stage determination rule, and a third-stage determination rule; The controller, based on a preset order determination rule, determines the dynamic change type of the text on the screen according to the text content change characteristics and the detection box geometric change characteristics, and is specifically configured as follows: The first-stage classification is performed by invoking the first-stage determination rule. If the first-stage classification fails to match the classification result, the second-stage judgment rule is invoked to perform the second-stage classification. If the second-stage classification fails to match the classification result, the third-stage judgment rule is invoked to perform the third-stage classification. If the third-stage classification does not match the classification result, then the dynamic change type is determined to be static occurrence type; The first stage determination rule includes performing prefix growth verification on the text in the image based on the text content change features. If the prefix growth verification passes, the dynamic change type is determined to be prefix growth type. The second stage determination rule includes performing area expansion determination based on the detection box geometric change features. If a preset area expansion condition is met, the dynamic change type is determined to be a physical scaling subclass under geometric change type, combined with the text content change features. The third stage determination rule includes performing length growth determination based on the text content change features. If a preset length growth condition is met, the dynamic change type is determined to be a lens zoom-out exposure subclass under prefix growth type or geometric change type, combined with the detection box geometric change features.

[0009] The above technical solution has the following beneficial effects or advantages: It adopts a three-stage progressive sequential judgment architecture, which filters in layers according to the logic of feature recognition from high to low and computational load from small to large. This ensures classification efficiency in mainstream scenarios while covering classification accuracy in edge scenarios. It can distinguish various dynamic types such as prefix growth, physical scaling, and camera zooming out, and adapts to the dynamic presentation scenarios of various screen texts in videos.

[0010] In some embodiments of this application, the controller performs prefix growth verification based on the text content change features, specifically configured as follows: Iterate through all target adjacent frame pairs whose text length increases in the text content sequence; The target adjacent frame pairs that satisfy the prefix relationship are marked as valid growing frame pairs, wherein satisfying the prefix relationship means that the text content of the later frame in the target adjacent frame pair is prefixed with the text content of the previous frame. Calculate the ratio of the number of effective growing frame pairs to the number of target adjacent frame pairs to obtain prefix consistency; Detect the number of characters incremented in a single step corresponding to the increase in the length of the text in the target adjacent frame pair; If the prefix consistency is greater than or equal to a preset ratio threshold, and the number of characters increased in a single step is less than or equal to a preset number of characters threshold, then the prefix growth verification is deemed successful.

[0011] The above technical solution has the following beneficial effects or advantages: By verifying the prefix growth feature using both the prefix consistency ratio and the number of characters increasing in a single step, we can distinguish between real typewriter animation and noise generated by OCR progressive recognition, reduce the misjudgment rate of prefix growth type, avoid misjudging recognition fluctuations as text animation, and thus improve the reliability of classification results.

[0012] In some embodiments of this application, the geometric change features of the detection frame include the area expansion ratio, aspect ratio change, and height increase of the detection frame corresponding to the text in the image, wherein the area expansion ratio is the ratio of the maximum area of ​​the detection frame to the minimum area of ​​the detection frame; the text content change features include the increase in the text length corresponding to the text in the image. The controller performs an area expansion determination based on the geometric change features of the detection box; if a preset area expansion condition is met, it determines the dynamic change type as a physical scaling subclass under the geometric change type, based on the text content change features, specifically configured as follows: When the area expansion ratio of the detection frame is greater than or equal to the preset area threshold, it is determined that the preset area expansion condition is met. If the preset area expansion condition is met, and the multiple by which the text length increases is less than the preset growth threshold, the physical scaling subclass determination is performed; The determination of the physical scaling subclass includes the following: if the aspect ratio change of the detection box is greater than or equal to a preset aspect ratio threshold, or if the height increase of the detection box is greater than or equal to a preset height threshold, then the dynamic change type is determined to be a physical scaling subclass.

[0013] The above technical solution has the following beneficial effects or advantages: By combining multi-dimensional geometric features such as area expansion, text stability, aspect ratio change, and height increase to determine the physical scaling effect, it can accurately distinguish between real physical scaling animation effects and ordinary area fluctuations, covering various film and television scenes such as camera zoom-in and text magnification, improving the accuracy of geometric animation effect recognition and avoiding false triggering of scaling animation effects.

[0014] In some embodiments of this application, the geometric change features of the detection box include the increase in the width of the detection box corresponding to the text on the screen; If the controller executes a preset length growth condition, it determines, based on the geometric change characteristics of the detection frame, that the dynamic change type is either a prefix-growth type or a lens zoom-out exposure subclass under a geometric change type, specifically configured as follows: When the multiple by which the text length increases is greater than or equal to the preset growth threshold, it is determined that the preset length growth condition is met; If the preset length growth condition is met, and the prefix growth verification is passed, the width growth of the detection box is greater than or equal to the preset width threshold, and the height growth of the detection box is less than or equal to the preset height stability threshold, then the dynamic change type is determined to be prefix growth type. If the preset length growth condition is met, and the prefix growth verification fails, and the width growth of the detection box is lower than the preset width threshold, then the dynamic change type is determined to be the lens zoom-out exposure subclass.

[0015] The above technical solution has the following beneficial effects or advantages: For scenarios where text length increases, a secondary verification is performed. By combining the prefix verification results with the characteristics of changes in the width and height of the detection box, the prefix growth animation effect and the camera zoom-out reveal animation effect can be accurately distinguished, solving the problem of easy confusion between the two types of scene features and covering text reveal scenarios caused by camera movement.

[0016] In some embodiments of this application, the animation configuration parameters include the presentation parameters of the translated subtitles, which are used to present a word-by-word incrementing effect; if the dynamic change type is a prefix-incrementing type, the controller executes a mapping strategy corresponding to the dynamic change type to map the dynamic effect of the on-screen text across languages ​​to the translated subtitles, thereby obtaining the animation configuration parameters, which are specifically configured as follows: Extract the first total number of characters of the image text from the text content sequence, and the timestamp of the first recognition of each source character in the image text; Based on the first total number of characters and the second total number of characters of the translated subtitles, the target number of characters to be displayed in the corresponding display stage of the translated subtitles is obtained by mapping the source characters to the translated subtitles according to the position ratio of the source characters in the text on the screen. The timestamp is used as the time boundary for the corresponding display stage of the translated subtitle; The rendering parameters are generated based on the target number of characters to be displayed and the corresponding time boundary.

[0017] The above technical solution has the following beneficial effects or advantages: By using character position ratio mapping to achieve cross-language transfer of prefix growth animation, it is not affected by the difference in the number of characters between the source and target languages. It can restore the visual rhythm of the original text appearing word by word, ensuring that the dynamic presentation rhythm of the translated subtitles is completely consistent with that of the original text, preserving the visual narrative effect of the original video and enhancing the viewing immersion.

[0018] In some embodiments of this application, the animation configuration parameters include the initial scaling ratio, scaling duration, layout position, displacement vector, and motion duration of the translated subtitles; if the dynamic change type is geometric change, the controller executes a mapping strategy corresponding to the dynamic change type to map the dynamic effect of the on-screen text across languages ​​to the translated subtitles, thereby obtaining the animation configuration parameters, which are specifically configured as follows: For the physical scaling subclass, the initial scaling ratio is calculated based on the area ratio of the detection boxes corresponding to the first and last frames in the detection box position sequence, and the scaling duration is determined based on the time span of the detection box position sequence. For the subclass exposed by the lens being pulled back, the target position of the detection box with the largest area in the detection box position sequence is determined as the layout position; If the detection frame has continuous displacement, the displacement vector and the motion duration are calculated based on the anchor point coordinates of the first and last positions of the detection frame.

[0019] The above technical solution has the following beneficial effects or advantages: Different mapping strategies are designed for different subclasses of geometric transformations, respectively realizing scaling parameter migration, optimal layout adaptation and motion trajectory synchronization. This can reproduce the geometric dynamic effects of scaling and displacement of the original text, ensuring that the translation subtitles and the original text's motion state and display position are accurately matched, and avoiding misalignment and disconnection between the subtitles and the original text.

[0020] In some embodiments of this application, before determining the type of dynamic change of the on-screen text, the controller is further configured to: Detect the trend of text length change based on the aforementioned text content change features; Based on the changing trend, a target change curve is identified, which is a U-shaped change curve in which the text length first shrinks and then recovers. If the similarity between the target change curve, the restored text, and the original text is found to meet a preset similarity threshold, and the text located in the trough of the target change curve is identified as a substring of stable text, then a static display mapping strategy is adopted for the translated subtitles corresponding to the text on the screen. Wherein, the stable text is the complete text content of the on-screen text determined from the text content sequence; the static display mapping strategy is to not generate animation configuration parameters corresponding to dynamic effects.

[0021] The above technical solution has the following beneficial effects or advantages: By identifying the U-shaped length change curve to determine the pseudo-change of local occlusion, a static display strategy is adopted for pseudo-dynamic scenes caused by foreground object occlusion. This can avoid the accidental triggering of dynamic effects due to occlusion, reduce the interference of abnormal animation effects on the viewing experience, and improve the rationality and accuracy of subtitle rendering.

[0022] In some embodiments of this application, the trajectory data further includes a recognition confidence sequence corresponding to the text in each frame; When performing feature extraction on the trajectory data, the controller is also configured to: Based on the detection box area change sequence determined by the detection box position sequence, and combined with the recognition confidence sequence, the life cycle state of the text on the screen is determined. The life cycle state includes a shrinking state where the area is continuously shrinking and a fading state where the confidence is continuously decreasing. When the controller executes the command to overlay and render the translated subtitles onto the corresponding area of ​​the video screen, it is specifically configured as follows: If the text on the screen is detected to be in the shrunk or faded state, the rendering priority of the corresponding translated subtitle is reduced, and the display is controlled to overlay the translated subtitle onto the corresponding area of ​​the video screen according to the rendering priority.

[0023] The above technical solution has the following beneficial effects or advantages: By inferring the lifecycle state of text events through changes in area and confidence, and dynamically adjusting the rendering priority for exit scenarios where text shrinks or fades, the rendering order in multi-subtitle scenarios can be controlled, avoiding subtitle overlap and conflict, and improving the coordination and readability of subtitle display in multi-text scenarios.

[0024] Secondly, this application also provides a method for displaying translated subtitles, including: In a scenario where video footage is translated, trajectory data of text within the video footage is acquired; the trajectory data includes a sequence of detection box positions and a sequence of text content for the text within consecutive frames. Feature extraction is performed on the trajectory data to obtain the geometric change features of the detection boxes corresponding to the detection box position sequence and the text content change features corresponding to the text content sequence; Based on a preset order determination rule, the dynamic change type of the text on the screen is determined according to the text content change characteristics and the detection box geometric change characteristics; Based on the mapping strategy corresponding to the dynamic change type, the dynamic effect of the text on the screen is mapped across languages ​​to the translated subtitles to obtain the animation configuration parameters; Based on the animation configuration parameters, translated subtitles with corresponding dynamic effects are generated, and the display is controlled to overlay and render the translated subtitles onto the corresponding area of ​​the video screen.

[0025] The above technical solution has the following beneficial effects or advantages: It enables automated recognition and cross-language mapping rendering of text animation effects in video footage, restoring the dynamic visual effects of the original text without the need for manual configuration of animation parameters, thus improving the efficiency of subtitle production; at the same time, it ensures that the translated subtitles are consistent with the visual rhythm of the original footage, preserving the visual narrative design of the original video, thereby enhancing the user's cross-language viewing immersion. Attached Figure Description

[0026] To more clearly illustrate the technical solution of this application, the drawings used in the embodiments will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0027] Figure 1 This is a schematic diagram illustrating the operation scenarios of a display device provided in some embodiments of this application; Figure 2 This is a schematic diagram of the hardware configuration of a display device provided in some embodiments of this application; Figure 3 This is a schematic diagram of the software configuration of a display device provided in some embodiments of this application; Figure 4 A flowchart illustrating a method for displaying translated subtitles provided in some embodiments of this application; Figure 5 A schematic diagram of the dynamic change type classification decision-making process provided for some embodiments of this application; Figure 6 This application provides a schematic diagram of the prefix growth verification process for some embodiments. Figure 7This is a schematic diagram of the area expansion determination process provided in some embodiments of this application; Figure 8 This application provides a schematic diagram of a prefix-growing motion effect mapping process for some embodiments. Figure 9 This application provides schematic diagrams of geometric transformation motion effect mapping processes for some embodiments. Figure 10 A schematic diagram illustrating the process of inferring the lifecycle state of text on the screen and controlling the rendering priority, provided for some embodiments of this application; Figure 11 A schematic diagram illustrating the process for determining pseudo-dynamic scenes provided in some embodiments of this application; Figure 12 This is an interactive schematic diagram illustrating the display of translated subtitles provided in some embodiments of this application. Detailed Implementation

[0028] The embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described below do not represent all embodiments consistent with this application. They are merely examples of systems and methods consistent with some aspects of this application as detailed in the claims.

[0029] In this embodiment, display device 200 generally refers to a device with screen display and data processing capabilities. For example, display device 200 includes, but is not limited to, smart TVs, mobile terminals, computers, monitors, advertising screens, wearable devices, virtual reality devices, augmented reality devices, etc.

[0030] Figure 1 This is a schematic diagram illustrating an operational scenario between a display device and a control device provided in some embodiments of this application. For example... Figure 1 As shown, users can operate the display device 200 via touch operation, mobile terminal 300, and control device 100. For example, control device 100 can be a remote control, stylus, gamepad, etc.

[0031] The mobile terminal 300 can function as a control device for human-computer interaction between the user and the display device 200. It can also function as a communication device for establishing a communication connection with the display device 200 and exchanging data. In some embodiments, the mobile terminal 300 can have software applications installed on it and communicate with the display device 200 via network communication protocols to achieve one-to-one control and data communication. Furthermore, it can transmit audio and video content displayed on the mobile terminal 300 to the display device 200 for synchronized display.

[0032] like Figure 1The diagram also shows that the display device 200 communicates with the server 400 via various communication methods. This allows the display device 200 to communicate via a local area network (LAN), a wireless local area network (WLAN), and other networks.

[0033] Display device 200 can provide broadcast television reception function, and can also be equipped with intelligent network television function that provides computer support function, including but not limited to network television, smart television, Internet Protocol television (IPTV), etc.

[0034] Figure 2 Provided for some embodiments of this application Figure 1 Hardware configuration block diagram of display device 200.

[0035] In some embodiments, the display device 200 may include at least one of a tuner 210, a communication device 220, a detector 230, a device interface 240, a controller 250, a display 260, an audio output device 270, a memory, a power supply, and a user input interface 280.

[0036] In some embodiments, detector 230 is used to collect signals from the external environment or signals interacting with the outside world. For example, detector 230 may include millimeter-wave radar, which can be used to detect whether a user is present within a preset range. Detector 230 may also include a voice acquisition unit to collect voice commands input by the user.

[0037] In some embodiments, the display 260 includes display function components for presenting images and driving components for driving image display. The display 260 is used to receive and display image signals output from the controller 250. For example, the display 260 can be used to display video content, image content, menu control interface components, and user control UI interfaces, etc.

[0038] In some embodiments, the communication device 220 is a component used to communicate with external devices or the server 400 according to various communication protocol types. The display device 200 may have multiple communication devices 220 depending on the supported communication methods. For example, when the display device 200 supports wireless network communication, it may have a communication device 220 with WiFi functionality. When the display device 200 supports Bluetooth connectivity, it needs to have a communication device 220 with Bluetooth functionality.

[0039] The communication device 220 enables the display device 200 to communicate with external devices or the server 400 via wireless or wired connections. Wired connections utilize data cables, interfaces, or other components to connect the display device 200 to external devices. Wireless connections utilize wireless signals or wireless networks. The display device 200 can directly establish a connection with external devices or indirectly through gateways, routers, or other connection devices.

[0040] In some embodiments, the controller 250 may include at least one of a central processing unit, a video processor, an audio processor, a graphics processor, and a power processor, and a first to an nth interface for input / output. The controller 250 controls the operation of the display device and responds to user operations through various software control programs stored in memory. The controller 250 controls the overall operation of the display device 200.

[0041] In some embodiments, the controller 250 and the tuner 210 may be located in different separate devices, that is, the tuner 210 may also be located in an external device of the main device where the controller 250 is located, such as an external set-top box.

[0042] In some embodiments, a user can input user commands through a graphical user interface (GUI) displayed on a display 260, and the user input interface receives user input commands through the graphical user interface (GUI).

[0043] In some embodiments, the audio output device 270 can be a built-in speaker of the display device 200 or an external audio output device connected to the display device 200. For the external audio output device connected to the display device 200, the display device 200 may also be provided with an external audio output terminal, through which the audio output device can be connected to the display device 200 to output sound from the display device 200.

[0044] In some embodiments, the user input interface 280 can be used to receive instructions input by a user. The user input interface 280 may include at least one of a microphone, touchpad, sensor, remote control, etc. The display device 200 can then receive user-input instructions based on the user input interface 280 to perform interactive functions with the user.

[0045] In some embodiments, to enable user interaction, the display device 200 may run an operating system. An operating system is a computer program that manages and controls the hardware and software resources of the display device 200. The operating system can control the display device to provide a user interface; for example, the operating system can directly control the display device to provide a user interface, or it can provide a user interface by running applications. The operating system also allows users to interact with the display device 200.

[0046] It should be noted that the operating system can be a native operating system based on a specific operating platform, a third-party operating system that is deeply customized based on a specific operating platform, or an independent operating system specifically developed for display devices.

[0047] An operating system can be divided into different modules or levels based on the functions it implements, for example... Figure 3 As shown, in some embodiments, the system is divided into four layers, from top to bottom: the Applications layer (referred to as the "Application Layer"), the Application Framework layer (referred to as the "Framework Layer"), the System Library layer, and the Kernel layer.

[0048] In some embodiments, the application layer provides services and interfaces for applications, enabling the display device 200 to run applications and interact with the user based on the applications. The application layer may contain at least one application, which may be a built-in Windows program, system settings program, or clock program of the operating system; or it may be an application developed by a third-party developer. In specific implementations, the application packages in the application layer are not limited to the examples above.

[0049] The framework layer provides application programming interfaces (APIs) and a programming framework for applications. The application framework layer includes predefined functions. It acts as a central processing unit, determining the actions taken by applications within the application layer. Through the API, applications can access system resources and obtain system services during execution.

[0050] like Figure 3As shown, the application framework layer in this embodiment includes a view system, managers, and content providers. The view system designs and implements the application's interface and interactions, and includes lists, grids, text boxes, and buttons. The managers include at least one of the following modules: an activity manager for interacting with all running activities in the system; a location manager for providing system services or applications with access to system location services; a package manager for retrieving various information related to application packages currently installed on the device; a notification manager for controlling the display and clearing of notification messages; and a window manager for managing icons, windows, toolbars, wallpapers, and desktop widgets on the user interface.

[0051] In some embodiments, the Activity Manager manages the lifecycle of individual applications and common navigation and back functions, such as controlling application exit, opening, and back actions. The Window Manager manages all window programs, such as obtaining the screen size, determining if a status bar is present, locking the screen, capturing the screen, and controlling changes to the display window, such as shrinking the display window, shaking the display, or distorting the display.

[0052] In some embodiments, the system runtime library layer can provide support for the framework layer. When the framework layer is used, the operating system runs the instruction library contained in the system runtime library layer, such as the C / C++ instruction library, to implement the functions to be performed by the framework layer.

[0053] In some embodiments, the kernel layer is a functional layer situated between the hardware and software of the display device 200. The kernel layer can implement functions such as hardware abstraction, multitasking, and memory management. For example, ... Figure 3 As shown, hardware drivers can be configured in the kernel layer. The kernel layer can contain at least one of the following drivers: audio driver, display driver, Bluetooth driver, camera driver, WIFI driver, USB driver, HDMI driver, sensor driver (such as fingerprint sensor, temperature sensor, pressure sensor, etc.), and power driver, etc.

[0054] In some embodiments, the display device 200 supports subtitle display functionality in video translation scenarios. The subtitle display function is based on OCR technology, which detects, recognizes, and tracks embedded text within the video frame across frames. The recognized text is then translated and overlaid onto the corresponding area of ​​the screen as subtitles, thus achieving cross-language translation of the text. However, this method of presenting translated subtitles fails to adapt to the diverse dynamic expressions of video text, resulting in a disconnect between the translated subtitles and the visual dynamic rhythm of the original text. This makes it difficult to reproduce the original visual effects of the video, thereby affecting the user's viewing immersion.

[0055] To address the aforementioned issues, this application provides a display device and a method for displaying translated subtitles. The display device 200 extracts multi-dimensional features from the OCR trajectory data of the text on the screen, determines the dynamic change type of the text on the screen using a multi-stage sequential judgment, designs differentiated cross-language motion effect mapping strategies for each type, generates translated subtitles carrying corresponding motion effects, and overlays and renders them onto the video screen, thereby restoring the dynamic visual performance of the original text and enhancing the user's cross-language viewing immersion.

[0056] Figure 4 This is a flowchart illustrating a method for displaying translated subtitles provided in some embodiments of this application. Figure 4 As shown, controller 250 is configured to perform the following steps: S401. In the scenario of translating video footage, obtain the trajectory data of the text in the video footage.

[0057] The scenario of translating video footage refers to performing a translation function on the video footage. In some embodiments, when the display device 200 performs the translation function on the video footage, it uses an OCR module to detect, recognize, and track text in consecutive video frames, generating trajectory data of the text. The trajectory data includes a sequence of detection box positions for the text in consecutive frames and a sequence of text content. The detection box position sequence records the coordinates (x, y, w, h) of the detection box for the text in each frame and its corresponding timestamp, while the text content sequence records the text content recognized by the OCR in each frame.

[0058] In some embodiments, the trajectory data also includes an OCR recognition confidence sequence for each frame, which reflects the recognition reliability of the current frame.

[0059] In some embodiments, the text in the image may exhibit various dynamic change patterns in consecutive frames, such as the title text gradually increasing in size as the camera zooms in (physical scaling), text appearing one character at a time under a typewriter effect (prefix growth), more text gradually appearing in the image as the camera slowly pulls back (lens zoom-out reveal), and pseudo-dynamic scenes where text is temporarily obscured by foreground objects and then reappears (partial occlusion). The controller 250 identifies these different dynamic change patterns by analyzing the trajectory data, providing a basis for subsequent differentiated motion effect mapping.

[0060] S402. Perform feature extraction on the trajectory data to obtain the geometric change features of the detection boxes corresponding to the detection box position sequence and the text content change features corresponding to the text content sequence.

[0061] The controller 250 extracts multidimensional features from the trajectory data for dynamic change classification, including text content change features and detection box geometric change features.

[0062] In some embodiments, the text content change features include a text length sequence and its inter-frame change direction sequence, a text length growth ratio, and prefix growth features (prefix consistency and single-step growth character count). For example, the controller 250 calculates the text length of each frame in the text content sequence, counts adjacent frame pairs where the text length increases, detects whether adjacent frame pairs satisfy a prefix relationship (the text content of the next frame is prefixed by the text content of the previous frame), calculates the ratio of the number of frame pairs satisfying the prefix relationship to all increasing frame pairs to obtain prefix consistency, and counts the number of characters increased each time to obtain the single-step growth character count.

[0063] In this embodiment, length growth refers to the text length of the later frame being greater than the text length of the earlier frame between two adjacent frames. An increasing frame pair is an adjacent frame pair that satisfies the length growth condition.

[0064] In some embodiments, the geometrical variation features of the detection frame include area expansion ratio, aspect ratio change, height increase, and width increase. Specifically, the area expansion ratio is the ratio of the maximum to the minimum area in the detection frame position sequence; the aspect ratio change is the ratio of the maximum to the minimum aspect ratio in the detection frame aspect ratio sequence; the height increase is the ratio of the maximum to the minimum height in the detection frame height sequence; and the width increase is the ratio of the maximum to the minimum width in the detection frame width sequence.

[0065] In addition, in some embodiments, the controller 250 also calculates the cumulative displacement of the center point of the detection frame to identify the translational movement of the text on the screen.

[0066] S403. Based on the preset order judgment rules, determine the dynamic change type of the text on the screen according to the text content change characteristics and the detection box geometric change characteristics.

[0067] Among them, the preset order determination rule can be a multi-stage progressive order determination rule.

[0068] In some embodiments, the controller 250 employs a three-stage progressive sequential decision architecture for dynamic change classification, executing each stage of classification sequentially according to the logic of feature recognition from high to low, and only proceeding to the next stage when a stage fails to hit the target.

[0069] In some embodiments, the first-stage determination rule prioritizes prefix growth verification to identify the typewriter effect. If the prefix growth verification passes, the dynamic change type is directly determined to be prefix growth type, and the classification ends. Prefix growth type corresponds to the typewriter animation effect where text appears character by character in the video. Its text content sequence strictly satisfies the prefix increment relationship, and the number of characters increased in each step does not exceed a preset character threshold.

[0070] In some embodiments, if the first stage fails, a second stage judgment rule is executed, which determines the area expansion based on the geometric change features of the detection box. If a preset area expansion condition is met (i.e., the area expansion ratio is greater than or equal to a preset area threshold), the text content change features are combined to further determine whether it belongs to the physical scaling subclass. The physical scaling subclass corresponds to scenes in videos where the text is physically enlarged due to factors such as camera zooming in. In these scenes, the detection box area expands significantly, and the aspect ratio or height changes noticeably, while the text content remains relatively stable.

[0071] In some embodiments, if the second stage fails, a third stage judgment rule is executed to perform a second judgment on the text length growth. If the preset length growth condition is met (i.e., the text length growth multiple is greater than or equal to the preset growth threshold), the prefix growth type and the camera zoom-out exposure type are distinguished by combining the prefix verification result with the detection box width and height change characteristics. The camera zoom-out exposure type corresponds to a scene where the camera slowly zooms out, gradually revealing more text in the image. Its text length increases but does not satisfy the strict prefix relationship, and the detection box width expansion is not significant.

[0072] In some embodiments, if all stages are not hit, the dynamic change type is determined to be static and no dynamic effect is generated.

[0073] S404. Based on the mapping strategy corresponding to the dynamic change type, map the dynamic effects of the text on the screen to the translated subtitles across languages ​​to obtain the animation configuration parameters.

[0074] Controller 250 executes differentiated cross-language motion mapping strategies for different types of dynamic changes, generating corresponding animation configuration parameters.

[0075] In some embodiments, the prefix-incrementing type employs a character position ratio mapping strategy, mapping the position ratio of source text characters in the original text to the corresponding translation character positions in the translated subtitles. At the same time, it reuses the character timestamps of the source text as the display time boundaries of the translated subtitles, thereby achieving cross-language word-by-word display rhythm alignment.

[0076] In some embodiments, the geometric transformation type migrates scaling parameters (initial scaling ratio and scaling duration) for the physical scaling subclass, determines the optimal layout position for the zoom-out subclass, and migrates the displacement vector and motion duration for on-screen text with continuous displacement.

[0077] S405. Generate translated subtitles with corresponding dynamic effects according to the animation configuration parameters, and control the monitor to overlay and render the translated subtitles onto the corresponding area of ​​the video screen.

[0078] According to the animation configuration parameters, the controller 250 renders the translated subtitles as subtitle events with corresponding dynamic effects, and controls the display 260 to overlay the translated subtitles onto the corresponding area of ​​the video screen.

[0079] In some embodiments, prefix-incrementing translated subtitles are rendered as a sequence of multiple progressively increasing prefix events, with each event displaying a different length prefix of the translated subtitle; geometrically variable translated subtitles carry scaling or movement tags to achieve a dynamic effect synchronized with the physical movement of the original text; statically appearing subtitles are given a default fade-in / fade-out effect.

[0080] This embodiment of the disclosure achieves automated recognition and cross-language mapping rendering of dynamic text effects in video footage through the above steps. It eliminates the need for manual configuration of motion effect parameters, faithfully reproducing various visual effects from the original text in the translated subtitles, thus enhancing the user's immersive cross-language viewing experience. Furthermore, all classifications and mappings are based on existing OCR trajectory data for feature extraction and rule judgment, without introducing additional deep learning model inference, resulting in extremely low computational overhead and adaptability to the computing power constraints of display devices.

[0081] Figure 5 This is a schematic diagram illustrating the dynamic change type classification decision-making process provided for some embodiments of this application. For example... Figure 5 As shown, the classification process of this application embodiment takes OCR trajectory features (i.e., detection box geometric change features and text content sequence) as input, judges them in three stages in sequence, and finally outputs the dynamic change type.

[0082] In some embodiments, the first stage is prefix growth detection, where the controller 250 calls the first stage judgment rule to perform the first stage classification. That is, based on the text content change features, prefix growth verification is performed on the text on the screen to determine whether the text content sequence satisfies the prefix growth feature. If the prefix consistency is greater than or equal to a preset percentage threshold (e.g., 60%), and the number of characters in each single step growth is less than or equal to a preset character number threshold (e.g., 3 characters), then the prefix growth verification is deemed to have passed, and the dynamic change type is directly determined to be prefix growth type (typewriter). The reason for prioritizing the execution of the first stage is that the prefix growth feature has the highest recognition rate and the judgment computational overhead is the lowest, which can quickly filter out mainstream typewriter effect scenarios.

[0083] In some embodiments, the second stage is bounding box (bbox) geometric change detection. If the first stage fails to identify the correct classification result, the controller 250 enters the second stage and executes the second stage classification by invoking the second stage judgment rules. That is, it detects whether the bounding box has significant area expansion (area expansion ratio greater than or equal to a preset area threshold, such as 1.2). If the area expansion condition is met, the physical scaling subclass is determined based on the text content change: when the text content is stable (text length growth multiple less than a preset growth threshold), and the aspect ratio change of the bounding box is greater than or equal to a preset aspect ratio threshold (such as 1.3), or the height growth of the bounding box is greater than or equal to a preset height threshold (such as 1.5), the dynamic change type is determined to be a geometric change type physical scaling subclass (scale_blur_reveal). In addition, if the text content is unstable but the tail text tends to be stable, and the above aspect ratio or height change conditions are met, it is also classified as a physical scaling subclass.

[0084] In some embodiments, the third stage is a secondary determination of text length growth. If the second stage fails to match the classification result, the controller 250 enters the third stage and calls the third stage determination rules to perform the third stage classification. That is, it detects whether there is a significant increase in the length of the text content (i.e., the text length growth multiple is greater than or equal to a preset growth threshold, such as 1.15, and the minimum text length is not lower than a preset length lower limit, such as 3 characters). If the length growth condition is met, a secondary distinction is made by combining the prefix verification result with the width and height change features of the detection box: if the prefix growth verification passes, the width growth of the detection box is greater than or equal to a preset width threshold (such as 1.3), and the height growth is less than or equal to a preset height stability threshold (such as 1.5), then the dynamic change type is determined to be prefix growth type; if the prefix growth verification fails, and the width growth of the detection box is lower than the preset width threshold, then the dynamic change type is determined to be the geometric change type of camera_reveal subclass.

[0085] In some embodiments, if none of the above stages are hit, the dynamic change type is determined to be static, which corresponds to a normal scenario where the text has no special dynamic effects, and only the default fade-in and fade-out effects are added.

[0086] This embodiment of the disclosure adopts a three-stage progressive sequential judgment architecture, which filters according to the logic of feature recognition from high to low and computational amount from small to large. It not only ensures the classification efficiency of mainstream scenes (prefix growth type) but also covers the classification accuracy of edge scenes (geometric change type, lens zoom-out exposure type). It can distinguish various dynamic types such as prefix growth, physical scaling, and lens zoom-out exposure, and adapt to the dynamic presentation scenarios of various screen texts in videos.

[0087] Figure 6 This is a schematic diagram illustrating the prefix growth verification process provided for some embodiments of this application. For example... Figure 6 As shown, when performing prefix growth verification on the text on the screen, controller 250 is configured to perform the following steps: S601. Traverse all target adjacent frame pairs with increasing text length in the text content sequence.

[0088] Controller 250 iterates through the text content sequence, tracking changes in text length between adjacent frames, and selects adjacent frame pairs where the text length increases as target adjacent frame pairs. A target adjacent frame pair is defined as one where the text length of the subsequent frame is strictly greater than the text length of the preceding frame.

[0089] S602. Mark the target adjacent frame pairs that satisfy the prefix relationship as valid growing frame pairs.

[0090] For each target adjacent frame pair, the controller 250 determines whether the text content of the next frame is a prefix of the text content of the previous frame (i.e., the starting part of the text content of the next frame is exactly the same as the text content of the previous frame, and the text length of the next frame is longer). Target adjacent frame pairs that satisfy the prefix relationship are marked as valid growing frame pairs, and those that do not are marked as invalid growing frame pairs.

[0091] S603. Calculate the ratio of the number of valid growing frame pairs to the number of target adjacent frame pairs to obtain prefix consistency; and detect the number of single-step growing characters in the target adjacent frame pairs.

[0092] The controller 250 counts the number of valid growing frame pairs and the target adjacent frame pairs, calculates the ratio between the two, and obtains the prefix consistency. For example, if there are 10 adjacent frame pairs with increasing length in the text content sequence, and 8 of them satisfy the prefix relationship, then the prefix consistency is 80%. At the same time, the controller 250 detects the increase in the length of the text in the next frame relative to the previous frame in each target adjacent frame pair, and obtains the number of characters increased in a single step.

[0093] S604. If the prefix consistency is greater than or equal to the preset ratio threshold, and the number of characters increased in a single step is less than or equal to the preset number of characters threshold, then the prefix growth verification is deemed to have passed.

[0094] The controller 250 determines whether the prefix consistency is greater than or equal to a preset percentage threshold (e.g., 60%), and whether the number of characters increased in a single step for all adjacent target frame pairs is less than or equal to a preset character count threshold (e.g., 3 characters). If both conditions are met, the prefix growth verification is deemed successful, and the dynamic change type of the text on the screen is determined to be prefix growth type. If either condition is not met, the prefix growth verification is deemed unsuccessful, and the subsequent classification stage continues.

[0095] In some embodiments, setting the condition that the number of characters increased per step does not exceed a preset character threshold is to distinguish between a real typewriter animation (displayed character by character, increasing by 1-3 characters per step) and noise generated by OCR progressive recognition (OCR may recognize a large number of characters at once when the resolution or lighting changes). If the number of characters increased per step is too large (e.g., more than 3), it is more likely to be OCR progressive recognition noise rather than a real typewriter effect.

[0096] In this embodiment, the prefix growth feature is verified by a dual criterion of prefix consistency ratio and single-step character growth number. This can distinguish between real typewriter animation and noise generated by OCR progressive recognition, reduce the misjudgment rate of prefix growth type, avoid misjudging recognition fluctuations as text animation, and thus improve the reliability of classification results.

[0097] Figure 7 This is a schematic diagram illustrating the area expansion determination process provided in some embodiments of this application. For example... Figure 7 As shown, when performing area expansion determination based on the geometric change features of the detection box on the text in the image to determine whether it belongs to a physical scaling subclass, the controller 250 is configured to perform the following steps: S701. Calculate the area expansion ratio in the detection box position sequence.

[0098] The controller 250 calculates the area (width × height) of each detection box in the detection box position sequence, obtains the maximum and minimum areas in the sequence, and calculates the ratio between the two to obtain the area expansion ratio. For example, if the maximum area in the detection box position sequence is 10,000 pixels and the minimum area is 5,000 pixels, then the area expansion ratio is 2.0.

[0099] S702. Determine whether the area expansion ratio meets the preset area expansion conditions.

[0100] The controller 250 determines whether the area expansion ratio is greater than or equal to a preset area threshold (e.g., 1.2). If the area expansion ratio is greater than or equal to the preset area threshold, it is determined that the preset area expansion condition is met; otherwise, it is determined that the preset area expansion condition is not met, and it directly jumps to the third stage classification.

[0101] S703. If the preset area expansion condition is met, determine whether the multiple of text length growth is less than the preset growth threshold.

[0102] If the preset area expansion condition is met, the controller 250 further determines whether the text length growth factor in the text content change feature is less than a preset growth threshold (e.g., 1.15). The text length growth factor is the ratio of the maximum text length to the minimum text length in the text content sequence. If the text length growth factor is less than the preset growth threshold, it indicates that the text content is basically stable, meaning that the expansion of the detection box area is caused by physical scaling (such as zooming in or text magnification) rather than an increase in the text content itself. In this case, the physical scaling subclass is judged. If the text length growth factor is not less than the preset growth threshold, the text content has changed significantly, and further differentiation is needed based on text stability.

[0103] S704, Execute the determination of the physical scaling subclass.

[0104] The controller 250 determines whether the change in the aspect ratio of the detection frame is greater than or equal to a preset aspect ratio threshold, or whether the increase in the height of the detection frame is greater than or equal to a preset height threshold. Here, the aspect ratio change is the ratio of the maximum to the minimum value of the aspect ratio sequence of the detection frame, and the height increase is the ratio of the maximum to the minimum value of the height sequence of the detection frame. If the change in the aspect ratio of the detection frame is greater than or equal to the preset aspect ratio threshold (e.g., 1.3), or the increase in the height of the detection frame is greater than or equal to the preset height threshold (e.g., 1.5), then the dynamic change type is determined to be a physical scaling subclass.

[0105] In some embodiments, for scenarios where the text content is unstable but the text at the end tends to be stable (i.e., the text length of the last few frames of the text content sequence has tended to be stable), if the aspect ratio change or height increase condition is also met, it is also determined to be a physical scaling subclass. This is because in some film and television scenes, the process of the camera zooming in is accompanied by the gradual and complete recognition of the text (the text length increases), but the text content is eventually fully displayed, which is still a physical scaling effect.

[0106] This embodiment combines multi-dimensional geometric features such as area expansion, text stability, aspect ratio change, and height increase to determine the physical scaling effect. It can accurately distinguish between real physical scaling animation effects and ordinary area fluctuations, covering various film and television scenes such as camera zoom-in and text magnification, improving the accuracy of geometric animation effect recognition and avoiding false triggering of scaling animation effects.

[0107] Furthermore, regarding the aforementioned third-stage judgment rule—the secondary judgment of text length growth performed after the second-stage classification miss—it is necessary to combine the prefix verification results with the detection box width and height change features to distinguish between prefix-growing and camera-retreat-revealed subclasses. The controller 250 also checks whether the text length growth factor meets the preset length growth condition. That is, the controller 250 extracts the minimum and maximum text lengths of the on-screen text from the text content sequence and calculates the text length growth factor (i.e., the ratio of the maximum text length to the minimum text length). When the text length growth factor is greater than or equal to the preset growth threshold (e.g., 1.15), and the minimum text length is not lower than the preset length lower limit (e.g., 3 characters), it is determined that the preset length growth condition is met. Meeting the preset length growth condition indicates that the on-screen text has undergone significant text length growth within the trajectory period, requiring further differentiation of the specific dynamic change type based on other features. If the preset length growth condition is not met, the subsequent default classification continues (determining the dynamic change type as static appearance).

[0108] For on-screen text that meets the preset length growth conditions, the controller 250 first checks the previously performed prefix growth verification result. If the prefix growth verification passes, the controller 250 extracts the width sequence from the detection box position sequence and calculates the width growth ratio (i.e., the ratio of the maximum to the minimum value in the width sequence). If the width growth is greater than or equal to a preset width threshold (e.g., 1.3), it indicates that the detection box width has significantly expanded, such as due to interference from non-animation factors like changes in light source, changes in lens angle, or fluctuations in the OCR recognition area, rather than from the width expansion feature of the scene being revealed by the camera zooming out. The controller 250 also extracts the height sequence from the detection box position sequence and calculates the height growth ratio (i.e., the ratio of the maximum to the minimum value in the height sequence). If the height growth is less than or equal to a preset height stability threshold (e.g., 1.5), it indicates that the detection box height is relatively stable, which conforms to the characteristic of text only gradually appearing horizontally or vertically in prefix growth scenarios while the height remains basically unchanged.

[0109] If the prefix growth verification passes, the width increase is greater than or equal to the preset width threshold, and the height increase is less than or equal to the preset height stability threshold, then the dynamic change type is determined to be prefix growth type. If the preset length growth condition is met but the prefix growth verification fails, the geometric features of the detection box are used to determine whether it belongs to the "lens zooming out" subclass.

[0110] For text in the image that meets the preset length growth condition but fails the prefix growth verification, the controller 250 further determines whether the width growth of the detection box is lower than a preset width threshold (e.g., 1.3). If the width growth of the detection box is lower than the preset width threshold (indicating that the width of the detection box has not significantly expanded), then the dynamic change type is determined to be the lens zoom-out exposure subclass under the geometric change type. The lens zoom-out exposure subclass corresponds to a film and television scene where the camera slowly zooms out (such as aerial shots or panoramic transitions), and more text is gradually revealed in the image. In this scene, although the text length increases (because more and more text enters the field of view), it does not satisfy a strict prefix increment relationship (because the newly appearing text may be located at any position in the original text), and the width expansion of the detection box is not significant (because the absolute size of the text does not increase during the lens zoom-out process but may even decrease).

[0111] If the width of the detection box increases to be greater than or equal to the preset width threshold, or the height increases to exceed the preset height stability threshold, it indicates that the detection box exhibits atypical change characteristics. In this case, it is not classified as the "lens zoom-out" subclass, but the default classification process continues.

[0112] In this embodiment, the prefix verification result is used as the primary distinguishing criterion, the increase in the width of the detection box is used as an auxiliary judgment feature, and the stability of the detection box height is used as a constraint condition. This can distinguish between prefix growth animation (typewriter effect) and camera zoom-out exposure animation (camera field of view expansion), solving the problem that the two types of scenes have similar features and are easily confused in the dimension of text length growth. At the same time, the non-interchangeable stage sequence design ensures the logical rigor of the overall classification architecture, covering text exposure effects caused by camera movement in various film and television scenes.

[0113] For example, as shown in Table 1, Table 1 uses the time axis as the horizontal axis to show a comparison of four types of dynamically changing OCR trajectory features (i.e., detection box geometric change features and text content sequences).

[0114]

[0115] Table 1 Figure 8 This is a schematic diagram illustrating the prefix-growing motion effect mapping process provided in some embodiments of this application. For example... Figure 8 As shown, if the dynamic change type of the on-screen text is prefix-incrementing, the controller 250 is configured to perform the following steps to map the dynamic effect of the on-screen text across languages ​​to the translated subtitles: S801, Extract the first total number of characters of the image text from the text content sequence, and the timestamp of the first recognition of each source character in the image text.

[0116] The controller 250 extracts the total number of characters of the final complete text of the text on the screen from the text content sequence, as the first total character count (i.e., the source character count). At the same time, the controller 250 extracts the timestamp corresponding to the first OCR recognition of each source character index from the character timestamp record, forming a character timestamp sequence (such as the format [(character index, timestamp),...]).

[0117] For example, for the typewriter effect "HELLO", the character timestamp sequence might be [(0,1000ms),(1,1200ms),(2,1400ms),(3,1600ms),(4,1800ms)], representing the timestamps of the first appearance of H, E, L, L, and O, respectively.

[0118] S802. Based on the first total number of characters and the second total number of characters of the translated subtitles, the target number of characters to be displayed in the corresponding display stage of the translated subtitles is obtained by mapping according to the character position ratio.

[0119] The controller 250 obtains the total number of characters in the translated subtitles as the second total character count (i.e., the target character count). For each source character index in the character timestamp sequence, the character position ratio is mapped according to the following formula: Target display character count = round((character index + 1) / first total character count × second total character count). The position ratio (round) of the source character in the original text is mapped to the number of characters that should be displayed at the same position ratio in the translated subtitles, thereby achieving cross-language character position alignment.

[0120] S803. Use the timestamp corresponding to the source character as the time boundary for the corresponding display stage of the translated subtitle.

[0121] The controller 250 directly uses the timestamp of each source character in the character timestamp sequence as the time boundary for the corresponding display stage of the translated subtitle. That is, the translated subtitle switches to the new display stage at the time point corresponding to the timestamp of the source character, displaying the prefix text corresponding to the target number of characters. In this way, the display rhythm of the translated subtitle is completely consistent with the typing rhythm of the original text.

[0122] S804. Generate rendering parameters based on the target number of characters to be displayed and the corresponding time boundary.

[0123] The controller 250 generates presentation parameters for the translated subtitles based on the time boundaries of each display stage and the target number of characters to be displayed, i.e., a sequence of events that increment character by character. The controller 250 generates multiple translated subtitle events, each corresponding to a display stage, displaying different length prefixes of the translated subtitles, with the time intervals closely aligned with the source character timestamps. The last event contains the complete translated subtitle and continues until the end of the subtitle display time.

[0124] In this embodiment, a character position ratio mapping method is used to achieve cross-language transfer of prefix growth animation. It is not affected by the difference in the number of characters between the source language and the target language. It can restore the visual rhythm of the original text appearing word by word, ensure that the dynamic presentation rhythm of the translated subtitles is completely consistent with that of the original text, retain the visual narrative effect of the original video, and enhance the viewing immersion.

[0125] Figure 9 This is a schematic diagram illustrating the geometric transformation motion effect mapping process provided in some embodiments of this application. For example... Figure 9 As shown, if the dynamic change type of the on-screen text is geometrical, the controller 250 is configured to perform the following steps to map the dynamic effect of the on-screen text across languages ​​to the translated subtitles: S901. Determine the corresponding mapping strategy based on the type of geometric transformation.

[0126] The controller 250 executes the following differentiated mapping strategies based on the geometric change subtypes determined by the classification.

[0127] S902. For the physical scaling subclass, extract the scaling parameters and map them to the translated subtitles.

[0128] For the physical scaling subclass, the controller 250 calculates the initial scaling ratio based on the area ratio of the corresponding detection boxes in the first and last frames of the detection box position sequence. The initial scaling ratio is calculated as max(floor(100 / sqrt(area of ​​the last frame / area of ​​the first frame)), 30), where the initial scaling ratio corresponds to the starting ratio (lower limit 30%) at which the translated subtitles are scaled from a smaller size to the full size. Simultaneously, the controller 250 determines the scaling duration based on the time span of the detection box position sequence (the difference between the timestamps of the first and last frames). Based on the initial scaling ratio and scaling duration, scaling animation parameters for the translated subtitles are generated, causing the translated subtitles to display a scaling animation that gradually changes from the initial scaling ratio to the full size, with the visual rhythm synchronized with the physical enlargement process of the original text.

[0129] S903. For subcategories that are exposed when the camera zooms out, determine the layout and position of the translated subtitles.

[0130] For subcategories exposed when the camera zooms out, the controller 250 finds the largest detection box from the detection box position sequence and determines its corresponding screen position as the layout position for the translated subtitles. This is because the largest detection box corresponds to the position of the text when it is fully exposed after the camera zooms out. Placing the translated subtitles at this position ensures that they accurately correspond to the display area of ​​the original text after the text is fully exposed, avoiding misalignment between the subtitles and the original text.

[0131] S904. If the detection frame has continuous displacement, calculate the displacement vector and motion duration.

[0132] The controller 250 calculates the movement distance of the center point of the detection box in adjacent frames of the detection box position sequence. If the cumulative displacement is greater than or equal to a preset displacement threshold (e.g., 20 pixels), it is considered that the detection box has continuous displacement (i.e., the text on the screen has undergone translational movement). At this time, the controller 250 calculates the displacement vector (starting and ending coordinates) based on the anchor point coordinates of the beginning and end positions of the detection box (selecting the top left corner, top edge midpoint, or top right corner anchor point according to the text alignment method), and determines the movement duration based on the time span of the detection box position sequence to generate the translational motion effect parameters for the translated subtitles. If the cumulative displacement is less than the preset displacement threshold, no positional movement effect is generated to avoid accidental triggering of the movement animation due to slight jitter of the detection box.

[0133] In this embodiment, differentiated mapping strategies are designed for different subclasses of geometric transformations to achieve scaling parameter migration, optimal layout adaptation, and motion trajectory synchronization. This can reproduce the geometric dynamic effects of scaling and displacement of the original text, ensuring that the translation subtitles and the original text's motion state and display position are accurately matched, and avoiding misalignment or disconnection between the subtitles and the original text.

[0134] Figure 10 This is a schematic diagram illustrating the process of inferring the lifecycle state of on-screen text and controlling rendering priority in some embodiments of this application. In some embodiments, in order to control the rendering order in multi-text scenes and avoid subtitle overlap conflicts, the controller 250 is also configured to determine the lifecycle state of the on-screen text when extracting features from the trajectory data, and adjust the rendering priority according to the lifecycle state during the rendering stage, and then render the translated subtitles according to the rendering priority.

[0135] In some embodiments, the controller 250 infers a shrinking state where the area is continuously shrinking and a fading state where the confidence level is continuously decreasing, based on the detection frame area change sequence and the recognition confidence sequence, respectively.

[0136] For the zoomed-out state, the controller 250 examines the area of ​​the detection boxes in the last few frames of the detection box position sequence (e.g., the most recent 4 frames). If the area at the end is less than a preset proportion (e.g., 60%) of the peak area, it determines that the text on the screen is in a zoomed-out state. The zoomed-out state corresponds to a scenario where the text on the screen is exiting the screen (e.g., the text gradually shrinks and disappears as the camera zooms out).

[0137] For the fading state, the controller 250 calculates the average confidence of the recognition confidence sequence in the early stage (e.g., the first 1 / 3 frame) and the recent stage (e.g., the last 4 frames). If the recent average confidence is lower than the preset proportion (e.g., 60%) of the early average confidence, the text on the screen is determined to be in the fading state. The fading state corresponds to a scenario where the confidence of the text on the screen continuously decreases due to reduced brightness, blurring, or visual fading.

[0138] In some embodiments, when the controller 250 executes the control display 260 to overlay and render the translated subtitles onto the corresponding area of ​​the video screen, if it detects that the text on the screen is in a shrunk or faded state, it reduces the rendering priority of the corresponding translated subtitles and sorts the multiple translated subtitles according to the rendering priority, giving priority to rendering the translated subtitles with higher priority, and suppressing or delaying the display of the translated subtitles with lower priority when there is a spatial conflict.

[0139] In some embodiments, the rendering priority can be defined as follows: prefix-growing subtitles have a higher priority than physically scaled subtitles, physically scaled subtitles have a higher priority than those that appear when the camera zooms out, and static subtitles have the lowest priority. When the text on the screen is shrinking or fading out, the rendering priority of the corresponding subtitle is further reduced on top of the above basic priority, so that the spatial conflict between the exiting subtitle and the newly appearing subtitle is reasonably controlled.

[0140] This embodiment of the disclosure infers the lifecycle state of text events by changing the area and confidence level, and dynamically adjusts the rendering priority for exit scenarios where text shrinks or fades, thereby controlling the rendering order in multi-subtitle scenarios, avoiding subtitle overlap conflicts, and improving the coordination and readability of subtitle display in multi-text scenarios.

[0141] Furthermore, in some embodiments, before classifying dynamic changes, the controller 250 performs U-curve detection on the text length change trend to eliminate pseudo-dynamic scenes. That is, before determining the type of dynamic change in the text on the screen, the controller 250 also predicts the text length change trend based on the text content change characteristics, identifying pseudo-dynamic scenes caused by foreground objects temporarily obscuring the text, thus avoiding misidentifying partial occlusion as a dynamic change type. For example... Figure 11 As shown, controller 250 is configured to perform the following steps: S1101. Detect the trend of text length change based on text content change features.

[0142] The controller 250 iterates through the text content sequence, calculates the text length of each frame, and forms a sequence curve of text length changing over time.

[0143] S1102. Identify target change curves based on change trends.

[0144] The target change curve is a U-shaped curve in which the text length first shrinks and then recovers. The controller 250 detects the presence of a U-shaped pattern of shrinkage followed by recovery in the text length change sequence. If the text length gradually decreases from its initial value to a trough, and then gradually recovers from the trough to near its initial value, the target change curve is identified. The U-shaped curve corresponds to the following scenario: the complete text in the image is temporarily obscured by a foreground object (such as a person or prop), causing the OCR to only recognize the unobscured portion of the text (text length reduction); when the foreground object is removed, the text in the image reappears completely (text length recovery).

[0145] S1103. If the target change curve is identified, perform the determination of pseudo-dynamic scene conditions.

[0146] The pseudo-dynamic scene verification conditions include: whether the similarity between the restored text and the original text meets a preset similarity threshold (e.g., 80%), that is, calculating the character-matching similarity between the restored complete text and the original complete text; and whether the text located in the trough of the target change curve (i.e., the reduced text during the occlusion period) is a substring of the stable text, where the stable text is the complete text content of the image text determined from the text content sequence. If the restored text and the original text are highly similar, and the reduced text in the trough period is exactly a substring of the complete text, then it proves that an occlusion restoration process does exist.

[0147] If all the above verification conditions are met, the current text on the screen is determined to be a pseudo-dynamic scene of partial occlusion followed by restoration. A static display mapping strategy is then adopted for the translated subtitles corresponding to the text on the screen. The static display mapping strategy does not generate animation configuration parameters corresponding to dynamic effects; that is, the translated subtitles are presented directly in a static manner, with only default fade-in and fade-out effects added, to avoid abnormal subtitle animations triggered by pseudo-dynamics caused by occlusion.

[0148] In this embodiment of the disclosure, by adding a U-shaped curve detection prediction step before determining the dynamic change type, it is possible to identify pseudo-dynamic scenes caused by foreground object occlusion. For such scenes, a static display strategy can be adopted to avoid erroneous triggering of dynamic effects, reduce the interference of abnormal motion effects on the viewing experience, and improve the rationality and accuracy of subtitle rendering.

[0149] Figure 12 These are interactive illustrations of translated subtitle display provided for some embodiments of this application. For example... Figure 12 As shown, in some embodiments, the controller 250 may run a video playback application, a subtitle translation service, an OCR tracking module, a motion effect classification and mapping module, and a window rendering service.

[0150] After a user triggers the text translation function on the video screen, the video playback application sends a request to the subtitle translation service to generate translated subtitles. The subtitle translation service calls the OCR tracking module to detect, recognize, and track the trajectory of consecutive video frames, generating trajectory data for the text on the screen and returning it. The subtitle translation service sends the trajectory data to the motion effect classification and mapping module, which, through feature extraction and multi-stage sequence determination, outputs the dynamic change type of the text on the screen. The subtitle translation service matches the corresponding mapping strategy, calls the motion effect classification and mapping module to perform cross-language motion effect mapping, and obtains the translated text and corresponding animation configuration parameters. The subtitle translation service then sends the translated text and animation configuration parameters to the window rendering service, which generates translated subtitles with dynamic effects. After completing the overlay rendering, the service controls the monitor 260 to output the video screen with the overlaid translated subtitles.

[0151] In some embodiments, when the user turns off the translation function, the video playback application sends a stop request to the subtitle translation service, the subtitle translation service cancels the processing task, the notification window rendering service removes the subtitle layer, and the monitor 260 resumes displaying the original video image.

[0152] Based on the display device 200 described above, some embodiments of this application also provide a method for displaying translated subtitles, which can be applied to the display device 200. For example... Figure 4 As shown, the method includes the following steps: S401. In the scenario of translating video footage, obtain the trajectory data of the text in the video footage.

[0153] The trajectory data includes the sequence of detection box positions and the sequence of text content in consecutive frames of the image.

[0154] S402. Perform feature extraction on the trajectory data to obtain the geometric change features of the detection boxes corresponding to the detection box position sequence and the text content change features corresponding to the text content sequence.

[0155] S403. Based on the preset order judgment rules, determine the dynamic change type of the text on the screen according to the text content change characteristics and the detection box geometric change characteristics.

[0156] S404. Based on the mapping strategy corresponding to the dynamic change type, map the dynamic effects of the text on the screen to the translated subtitles across languages ​​to obtain the animation configuration parameters.

[0157] S405. Generate translated subtitles with corresponding dynamic effects according to the animation configuration parameters, and control the monitor to overlay and render the translated subtitles onto the corresponding area of ​​the video screen.

[0158] Based on the above-mentioned method for displaying translated subtitles, it is possible to automatically recognize and cross-language map and render the text animation effects in the video screen without the need for manual configuration of animation parameters. It can restore the dynamic visual effects of the original text and improve the efficiency of subtitle production. At the same time, it ensures that the visual rhythm of the translated subtitles is consistent with that of the original screen, preserves the visual narrative design of the original video, and thus enhances the user's cross-language viewing immersion.

[0159] It should be noted that the method for displaying translated subtitles described in this application embodiment can adopt the same principle and implementation as the embodiment of the display device 200 described above, and will not be repeated here.

[0160] Similar parts between the embodiments provided in this application can be referred to mutually. The specific implementation methods provided above are only a few examples under the overall concept of this application and do not constitute a limitation on the scope of protection of this application. For those skilled in the art, any other implementation methods extended from the solution of this application without creative effort shall fall within the scope of protection of this application.

Claims

1. A display device, characterized in that, include: The monitor is configured to display video feeds; A controller, coupled to the display, is configured to: In the scenario of translating the video frame, the trajectory data of the text in the video frame is obtained; the trajectory data includes the sequence of detection box positions and the sequence of text content in consecutive frames. Feature extraction is performed on the trajectory data to obtain the geometric change features of the detection boxes corresponding to the detection box position sequence and the text content change features corresponding to the text content sequence; Based on a preset order determination rule, the dynamic change type of the text on the screen is determined according to the text content change characteristics and the detection box geometric change characteristics; Based on the mapping strategy corresponding to the dynamic change type, the dynamic effect of the text on the screen is mapped across languages ​​to the translated subtitles to obtain the animation configuration parameters; The system generates translated subtitles with corresponding dynamic effects based on the animation configuration parameters, and controls the display to overlay and render the translated subtitles onto the corresponding area of ​​the video screen.

2. The display device according to claim 1, characterized in that, The preset order determination rules include a first-stage determination rule, a second-stage determination rule, and a third-stage determination rule. The controller, based on a preset order determination rule, determines the dynamic change type of the text on the screen according to the text content change characteristics and the detection box geometric change characteristics, and is specifically configured as follows: The first-stage classification is performed by invoking the first-stage determination rule. If the first-stage classification fails to match the classification result, the second-stage judgment rule is invoked to perform the second-stage classification. If the second-stage classification fails to match the classification result, the third-stage judgment rule is invoked to perform the third-stage classification. If the third-stage classification does not match the classification result, then the dynamic change type is determined to be static occurrence type; The first stage determination rule includes performing prefix growth verification on the text in the image based on the text content change features. If the prefix growth verification passes, the dynamic change type is determined to be prefix growth type. The second stage determination rule includes performing area expansion determination based on the detection box geometric change features. If the preset area expansion condition is met, the dynamic change type is determined to be a physical scaling subclass under geometric change type, combined with the text content change features. The third-stage determination rule includes performing length growth determination based on the text content change features. If the preset length growth condition is met, the dynamic change type is determined to be either a prefix growth type or a lens zoom-out exposure type under the geometric change type, combined with the detection box geometric change features.

3. The display device according to claim 2, characterized in that, The controller performs prefix growth verification based on the text content change features, specifically configured as follows: Iterate through all target adjacent frame pairs whose text length increases in the text content sequence; The target adjacent frame pairs that satisfy the prefix relationship are marked as valid growing frame pairs, wherein satisfying the prefix relationship means that the text content of the later frame in the target adjacent frame pair is prefixed with the text content of the previous frame. Calculate the ratio of the number of effective growing frame pairs to the number of target adjacent frame pairs to obtain prefix consistency; Detect the number of characters incremented in a single step corresponding to the increase in the length of the text in the target adjacent frame pair; If the prefix consistency is greater than or equal to a preset ratio threshold, and the number of characters increased in a single step is less than or equal to a preset number of characters threshold, then the prefix growth verification is deemed successful.

4. The display device according to claim 2, characterized in that, The geometric change features of the detection frame include the area expansion ratio, aspect ratio change, and height increase of the detection frame corresponding to the text in the image. The area expansion ratio is the ratio of the maximum area of ​​the detection frame to the minimum area of ​​the detection frame. The text content change features include the increase in the text length corresponding to the text in the image. The controller performs an area expansion determination based on the geometric change features of the detection box; if a preset area expansion condition is met, it determines the dynamic change type as a physical scaling subclass under the geometric change type, based on the text content change features, specifically configured as follows: When the area expansion ratio of the detection frame is greater than or equal to the preset area threshold, it is determined that the preset area expansion condition is met. If the preset area expansion condition is met, and the multiple by which the text length increases is less than the preset growth threshold, the physical scaling subclass determination is performed; The determination of the physical scaling subclass includes the following: if the aspect ratio change of the detection box is greater than or equal to a preset aspect ratio threshold, or if the height increase of the detection box is greater than or equal to a preset height threshold, then the dynamic change type is determined to be a physical scaling subclass.

5. The display device according to claim 4, characterized in that, The geometric change features of the detection box include the increase in the width of the detection box corresponding to the text in the image; If the controller executes a preset length growth condition, it determines, based on the geometric change characteristics of the detection frame, that the dynamic change type is either a prefix-growth type or a lens zoom-out exposure subclass under a geometric change type, specifically configured as follows: When the multiple by which the text length increases is greater than or equal to the preset growth threshold, it is determined that the preset length growth condition is met; If the preset length growth condition is met, and the prefix growth verification is passed, the width growth of the detection box is greater than or equal to the preset width threshold, and the height growth of the detection box is less than or equal to the preset height stability threshold, then the dynamic change type is determined to be prefix growth type. If the preset length growth condition is met, and the prefix growth verification fails, and the width growth of the detection box is lower than the preset width threshold, then the dynamic change type is determined to be the lens zoom-out exposure subclass.

6. The display device according to claim 2, characterized in that, The animation configuration parameters include the presentation parameters of the translated subtitles, which are used to present a word-by-word incrementing effect. If the dynamic change type is a prefix-incrementing type, the controller executes a mapping strategy corresponding to the dynamic change type to map the dynamic effect of the on-screen text across languages ​​to the translated subtitles, thereby obtaining the animation configuration parameters, which are specifically configured as follows: Extract the first total number of characters of the image text from the text content sequence, and the timestamp of the first recognition of each source character in the image text; Based on the first total number of characters and the second total number of characters of the translated subtitles, the target number of characters to be displayed in the corresponding display stage of the translated subtitles is obtained by mapping the source characters to the translated subtitles according to the position ratio of the source characters in the text on the screen. The timestamp is used as the time boundary for the corresponding display stage of the translated subtitle; The rendering parameters are generated based on the target number of characters to be displayed and the corresponding time boundary.

7. The display device according to claim 2, characterized in that, The animation configuration parameters include the initial scaling ratio, scaling duration, layout position, displacement vector, and motion duration of the translated subtitles. If the dynamic change type is geometric change, the controller executes a mapping strategy corresponding to the dynamic change type to map the dynamic effect of the on-screen text across languages ​​to the translated subtitles, thereby obtaining the animation configuration parameters, which are specifically configured as follows: For the physical scaling subclass, the initial scaling ratio is calculated based on the area ratio of the detection boxes corresponding to the first and last frames in the detection box position sequence, and the scaling duration is determined based on the time span of the detection box position sequence. For the subclass exposed by the lens being pulled back, the target position of the detection box with the largest area in the detection box position sequence is determined as the layout position; If the detection frame has continuous displacement, the displacement vector and the motion duration are calculated based on the anchor point coordinates of the first and last positions of the detection frame.

8. The display device according to claim 1, characterized in that, Before determining the type of dynamic change in the on-screen text, the controller is also configured to: Detect the trend of text length change based on the aforementioned text content change features; Based on the changing trend, a target change curve is identified, which is a U-shaped change curve in which the text length first shrinks and then recovers. If the similarity between the target change curve, the restored text, and the original text is found to meet a preset similarity threshold, and the text located in the trough of the target change curve is identified as a substring of stable text, then a static display mapping strategy is adopted for the translated subtitles corresponding to the text on the screen. Wherein, the stable text is the complete text content of the on-screen text determined from the text content sequence; the static display mapping strategy is to not generate animation configuration parameters corresponding to dynamic effects.

9. The display device according to claim 1, characterized in that, The trajectory data also includes the recognition confidence sequence corresponding to the text in each frame; When performing feature extraction on the trajectory data, the controller is also configured to: Based on the detection box area change sequence determined by the detection box position sequence, and combined with the recognition confidence sequence, the life cycle state of the text on the screen is determined. The life cycle state includes a shrinking state where the area is continuously shrinking and a fading state where the confidence is continuously decreasing. When the controller executes the command to overlay and render the translated subtitles onto the corresponding area of ​​the video screen, it is specifically configured as follows: If the text on the screen is detected to be in the shrunk or faded state, the rendering priority of the corresponding translated subtitle is reduced, and the display is controlled to overlay the translated subtitle onto the corresponding area of ​​the video screen according to the rendering priority.

10. A method for displaying translated subtitles, characterized in that, include: In a scenario where video footage is translated, the trajectory data of the text in the video footage is obtained; The trajectory data includes the sequence of detection box positions and the sequence of text content in consecutive frames of the image; Feature extraction is performed on the trajectory data to obtain the geometric change features of the detection boxes corresponding to the detection box position sequence and the text content change features corresponding to the text content sequence; Based on a preset order determination rule, the dynamic change type of the text on the screen is determined according to the text content change characteristics and the detection box geometric change characteristics; Based on the mapping strategy corresponding to the dynamic change type, the dynamic effect of the text on the screen is mapped across languages ​​to the translated subtitles to obtain the animation configuration parameters; Based on the animation configuration parameters, translated subtitles with corresponding dynamic effects are generated, and the display is controlled to overlay and render the translated subtitles onto the corresponding area of ​​the video screen.

Images