Display apparatus and subtitle extraction method based on browser application

Abstract

The embodiment of the application provides a display device and a subtitle extraction method based on a browser application, a content provider is registered in the browser application, when a video media asset is played in an embedded playing mode in a target webpage, a resource address corresponding to the video media asset is identified from the target webpage, subtitle data corresponding to a current playing progress is extracted through a preset subtitle extraction mode, when the video media asset is switched from the embedded playing mode to an independent playing mode, the subtitle data is acquired from the content provider of the browser application through a content parser registered by a player, and is displayed on a player interface. Through the data interaction pipeline of the content provider and the content parser, the embodiment of the application realizes automatic extraction and seamless transmission of the subtitle data when the webpage video is switched from the embedded playing mode to the independent playing mode, avoids subtitle loss, and improves user watching experience.

Description

Technical Field

[0001] This application relates to the field of display device technology, and in particular to a display device and a method for extracting subtitles based on a browser application. Background Technology

[0002] Display devices are intelligent devices capable of presenting a user interface and supporting user interaction. Taking smart TVs as an example, smart TVs are television products based on Internet application technology, possessing an open operating system and chip, an open application platform, and enabling two-way human-computer interaction. They integrate multiple functions such as audio-visual entertainment and data to meet diverse and personalized user needs. Display devices can present different media asset recommendation interfaces, which may include multiple media asset options for users to select and play.

[0003] Display devices can provide users with a way to watch web videos through browsers. There are generally two ways to play web videos: one is to embed the video directly within the webpage using the browser's built-in rendering engine; the other is for the display device to obtain the resource address of the web video from the browser using video sniffing technology, and then pass the resource address to the display device's player for playback—this is the "independent playback" mode. Independent playback mode can utilize the player's stronger decoding capabilities and hardware acceleration capabilities to provide users with a smoother playback experience.

[0004] However, the subtitle data of web videos is presented on the web page according to the web page technology standard. The display device can only obtain the resource address of the web video, but cannot directly obtain the subtitle data of the web page. This causes users to manually download the subtitle data and import it through the local subtitle loading function of the player, which is complicated and has a poor user experience. Summary of the Invention

[0005] This application provides a display device and a subtitle extraction method based on a browser application to solve the problem that existing display devices cannot obtain webpage subtitle data when switching webpage videos from browser embedded playback mode to independent playback mode, resulting in subtitle loss.

[0006] In a first aspect, some embodiments of this application provide a display device, including: The monitor is configured to display the user interface; The controller is configured as follows: In response to a user's command to load a target webpage based on a browser application, the system controls the display to show the target webpage on the browser interface. The target webpage plays video media in an embedded playback mode, and the browser application has a registered content provider. Using a preset subtitle extraction method, subtitle data corresponding to the current playback progress of the video media asset is extracted from the target webpage. In response to the video media asset switching from embedded playback mode to standalone playback mode, the control monitor displays the video media asset image in the player interface. Standalone playback mode is a mode in which video media assets are played through a player other than a browser application. The player is registered with a content parser that interacts with the content provider. Based on the player's content parser, obtain subtitle data sent by the browser application through the content provider; Control the display to show subtitle data in the media asset screen of the player interface.

[0007] The above technical solutions have the following beneficial effects or advantages: By registering content providers in the browser application, subtitle data is automatically extracted when web videos are played in embedded playback mode. When the video media asset switches to independent playback mode, the content parser registered with the player obtains subtitle data from the content provider and displays it on the player interface. This achieves automatic extraction and seamless transmission of subtitle data when web videos switch from embedded playback mode to independent playback mode, avoiding the problem of subtitle loss due to the player's inability to obtain web subtitle data. Users do not need to manually download and import subtitles, thus improving the user viewing experience.

[0008] In some embodiments of this application, the subtitle extraction method includes a first extraction method, in which the controller extracts subtitle data corresponding to the current playback progress of the video media asset from the target webpage using a preset subtitle extraction method, specifically configured as follows: The target webpage is broken down into multiple object nodes based on the first extraction method; Calculate the caption prediction score for the object node. The caption prediction score is used to evaluate the probability that the object node is caption data. Object nodes whose predicted subtitle score is greater than or equal to a preset score threshold are marked as subtitle data corresponding to the current playback progress of the video media asset.

[0009] The above technical solutions have the following beneficial effects or advantages: By breaking down the target webpage into object nodes and calculating the subtitle prediction score, the subtitle data is automatically filtered based on the score threshold. This eliminates the need to customize rules for specific webpages, enabling the automatic extraction of subtitle data from the target webpage based on the subtitle prediction score, thus improving the adaptability and accuracy of subtitle extraction.

[0010] In some embodiments of this application, the controller performs caption prediction scoring for the object node, specifically configured as follows: The content change frequency score, caption feature score, and semantic rule score of the object node are calculated. The content change frequency score is used to evaluate the frequency of changes of the object node with video media assets, the caption feature score is used to evaluate the layout features of the text content contained in the object node, and the semantic rule score is used to evaluate the degree to which the text in the object node conforms to the common language expression logic. Based on preset weight ratios, a weighted sum is performed on the content change frequency score, subtitle feature score, and semantic rule score to obtain the subtitle prediction score.

[0011] The above technical solutions have the following beneficial effects or advantages: By comprehensively evaluating object nodes from three dimensions—content change frequency score, subtitle feature score, and semantic rule score—and weighting and summing them, multi-dimensional cross-validation improves the accuracy of subtitle container recognition, reduces the false judgment rate of non-subtitle nodes, and ensures the reliability of subtitle prediction score.

[0012] In some embodiments of this application, the subtitle extraction method includes a second extraction method, wherein the controller executes a preset subtitle extraction method to extract subtitle data corresponding to the current playback progress of the video media asset from the target webpage, specifically configured as follows: Extract a list of child nodes for video media assets from the target webpage based on the second extraction method; Detect subtitle track elements added to the child node list; When a subtitle track element is detected to be added to the child node list, the subtitle file address of the video media asset is read based on the source attribute corresponding to the subtitle track element; Extract subtitle data corresponding to the current playback progress of the video media asset from the subtitle file address.

[0013] The above technical solutions have the following beneficial effects or advantages: By detecting the subtitle track elements added to the child node list and reading the source attributes to obtain the subtitle file address, the system can automatically extract subtitles from video web pages that use standard subtitles. This can meet the subtitle extraction needs of web videos with more different architectures, improving versatility and stability.

[0014] In some embodiments of this application, the subtitle extraction method includes a third extraction method, in which the controller extracts subtitle data corresponding to the current playback progress of the video media asset from the target webpage using a preset subtitle extraction method, specifically configured as follows: Based on the third extraction method, monitor the subtitle track addition method corresponding to the video media asset; When the subtitle track addition method is called, record the newly created subtitle track object; When a change in subtitle prompts is detected in the subtitle track object, the set of active subtitle prompts for the subtitle track object is obtained; Extract subtitle data corresponding to the current playback progress of the video media asset from the active subtitle prompt set.

[0015] The above technical solution has the following beneficial effects or advantages: by intercepting the dynamic subtitle track addition method and listening for changes in subtitle prompts in the subtitle track object, the active subtitle prompt set of the subtitle track object is obtained, thereby extracting subtitle data from the active subtitle prompt set and improving the coverage of subtitle extraction.

[0016] In some embodiments of this application, after the controller executes a player-based content parser and obtains the subtitle data sent by the browser application through the content provider, it is further configured to: The browser application's content provider provides the player's content parser with subtitle track query interface and subtitle prompt query interface; In response to the track query request initiated by the content parser through the subtitle track query interface, the content provider returns a list of subtitle track information to the content parser. The list of subtitle track information includes multiple subtitle track information, and the subtitle track information includes track identifier and source type. In response to a subtitle prompt query request initiated by the content parser through the subtitle prompt query interface, the content provider returns subtitle prompt information within the target time window to the content parser. The subtitle prompt information includes at least the subtitle start time, subtitle end time, and subtitle text content. The target time window is a time interval defined centered on the current playback progress of the video media asset.

[0017] The above technical solution has the following beneficial effects or advantages: By sending standardized subtitle track query interface and subtitle prompt query interface through the content provider, the player obtains subtitle data according to the playback progress time window through the content parser in an inter-process communication manner, realizing a decoupled architecture between the browser and the player. The player does not need to care about the source of the subtitles and can obtain the subtitle data through the corresponding subtitle interface, thereby improving the efficiency of subtitle transmission.

[0018] In some embodiments of this application, the controller is further configured to: Set an authorization string between the browser application and the player so that the content parser can create the query addresses corresponding to the subtitle track query interface and the subtitle prompt query interface based on the authorization string; Alternatively, when the browser application launches the player in standalone playback mode, the authorization string is passed to the player so that the content parser can create a query address based on the received authorization string.

[0019] The above technical solutions have the following beneficial effects or advantages: by setting an authorization string between the browser application and the player, the content parser creates a query address based on the authorization string, realizing security verification of cross-process subtitle data access, preventing unauthorized applications from maliciously reading subtitle data, and supporting both static agreement and dynamic transmission authorization methods, thereby improving the security and flexibility of the subtitle transmission pipeline.

[0020] In some embodiments of this application, when multiple subtitle data extracted by different subtitle extraction methods are detected simultaneously on the target webpage, the controller is further configured to: Map multiple subtitle data to a unified timeline; Detect whether there is time window overlap in subtitle data corresponding to different subtitle extraction methods on a unified timeline; When the overlap of time windows meets the preset overlap conditions, deduplication processing is performed on the subtitle data with overlapping time windows based on the preset priority order, where the priority order represents the order of confidence of the subtitle data corresponding to different subtitle extraction methods.

[0021] The above technical solution has the following beneficial effects or advantages: by mapping subtitles from multiple sources to a unified timeline and detecting overlapping time windows, and performing deduplication processing on overlapping subtitles based on priority order, it avoids the problem of repeated subtitle display when multiple subtitle extraction methods hit the same target at the same time, and ensures the uniqueness and accuracy of subtitle data.

[0022] In some embodiments of this application, the controller is further configured to: In the absence of a language identifier in the subtitle data, a character set range detection is performed on the subtitle data. The language identifier is used to indicate the language of the subtitle data. If the subtitle data contains text characters of a preset type, the subtitle data is used to perform language recognition through a preset language model to obtain the language recognition result; The language recognition results are labeled in the language identifier field of the subtitle data.

[0023] The above technical solution has the following beneficial effects or advantages: By combining character set range detection with a preset language model, it automatically recognizes the language of subtitle data without language identifiers and marks the recognition results in the language identifier field, thereby realizing automatic labeling of subtitle languages. This makes it easier for the player to automatically select the subtitle track according to the user's language preference and improves the convenience of subtitle management.

[0024] Secondly, some embodiments of this application provide a subtitle extraction method based on a browser application, including: In response to a user's command to load a target webpage based on a browser application, the display is controlled to show the target webpage in the browser interface, where the target webpage plays video media in an embedded playback mode, and the browser application is registered with a content provider. Using a preset subtitle extraction method, subtitle data corresponding to the current playback progress of the video media asset is extracted from the target webpage. In response to the video media asset switching from embedded playback mode to standalone playback mode, the control monitor displays the video media asset image in the player interface. Standalone playback mode is a mode in which video media assets are played through a player other than a browser application. The player is registered with a content parser that interacts with the content provider. Based on the player's content parser, obtain subtitle data sent by the browser application through the content provider; Control the display to show subtitle data in the media asset screen of the player interface.

[0025] As can be seen from the above technical solutions, this application provides a display device and a subtitle extraction method based on a browser application. By registering a content provider in the browser application, when video media is played in embedded playback mode on a target webpage, the resource address corresponding to the video media is identified from the target webpage, and subtitle data corresponding to the current playback progress is extracted using a preset subtitle extraction method. When the video media switches from embedded playback mode to independent playback mode, the subtitle data is obtained from the content provider in the browser application through a content parser registered with the player and displayed on the player interface. This application, through the data interaction channel between the content provider and the content parser, achieves automatic extraction and seamless transmission of subtitle data when webpage videos switch from embedded playback mode to independent playback mode, avoiding subtitle loss and improving the user viewing experience. Attached Figure Description

[0026] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0027] Figure 1 This is a schematic diagram illustrating an operational scenario between a display device and a control device provided in some embodiments of this application; Figure 2 This is a schematic diagram of the hardware configuration of a display device provided in some embodiments of this application; Figure 3 This is a schematic diagram of the software configuration of a display device provided in some embodiments of this application; Figure 4A flowchart illustrating a browser application-based subtitle extraction method performed by a display device according to some embodiments of this application; Figure 5 Timing diagrams showing the execution of a browser application-based subtitle extraction method on a display device provided in some embodiments of this application; Figure 6 A flowchart illustrating how a display device extracts subtitle data using a first extraction method, as provided in some embodiments of this application; Figure 7 A flowchart for calculating caption prediction scores for a display device provided in some embodiments of this application; Figure 8 A flowchart illustrating how a display device extracts subtitle data using a second extraction method, as provided in some embodiments of this application; Figure 9 A flowchart illustrating how a display device extracts subtitle data using a third extraction method, as provided in some embodiments of this application; Figure 10 A flowchart illustrating how a display device obtains subtitle data via a content parser, as provided in some embodiments of this application; Figure 11 A flowchart illustrating the multi-source caption deduplication process performed by a display device according to some embodiments of this application; Figure 12 A flowchart illustrating the subtitle language recognition process performed by a display device according to some embodiments of this application. Detailed Implementation

[0028] The embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described below do not represent all embodiments consistent with this application. They are merely examples of systems and methods consistent with some aspects of this application as detailed in the claims.

[0029] It should be noted that the brief descriptions of terms in this application are only for the convenience of understanding the embodiments described below, and are not intended to limit the embodiments of this application. Unless otherwise stated, these terms should be understood in their ordinary and common meaning.

[0030] The terms "first," "second," "third," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar or related objects or entities, and do not necessarily imply a specific order or sequence, unless otherwise specified. It should be understood that such terms are interchangeable where appropriate.

[0031] The terms “comprising” and “having”, and any variations thereof, are intended to cover but not exclude inclusion, for example, a product or device that includes a range of components is not necessarily limited to all of the components that are clearly listed, but may include other components that are not clearly listed or that are inherent to such product or device.

[0032] The term "module" refers to any known or subsequently developed hardware, software, firmware, artificial intelligence, fuzzy logic, or combination of hardware and / or software code that is capable of performing the functions associated with that element.

[0033] In this embodiment, the display device 200 generally refers to a device with screen display and data processing capabilities. For example, the display device 200 includes, but is not limited to, smart TVs, mobile terminals, computers, monitors, advertising screens, wearable devices, virtual reality devices, augmented reality devices, etc.

[0034] Figure 1 This is a schematic diagram illustrating an operational scenario between a display device and a control device provided in some embodiments of this application. For example... Figure 1 As shown, a user can operate the display device 200 via touch operation, a mobile terminal 300, and a control device 100. The control device 100 receives user input commands and converts them into control commands that the display device 200 can recognize and respond to. For example, the control device 100 can be a remote control, a stylus, a gamepad, etc.

[0035] The mobile terminal 300 can function as a control device for human-computer interaction between the user and the display device 200. It can also function as a communication device for establishing a communication connection with the display device 200 and exchanging data. In some embodiments, the mobile terminal 300 can have software applications installed on it and communicate with the display device 200 via network communication protocols to achieve one-to-one control and data communication. Furthermore, it can transmit audio and video content displayed on the mobile terminal 300 to the display device 200 for synchronized display.

[0036] In some embodiments, the mobile terminal 300 or other electronic devices may also simulate the functions of the control device 100 by running an application that controls the display device 200.

[0037] like Figure 1 The diagram also shows that the display device 200 communicates with the server 400 via various communication methods. This allows the display device 200 to communicate via a local area network (LAN), a wireless local area network (WLAN), and other networks.

[0038] Display device 200 can provide broadcast television reception function, and can also be equipped with intelligent network television function that provides computer support function, including but not limited to network television, smart television, Internet Protocol television (IPTV), etc.

[0039] Figure 2 Provided for some embodiments of this application Figure 1 Hardware configuration block diagram of display device 200.

[0040] In some embodiments, the display device 200 may include at least one of a tuner 210, a communication device 220, a detector 230, a device interface 240, a controller 250, a display 260, an audio output device 270, a memory, a power supply, and a user input interface 280.

[0041] In some embodiments, detector 230 is used to acquire signals from the external environment or to interact with the outside world. For example, detector 230 includes a light receiver, a sensor for acquiring ambient light intensity; or, detector 230 includes an image acquisition device, such as a camera, which can be used to acquire external environmental scenes, user attributes, or user interaction gestures; or, detector 230 includes a sound acquisition device, such as a microphone, for receiving external sounds.

[0042] In some embodiments, the display 260 includes display function components for presenting images and driving components for driving image display. The display 260 is used to receive and display image signals output from the controller 250. For example, the display 260 can be used to display video content, image content, menu control interface components, and user control UI interfaces, etc.

[0043] In some embodiments, the communication device 220 is a component used to communicate with external devices or the server 400 according to various communication protocol types. The display device 200 may have multiple communication devices 220 depending on the supported communication methods. For example, when the display device 200 supports wireless network communication, it may have a communication device 220 with WiFi functionality. When the display device 200 supports Bluetooth connectivity, it needs to have a communication device 220 with Bluetooth functionality.

[0044] The communication device 220 enables the display device 200 to communicate with external devices or the server 400 via wireless or wired connections. Wired connections utilize data cables, interfaces, or other components to connect the display device 200 to external devices. Wireless connections utilize wireless signals or wireless networks. The display device 200 can directly establish a connection with external devices or indirectly through gateways, routers, or other connection devices.

[0045] In some embodiments, the controller 250 may include at least one of a central processing unit, a video processor, an audio processor, a graphics processor, and a power processor, and a first to an nth interface for input / output. The controller 250 controls the operation of the display device and responds to user operations through various software control programs stored in memory. The controller 250 controls the overall operation of the display device 200.

[0046] In some embodiments, the controller 250 and the tuner 210 may be located in different separate devices, that is, the tuner 210 may also be located in an external device of the main device where the controller 250 is located, such as an external set-top box.

[0047] In some embodiments, a user can input user commands through a graphical user interface (GUI) displayed on a display 260, and the user input interface receives user input commands through the graphical user interface (GUI).

[0048] In some embodiments, the audio output device 270 can be a built-in speaker of the display device 200 or an external audio output device connected to the display device 200. For the external audio output device connected to the display device 200, the display device 200 may also be provided with an external audio output terminal, through which the audio output device can be connected to the display device 200 to output sound from the display device 200.

[0049] In some embodiments, the user input interface 280 can be used to receive instructions from user input.

[0050] To enable user interaction, in some embodiments, the display device 200 may run an operating system. The operating system is a computer program used to manage and control the hardware and software resources of the display device 200. The operating system can control the display device to provide a user interface; for example, the operating system can directly control the display device to provide a user interface, or it can provide a user interface by running an application. The operating system also allows users to interact with the display device 200.

[0051] It should be noted that the operating system can be a native operating system based on a specific operating platform, a third-party operating system that is deeply customized based on a specific operating platform, or an independent operating system specifically developed for display devices.

[0052] An operating system can be divided into different modules or levels based on the functions it implements, for example... Figure 3As shown, in some embodiments, the system is divided into four layers, from top to bottom: the Applications layer (referred to as the "Application Layer"), the Application Framework layer (referred to as the "Framework Layer"), the System Library layer, and the Kernel layer.

[0053] In some embodiments, the application layer provides services and interfaces for applications, enabling the display device 200 to run applications and interact with the user based on the applications. The application layer may contain at least one application, which may be a built-in Windows program, system settings program, or clock program of the operating system; or it may be an application developed by a third-party developer. In specific implementations, the application packages in the application layer are not limited to the examples above.

[0054] The framework layer provides application programming interfaces (APIs) and a programming framework for applications. The application framework layer includes predefined functions. It acts as a central processing unit, determining the actions taken by applications within the application layer. Through the API, applications can access system resources and obtain system services during execution.

[0055] like Figure 3 As shown, the application framework layer in this embodiment includes a view system, managers, and content providers. The view system designs and implements the application's interface and interactions, and includes lists, grids, text boxes, and buttons. The managers include at least one of the following modules: an activity manager for interacting with all running activities in the system; a location manager for providing system services or applications with access to system location services; a package manager for retrieving various information related to application packages currently installed on the device; a notification manager for controlling the display and clearing of notification messages; and a window manager for managing icons, windows, toolbars, wallpapers, and desktop widgets on the user interface.

[0056] In some embodiments, the Activity Manager manages the lifecycle of individual applications and common navigation and back functions, such as controlling application exit, opening, and back actions. The Window Manager manages all window programs, such as obtaining the screen size, determining if a status bar is present, locking the screen, capturing the screen, and controlling changes to the display window, such as shrinking the display window, shaking the display, or distorting the display.

[0057] In some embodiments, the system runtime library layer can provide support for the framework layer. When the framework layer is used, the operating system runs the instruction library contained in the system runtime library layer, such as the C / C++ instruction library, to implement the functions to be performed by the framework layer.

[0058] In some embodiments, the kernel layer is a functional layer situated between the hardware and software of the display device 200. The kernel layer can implement functions such as hardware abstraction, multitasking, and memory management. For example, ... Figure 3 As shown, hardware drivers can be configured in the kernel layer. The kernel layer can contain at least one of the following drivers: audio driver, display driver, Bluetooth driver, camera driver, WIFI driver, USB driver, HDMI driver, sensor driver (such as fingerprint sensor, temperature sensor, pressure sensor, etc.), and power driver, etc.

[0059] It should be noted that the above examples are merely a simple division of operating system functions and do not limit the specific form of the operating system of the display device 200 in this application embodiment. Depending on the function of the display device, the type of operating system, and other factors, the number of levels and the specific level type of the operating system may be expressed in other forms.

[0060] In some embodiments, the display device 200 can provide users with a channel to watch web videos through a browser application. There are generally two ways to play web videos: one is to embed the video directly into the web page using the browser's built-in web page rendering engine; the other is for the display device 200 to obtain the resource address of the web video from the browser using video sniffing technology and then pass the resource address to the player on the display device 200 for playback, i.e., independent playback mode. Independent playback mode can utilize the player's stronger decoding capabilities and hardware acceleration capabilities to provide users with a smoother playback experience.

[0061] However, since the subtitle data of web videos is presented on the web page according to the web page technology standard, the display device 200 can only obtain the resource address of the web video, but cannot directly obtain the subtitle data of the web page. This causes users to manually download the subtitle data and import it through the local subtitle loading function of the player, which is complicated and results in a poor user experience.

[0062] Based on the above technical problems, some embodiments of this application provide a display device 200, which includes at least a display 260 and a controller 250, wherein the display 260 is configured to display a user interface.

[0063] Figure 4 A flowchart illustrating a browser application-based subtitle extraction method performed by a display device according to some embodiments of this application. Figure 5 Timing diagram of a display device performing a browser application-based subtitle extraction method according to some embodiments of this application.

[0064] See Figure 4 and Figure 5 The controller 250 is configured to perform the following steps: S410: In response to a user's command to load a target webpage based on a browser application, control the display to show the target webpage on the browser interface.

[0065] In some embodiments, users can input loading instructions for the target webpage into the display device 200 via a remote control, mobile terminal, or voice commands.

[0066] In some embodiments, after receiving a loading instruction, the controller 250 controls the browser application to load the target webpage and display it on the monitor 260. The target webpage contains video media assets, which are played in an embedded playback mode within the target webpage, i.e., the video media assets are played directly within the webpage through the browser's built-in webpage rendering engine.

[0067] In some embodiments, when the browser application starts, the controller 250 registers a content provider in the browser application. The content provider is a cross-process data sharing component used to send caption data from within the browser application to other application processes for access. The content provider is registered with a pre-defined authorization string, which serves as an access identifier for other applications to query the caption data.

[0068] In some embodiments, the controller 250 identifies the resource address corresponding to the video media asset from the target webpage using video sniffing technology. Specifically, the controller 250 can inject a script, such as a JS (JavaScript) script, during the loading process of the target webpage. The JS script can identify the Media Resource Uniform Resource Locator (URL) of the video media asset in the target webpage, and then parse the URL to obtain the resource address of the video media asset.

[0069] For example, the resource address can be a streaming media address in m3u8 format, a file address in mp4 format, or a media resource address in other formats. After recognizing the resource address, the controller 250 can cache the resource address to local storage so that when the video media asset switches to independent playback mode, the resource address can be passed to the player of the display device 200, thereby allowing the player to parse the video media asset from the resource address and perform subsequent playback.

[0070] S420: Extracts subtitle data from the target webpage that corresponds to the current playback progress of the video media asset using a preset subtitle extraction method.

[0071] In some embodiments, the subtitle extraction method is based on preset rules that can extract subtitle data from web pages, specifically divided into two scenarios: web page tag extraction and interface data extraction. For example, when subtitle data is directly written into the web page tag elements of the target web page, the controller 250 can directly obtain the subtitle text content within the tag elements via JavaScript, and, combined with preset timestamp format rules, extract subtitle data corresponding to different playback progresses. When subtitle data is asynchronously loaded into the target web page via an independent interface, the controller 250 can intercept interface requests through injected scripts, obtain the complete subtitle data returned by the interface, and then parse the subtitle segments corresponding to the playback progress according to the corresponding subtitle format. After extracting the subtitle data, the controller exposes the subtitle data to the outside world through a registered content provider, allowing other application processes to call and obtain it.

[0072] In some embodiments, the target webpage may correspond to multiple subtitle extraction methods, and the controller 250 can automatically select the appropriate subtitle extraction method according to the subtitle presentation method of the target webpage.

[0073] In some embodiments, before extracting subtitle data, the controller 250 can obtain the current playback progress of the video media asset, thereby extracting subtitle data corresponding to the current playback progress of the video media asset from the target webpage. Multiple subtitle extraction methods can also be used simultaneously to improve the coverage of subtitle extraction.

[0074] S430: In response to the video media switching from embedded playback mode to standalone playback mode, control the display to show the video media's image on the player interface.

[0075] In some embodiments, the stand-alone playback mode is a mode in which video media is played through a player other than a browser application. The triggering conditions for switching video media from embedded playback mode to stand-alone playback mode may include: the user manually selecting stand-alone playback mode, the controller 250 detecting that the playback quality in embedded playback mode is lower than a preset quality threshold and automatically triggering the switch, or the controller 250 detecting that the player has superior decoding capabilities and automatically triggering the switch.

[0076] Taking the user's manual selection of the independent playback mode as an example, when switching to the independent playback mode, the controller 250 starts the player and passes the resource address of the video media asset to the player so that the player can obtain the video media stream based on the resource address and decode and play it.

[0077] In some embodiments, when the player starts, the controller 250 registers a content parser in the player. The content parser is a component used to query and retrieve data from a content provider. The content parser establishes a data interaction channel with the content provider based on the content provider's authorization string.

[0078] S440: A player-based content parser that retrieves subtitle data sent by the browser application through the content provider.

[0079] In some embodiments, the player's content parser constructs a query address based on an authorization string and initiates a subtitle data query request to the browser application's browser application using the query address. In response to the query request, the browser application retrieves subtitle data matching the query request from the memory data structure maintained by the browser application's subtitle engine through the content provider, encapsulates the subtitle data into structured data, and returns it to the content parser. The content parser then parses the structured data to obtain the subtitle data.

[0080] In some embodiments, the content parser continuously queries subtitle data according to the time window of the playback progress to achieve real-time synchronized display of subtitles.

[0081] S450: Controls the display to show subtitle data on the media asset screen of the player interface.

[0082] In some embodiments, after acquiring the subtitle data, the controller 250 controls the display 260 to render and display the subtitle text content in the media asset screen of the player interface when the video media asset is played to the corresponding time point, based on the subtitle start time and subtitle end time of the subtitle data.

[0083] In some embodiments, the controller 250 can automatically select a subtitle track that matches the user's language preference based on the language identifier in the subtitle data, or it can switch subtitle tracks based on the user's manual selection. When displaying subtitle data, the controller 250 can also adjust the display style of the subtitles according to the source type of the subtitle data, for example, maintaining the original style of the subtitle data in the target webpage, or using the player's default subtitle style.

[0084] Based on the above technical solutions, this application embodiment registers a content provider in the browser application, automatically extracts subtitle data when the web video is played in embedded playback mode, and obtains and displays subtitle data from the content provider through the content parser registered by the player when the video media asset switches to independent playback mode. This realizes the automatic extraction and seamless transmission of subtitle data when the web video switches from embedded playback mode to independent playback mode, avoids the problem of subtitle loss due to the player's inability to obtain web subtitle data, eliminates the need for users to manually download and import subtitles, and improves the user's viewing experience.

[0085] Figure 6 A flowchart illustrating how a display device, according to some embodiments of this application, extracts subtitle data using a first extraction method.

[0086] See Figure 6 In some embodiments, the subtitle extraction method includes a first extraction method. The controller 250 executes step S420, which involves extracting subtitle data corresponding to the current playback progress of the video media asset from the target webpage using a preset subtitle extraction method. This may include the following steps: S610: Based on the first extraction method, the target webpage is decomposed into multiple object nodes.

[0087] In some embodiments, the controller 250 traverses the Document Object Model (DOM) of the target webpage, breaking the target webpage down into multiple object nodes (DOM nodes). Specifically, the controller 250 starts with the video element corresponding to the video media asset, traverses its parent container and the subtree of its sibling elements, and treats each DOM node in the subtree as an object node.

[0088] In some embodiments, object nodes include types such as element nodes, text nodes, and attribute nodes. By decomposing the target webpage into object nodes, the controller 250 can analyze each object node independently to determine whether it is a subtitle container that carries subtitle rendering functionality.

[0089] In one implementation, controller 250 can inject a script early in the target webpage loading process to register a global change observer (MutationObserver) in the browser application. When a change in a node is observed, the controller can then perform subsequent calculations on the object node to predict caption scores.

[0090] S620: Calculate the caption prediction score for the object node.

[0091] In some embodiments, the controller 250 calculates a caption prediction score for each object node from multiple dimensions, wherein the caption prediction score is used to evaluate the probability that the object node is caption data.

[0092] Specifically, the caption prediction scoring comprehensively considers factors such as the content change frequency, layout features, and semantic rules of the object node to quantitatively assess the probability that the object node is a caption container. Content change frequency evaluates whether the text content of the object node changes synchronously with the video playback progress; caption containers typically have a high change frequency and are highly correlated with the video timestamp. Layout features evaluate whether the Cascading Style Sheet (CSS) attributes of the object node conform to typical caption layout patterns, such as absolute positioning, bottom alignment, text centering, and text shadow. Semantic rules evaluate whether the attribute naming and text content of the object node contain caption-related semantic keywords. Controller 250 can improve the accuracy of caption container recognition through multi-dimensional comprehensive scoring.

[0093] S630: Mark object nodes whose predicted subtitle score is greater than or equal to a preset score threshold as subtitle data corresponding to the current playback progress of the video media asset.

[0094] In some embodiments, the controller 250 compares the caption prediction score for each object node with a preset scoring threshold. The scoring threshold is an empirical value calibrated through experiments; for example, the caption prediction score can be normalized to a range of 0 to 1, and the scoring threshold can be set to 0.6.

[0095] In some embodiments, when the subtitle prediction score of an object node is greater than or equal to a score threshold, the controller 250 determines that the object node is a subtitle container and marks it as subtitle data corresponding to the current playback progress of the video media asset. It should be noted that since the video media asset of the target webpage changes dynamically during playback, the subtitle prediction score calculation in step S620 is based on the current playback progress of the video media asset, and the corresponding subtitle data also corresponds to the current playback progress of the video media asset.

[0096] In another embodiment, after marking the subtitle data, the controller 250 starts a timed acquisition loop, acquiring text snapshots of the subtitle container at preset time intervals (e.g., 100 milliseconds). The text snapshots are compared with the text snapshots from the previous cycle; if the text changes, new subtitle data is recorded. The subtitle data includes the subtitle text content, start time, and end time. The start time is set to the moment corresponding to the current video playback progress, and the end time is determined when the next subtitle data is generated. Then, the controller 250 exposes the acquired subtitle data to the player through a browser application via the content provider, enabling the player to update the subtitle data in real time according to the playback progress of the video media asset.

[0097] Based on the above technical solutions, this application embodiment decomposes the target webpage into object nodes and calculates the subtitle prediction score. The subtitle data is automatically filtered based on the score threshold, without the need to customize rules for specific webpages. This achieves automatic extraction of subtitle data from the target webpage based on the subtitle prediction score, thereby improving the adaptability and accuracy of subtitle extraction.

[0098] Figure 7 A flowchart illustrating the calculation of caption prediction scores for a display device provided in some embodiments of this application.

[0099] See Figure 7 In some embodiments, the controller 250 performs step S620, which involves calculating the caption prediction score for the object node, and may include the following steps: S710: Calculate the content change frequency score, subtitle feature score, and semantic rule score of the object node.

[0100] In some embodiments, the content change frequency score is used to evaluate the frequency with which object nodes change as video media assets change. Since the rhythm of subtitle text changes is closely related to the video playback progress, the content change frequency score is obtained by calculating the Pearson correlation coefficient between object node text changes and video playback progress.

[0101] Specifically, the controller 250 records the number of changes in the text content of the object node within a preset sliding time window (e.g., 2 seconds), and simultaneously records the change increment of the current playback time of the video media asset, constructs a text change count vector and a playback time increment vector, and calculates the Pearson correlation coefficient between the two.

[0102] In some embodiments, the formula for calculating the correlation coefficient can be expressed as follows: r = cov(X,Y) / (σX·σY); Where r is the Pearson correlation coefficient, ranging from -1 to 1; cov(X,Y) is the covariance of the text change count vector X and the playback time increment vector Y; σX is the standard deviation of the text change count vector X; and σY is the standard deviation of the playback time increment vector Y. The closer the correlation coefficient is to 1, the more synchronized the text changes of the object node are with the video playback progress, and the higher the probability that the object node is subtitle data.

[0103] In one implementation, when the correlation coefficient is greater than a preset correlation threshold (e.g., 0.7), the content change frequency score is set to full marks; otherwise, it is linearly mapped to the range of 0 to full marks.

[0104] The subtitle feature score is used to evaluate the layout features of text content contained in an object node. The subtitle feature score is obtained by feature matching based on the style calculated by the Cascading Style Sheet of the object node. Matching items include positioning attributes of fixed or absolute positioning, bottom attribute value less than the container height by a preset percentage (e.g., 20%), text alignment of center alignment, non-empty text shadow, semi-transparent background color, high-contrast text color, font size exceeding a preset font size threshold, and high layer index value (e.g., above the layer index value of video elements). The score is accumulated for each matching item, and then the score is normalized to obtain the subtitle feature score.

[0105] Semantic rule scoring is used to evaluate the degree to which the text within an object node conforms to the logic of common language expression. Semantic rule scoring is obtained by keyword matching of the object node's identifier, class name, accessibility label, and custom attributes. Matched keywords include subtitles, subtitle groups, explanatory text, dialogue, translation, etc. The score is accumulated for each matched keyword, and then the score is normalized to obtain the semantic rule score.

[0106] S720: Based on the preset weight ratio, the content change frequency score, subtitle feature score and semantic rule score are weighted and summed to obtain the subtitle prediction score.

[0107] In some embodiments, the controller 250 performs a weighted summation of the scores for the three dimensions based on preset weight ratios. The weight ratios reflect the importance of each dimension in caption container recognition.

[0108] In one implementation, the weight of the content change frequency score is 0.5, the weight of the subtitle feature score is 0.3, and the weight of the semantic rule score is 0.2. The content change frequency score has the highest weight because the correlation between the rhythm of subtitle text changes and the video playback progress is the most essential feature of the subtitle container. The subtitle feature score serves as an auxiliary dimension, narrowing down the candidate range through layout style features. The semantic rule score serves as a supplementary dimension, providing additional identification labels through attribute naming conventions.

[0109] In some embodiments, the weighted summation formula is: The subtitle prediction score is calculated as follows: 0.5 × Content Change Frequency Score + 0.3 × Subtitle Feature Score + 0.2 × Semantic Rule Score, where the subtitle prediction score ranges from 0 to 1. This weighted sum is then compared with a scoring threshold to determine whether an object node is a subtitle container.

[0110] Based on the above technical solutions, this application embodiment comprehensively evaluates and weights the object nodes from three dimensions: content change frequency, subtitle layout features, and semantic rules. Multi-dimensional cross-validation improves the accuracy of subtitle container recognition, reduces the misjudgment rate of non-subtitle nodes, and ensures the reliability and robustness of subtitle prediction scoring.

[0111] Figure 8 A flowchart illustrating how a display device, according to some embodiments of this application, extracts subtitle data using a second extraction method.

[0112] See Figure 8 In some embodiments, the subtitle extraction method includes a second extraction method. The controller 250 executes step S420, which involves extracting subtitle data corresponding to the current playback progress of the video media asset from the target webpage using a preset subtitle extraction method. This may include the following steps: S810: Extract a list of child nodes for video media assets from the target webpage based on the second extraction method.

[0113] In some embodiments, the controller 250, based on a second extraction method, listens to the list of child nodes of video elements on the target webpage through an injected script. When it detects that a video element has been added to the document object model of the target webpage, the controller 250 extracts the list of child nodes of the video media asset from the target webpage.

[0114] The list of child nodes may contain various child nodes such as resource elements and track elements. Controller 250 iterates through the list of child nodes, checking the tag type of each child node to identify whether the target webpage contains a subtitle track element.

[0115] S820: Detects subtitle track elements added to the child node list.

[0116] In some embodiments, controller 250 detects whether a track element exists in the list of child nodes. The track element is a tag in the HTML5 standard used to specify media file captions, caption titles, descriptions, chapters, or metadata.

[0117] When the target webpage uses the W3C-recommended standard subtitle scheme, it adds a `track` element to the child node list of the video element and specifies the subtitle data address and data type through the attributes of the `track` element. Based on this, the controller 250 can detect the added `track` element in the child node list of the target webpage.

[0118] S830: When a subtitle track element is detected to be added to the child node list, the subtitle file address of the video media asset is read based on the source attribute corresponding to the subtitle track element.

[0119] In some embodiments, when a track element is detected to be added to the child node list, the controller 250 parses the track element and reads the src attribute (source attribute) of the track element to obtain and read the Uniform Resource Locator (URL) of the subtitle data, i.e., the address of the subtitle file.

[0120] In some embodiments, the controller 250 can also read the kind attribute (type attribute) of the track element to determine the subtitle type (such as subtitles type, captions type, etc.), and the controller 250 can also read the srclang attribute (language attribute corresponding to the subtitle track element) to determine the language of the subtitle.

[0121] S840: Extract subtitle data corresponding to the current playback progress of the video media asset from the subtitle file address.

[0122] In some embodiments, the controller 250 can read the subtitle file address and extract subtitle data corresponding to the current playback progress of the video media asset from the subtitle file address.

[0123] In some embodiments, the controller 250 can determine the subtitle format based on the file extension or content type of the subtitle data, and use the parser corresponding to the subtitle format to parse the original content of the subtitle data into a standard subtitle prompt structure. Each subtitle prompt structure includes a start time, an end time, and text content.

[0124] In one implementation, the controller 250 filters subtitle data corresponding to the current playback progress from the parsed subtitle prompt sequence (including multiple subtitle prompt structures), i.e., subtitle data whose start time is less than or equal to the current playback progress and whose end time is greater than the current playback progress. The extracted subtitle data is exposed to the player through the content provider so that the player can synchronously display subtitles while playing the video media asset.

[0125] Based on the above technical solutions, this application embodiment obtains the subtitle file address by detecting the addition event of the track element and reading the source attribute, thereby realizing automatic subtitle extraction for target websites that adopt standard subtitle schemes and improving the standardization and compatibility of subtitle extraction.

[0126] Figure 9 A flowchart illustrating how a display device, according to some embodiments of this application, extracts subtitle data using a third extraction method.

[0127] See Figure 9 In some embodiments, the subtitle extraction method includes a third extraction method. The controller 250 executes step S420, which involves extracting subtitle data corresponding to the current playback progress of the video media asset from the target webpage using a preset subtitle extraction method. This may include the following steps: S910: A method for adding subtitle tracks corresponding to video media assets based on a third extraction method.

[0128] In some embodiments, during the early stages of target webpage loading, controller 250 listens for the subtitle track addition method (addTextTrack method) corresponding to the video media asset based on a third extraction method. The addTextTrack method is an application programming interface (API) in the HTML5 standard used to dynamically create subtitle tracks. Some video webpages dynamically load subtitles by calling the addTextTrack method via JavaScript.

[0129] Controller 250 saves the original addTextTrack method as a backup and replaces it with the injected interceptor function's addTextTrack method. When the target webpage's script calls the addTextTrack method to intercept video media assets, the interceptor function first executes the recording logic, and then calls the original addTextTrack method to intercept the video media assets.

[0130] S920: When the subtitle track addition method is called, record the newly created subtitle track object.

[0131] In some embodiments, when the controller 250 detects that the interception function has called the addTextTrack method, it obtains the newly created TextTrack (subtitle track) object returned by the addTextTrack method. The TextTrack object contains attributes such as the track's type label and language.

[0132] Simultaneously, controller 250 registers an oncuechange event listener for the newly created TextTrack object to monitor whether the subtitle track object undergoes a subtitle change. The subtitle change event is triggered when the active subtitle set of the subtitle track is updated, that is, when a new subtitle becomes active or an existing subtitle becomes inactive.

[0133] S930: When a change in subtitle prompts is detected in the subtitle track object, obtain the set of active subtitle prompts for the subtitle track object.

[0134] In some embodiments, when a caption change event is triggered, the controller 250 obtains the currently active caption set through the `activeCues` property of the `TextTrack` object. The active caption set contains all caption objects that should currently be displayed. Each caption object contains attributes such as the caption's start time, end time, and text content. The controller 250 iterates through the active caption set, extracts the attribute information of each caption object, and constructs structured caption data.

[0135] S940: Extract subtitle data corresponding to the current playback progress of the video media asset from the active subtitle prompt set.

[0136] In some embodiments, the controller 250 extracts subtitle data from the active subtitle prompt set, including subtitle text content, subtitle start time, and subtitle end time. Since the extraction from the active subtitle prompt set is automatically updated based on the current playback progress, the extracted subtitle data corresponds to the current playback progress of the video media asset.

[0137] The controller 250 exposes the extracted subtitle data to the player through the content provider, and notifies the player's content parser to call the new subtitle data for display when new subtitle data is generated, thereby realizing real-time push of subtitle data.

[0138] Based on the above technical solutions, this application embodiment improves the coverage of subtitle extraction by intercepting the dynamic subtitle track addition method and listening for changes in subtitle prompts in the subtitle track object.

[0139] Figure 10 This is a flowchart illustrating how a display device, according to some embodiments of this application, acquires subtitle data via a content parser.

[0140] See Figure 10 In some embodiments, after executing step S440, which involves the controller 250 obtaining subtitle data sent by the browser application through the content provider based on the player's content parser, it is further configured to perform the following steps: S1010: Provides the subtitle track query interface and subtitle prompt query interface to the player's content parser through the content provider of the browser application.

[0141] In some embodiments, the content provider provides the content parser with a subtitle track query interface and a subtitle prompt query interface. The subtitle track query interface is used to query the list of currently available subtitle tracks, and its query address format is: content: / / com.tv.browser.subtitle.provider / subtitles, where authority is the content provider's authorization string.

[0142] The subtitle prompt query interface is used to query subtitle prompt information for a specified subtitle track within a specified time window. The query address format is: content: / / com.tv.browser.subtitle.provider / subtitles / {trackId} / cues, where trackId is the subtitle track identifier.

[0143] S1020: In response to the track query request initiated by the content parser through the subtitle track query interface, the content provider returns a list of subtitle track information to the content parser.

[0144] In some embodiments, the content parser initiates a track query request to the content provider through the subtitle track query interface. In response to the track query request, the content provider retrieves all currently available subtitle track information from the subtitle engine's memory data structure and constructs a list of subtitle track information.

[0145] The subtitle track information list includes multiple pieces of information for each subtitle track. Each subtitle track information includes fields such as track identifier, language code, source type, readable tag, and whether it is the default track. The source type identifies the source from which the subtitle data was extracted, including first extraction method, second extraction method, and third extraction method. The content provider returns the subtitle track information list to the content parser. After receiving the subtitle track information list, the content parser can select the target subtitle track from the list based on the language code and the user's language preference.

[0146] S1030: In response to the subtitle prompt query request initiated by the content parser through the subtitle prompt query interface, the content provider returns the subtitle prompt information within the target time window to the content parser.

[0147] In some embodiments, the content parser initiates a subtitle prompt query to the content provider through a subtitle prompt query interface based on the track identifier of the selected target subtitle track. The query request carries a time window parameter, which is defined centered on the current playback progress of the video media asset, for example, the time interval from 5 seconds before to 15 seconds after the current playback progress.

[0148] The content provider retrieves all subtitle prompts within the given time window from the subtitle engine's memory data structure. Each subtitle prompt includes a track identifier, subtitle start time, subtitle end time, and subtitle text content. The content provider then encapsulates the subtitle prompts into structured data and returns it to the content parser.

[0149] In one implementation, for streaming subtitles such as DOM overlay subtitles, the content provider actively calls the notification method when new subtitle data is generated, triggering the global change observer registered by the content parser to call the new subtitle data. After receiving the notification, the content parser re-queries the subtitle data in the latest time window, realizing real-time push of subtitle data, avoiding periodic polling by the player, reducing power consumption and improving real-time performance.

[0150] Based on the above technical solutions, this application embodiment sends a standardized subtitle track query interface and a subtitle prompt query interface through the content provider. The player obtains subtitle data according to the playback progress time window through the content parser in an inter-process communication manner, realizing a decoupled architecture between the browser and the player. The player does not need to care about the source of the subtitles and can obtain the subtitle data through the corresponding subtitle interface, thereby improving the efficiency of subtitle transmission.

[0151] In some embodiments, the controller 250 is further configured to set an authorization string between the browser application and the player, so that the content parser creates query addresses corresponding to the subtitle track query interface and the subtitle prompt query interface based on the authorization string.

[0152] In some embodiments, the authorization string is a pre-agreed identifier string between the browser application and the player, used to construct a Uniform Resource Identifier for the content provider, allowing the content parser of the independent player to access the subtitle data of the target webpage in the browser application across processes based on the authorization string.

[0153] In some embodiments, when the player starts up, it constructs a query address based on a preset authorization string and sends a query request to the content provider through the content parser.

[0154] Alternatively, in some embodiments, the controller 250 is also configured to pass an authorization string to the player when the browser application launches the player via stand-alone playback mode, so that the content parser creates a query address based on the received authorization string.

[0155] In some embodiments, when a browser application launches the player via standalone playback mode, it dynamically passes an authorization string to the player via Intent Extra. After parsing the authorization string from the Intent Extra, the player creates a query address based on that authorization string.

[0156] Based on this, each browser application can use a different authorization string. After the authorization strings of different browser applications are passed to the player through intent attachment data, the player can determine the source of the corresponding subtitle data based on the received authorization string.

[0157] In one implementation, the browser application can also temporarily grant the player read access to the subtitle data by setting the content provider's Uniform Resource Identifier in the field of intent to attach data and setting a read permission flag.

[0158] Based on the above technical solutions, this application embodiment sets an authorization string between the browser application and the player, and the content parser creates a query address based on the authorization string to achieve secure authentication of cross-process subtitle data access, preventing unauthorized applications from maliciously reading subtitle data. At the same time, it supports two authorization methods, namely static agreement and dynamic transmission, to improve the security and flexibility of the subtitle transmission pipeline.

[0159] Figure 11 This is a flowchart illustrating the multi-source caption deduplication process performed by a display device according to some embodiments of this application.

[0160] See Figure 11 In some embodiments, when multiple subtitle data extracted by different subtitle extraction methods are detected simultaneously on the target webpage, the controller 250 is further configured to perform the following steps: S1110: Maps multiple subtitle data to a unified timeline.

[0161] In some embodiments, when multiple subtitle data extracted by a first extraction method, a second extraction method, and a third extraction method exist simultaneously on a target webpage, the controller 250 maps all the subtitle data from all sources to a unified timeline.

[0162] Since subtitle data from different sources may use different time bases, for example, subtitle files from track elements use the timestamps inside the subtitle file as the base, while subtitle data from DOM nodes use the current playback time of the video element as the base, controller 250 needs to unify the timestamps from different sources onto a unified timeline for subsequent overlap detection and deduplication processing.

[0163] S1120: Detect whether there is time window overlap in the subtitle data corresponding to different subtitle extraction methods on a unified timeline.

[0164] In some embodiments, the controller 250 detects whether there is time window overlap between subtitle data from different sources on a unified timeline. Specifically, for each pair of subtitle data from different sources, the controller 250 compares their subtitle start time and subtitle end time to determine whether their time intervals overlap.

[0165] In some embodiments, if there is an overlap between the time intervals of the two, and the length of the overlap exceeds a preset overlap ratio threshold (e.g., 50%), it is determined that there is an overlap of time windows.

[0166] S1130: When the time window overlap meets the preset overlap conditions, perform deduplication processing on the subtitle data with time window overlap based on the preset priority order.

[0167] In some embodiments, when the subtitle data corresponding to different subtitle extraction methods have time windows, the controller 250 can determine whether the overlap of the time windows meets a preset overlap condition. For example, the preset overlap condition can be that the text similarity exceeds a preset similarity threshold. Based on this, the controller 250 calculates the text similarity of the subtitle data corresponding to different subtitle extraction methods. The text similarity can be calculated based on edit distance or Jaccard similarity coefficient. This embodiment does not specifically limit the implementation method of calculating text similarity.

[0168] When the text similarity of at least two subtitle data exceeds a preset similarity threshold (e.g., 0.8), the controller 250 determines that the two are duplicate subtitles. Then, based on the preset priority order, the subtitle data with overlapping time windows are deduplicated. The priority order represents the order of confidence of the subtitle data corresponding to different subtitle extraction methods. The deduplication process retains the subtitle data with higher confidence and discards the subtitle data with lower confidence based on the preset priority order.

[0169] In some embodiments, if there are missing subtitles in the subtitle data extracted by one subtitle extraction method, the controller 250 can supplement the aforementioned subtitle data by extracting subtitle data by another subtitle extraction method. For example, the controller 250 can determine the missing position of the subtitle, extract the subtitle data to be supplemented from another subtitle data according to the missing position, and fill the missing position with the subtitle data to be supplemented.

[0170] Based on the above technical solutions, this application embodiment maps subtitles from multiple sources to a unified timeline and detects overlapping time windows. It then performs deduplication processing on overlapping subtitles based on priority order, avoiding the problem of repeated subtitle display caused by multiple subtitle extraction methods simultaneously extracting subtitles, thus ensuring the uniqueness and accuracy of subtitle data.

[0171] Figure 12 A flowchart illustrating the subtitle language recognition process performed by a display device according to some embodiments of this application.

[0172] See Figure 12 In some embodiments, the controller 250 is also configured to perform the following steps: S1210: When the subtitle data does not carry a language identifier, perform character set range detection on the subtitle data.

[0173] In some embodiments, some sourced subtitle data may not carry language identifiers, which are used to indicate the language of the subtitle data.

[0174] For subtitle data without language identifiers, the controller 250 first performs character set range detection on the subtitle text content. Character set range detection determines the character types contained in the subtitle text by analyzing the Unicode encoding range of the characters. For example, the Unicode encoding range U+4E00 to U+9FFF represents the CJK Unified Ideographs, U+0041 to U+007A represents Latin letters, and U+0400 to U+04FF represents Cyrillic letters. Through character set range detection, the controller 250 can preliminarily determine the character system to which the subtitle text belongs.

[0175] S1220: When the subtitle data is detected to contain text characters of a preset type, the subtitle data is used to perform language recognition through a preset language model to obtain the language recognition result.

[0176] In some embodiments, when character set range detection detects that the subtitle data contains text characters of a preset type, the controller 250 further performs accurate language recognition on the subtitle data using a preset language model. The preset type of text characters includes characters such as the Unified Ideographs (CHJK), which cannot be used to determine the specific language based solely on the character set range. For the Unified Ideographs, the controller 250 distinguishes between Chinese, Japanese, and Korean using an N-gram language model. The N-gram language model is trained on a large corpus and determines the language of the text by statistically analyzing the probability of occurrence of character sequences.

[0177] In one implementation, the controller 250 inputs the subtitle data into an N-gram language model, the model outputs the probability distribution of each candidate language, and selects the language with the highest probability as the language recognition result of the subtitle data.

[0178] S1230: Mark the language recognition results in the language identifier field of the subtitle data.

[0179] In some embodiments, the controller 250 writes the language recognition result into the language identifier field of the subtitle data. The language identifier field adopts a standard language code format (such as zh-CN for Simplified Chinese, en for English, ja for Japanese, etc.). When the annotated subtitle data is exposed to the player through the content provider, it carries complete language identifier information.

[0180] In some embodiments, the player can automatically select a subtitle track that matches the user's language preference settings to display subtitle data, eliminating the need for the user to manually select the subtitle language.

[0181] It should be noted that the language recognition results are also used in the deduplication process of multi-source subtitles, and deduplication is only performed between subtitle data in the same language to avoid incorrect deduplication of subtitle data in different languages.

[0182] Based on the above technical solutions, this application embodiment automatically recognizes the language of subtitle data without language identifiers by combining character set range detection with a preset language model, and marks the recognition results in the language identifier field to realize automatic labeling of subtitle languages. This makes it easier for the player to automatically select the subtitle track according to the user's language preference and improve the intelligence level of subtitle management.

[0183] Based on the display device 200 provided in the above embodiments, some embodiments of this application also provide a subtitle extraction method based on a browser application, applied to the display device 200, including: S410: In response to a user's command to load a target webpage based on a browser application, control the display 260 to display the target webpage on the browser interface.

[0184] The target webpage plays video media assets in an embedded playback mode, and the browser application is registered with a content provider.

[0185] S420: Extracts subtitle data from the target webpage that corresponds to the current playback progress of the video media asset using a preset subtitle extraction method.

[0186] S430: In response to the video media switching from embedded playback mode to standalone playback mode, control the display 260 to display the video media image in the player interface.

[0187] The standalone playback mode is a mode in which video media assets are played through a player other than a browser application. The player is registered with a content parser that interacts with the content provider.

[0188] S440: A player-based content parser that retrieves subtitle data sent by the browser application through the content provider.

[0189] S450: Controls the display to show subtitle data on the media asset screen of the player interface.

[0190] The implementation methods and beneficial effects of the embodiments of this application can be referred to the corresponding implementation methods and beneficial effects in the aforementioned embodiments of the display device 200, which will not be repeated here.

[0191] As can be seen from the above technical solutions, the display device and browser application-based subtitle extraction method provided in this application automatically extract subtitle data when the target webpage plays video media in embedded playback mode by registering a content provider in the browser application, and obtains and displays the subtitle data from the content provider through the content parser registered by the player when the video media switches to independent playback mode. This achieves automatic extraction and seamless transmission of subtitle data when the webpage video switches from embedded playback mode to independent playback mode, avoids subtitle loss, and improves the user's viewing experience.

[0192] The same or similar parts among the various embodiments in this specification can be referred to mutually, and will not be repeated here.

[0193] Those skilled in the art will clearly understand that the techniques in the embodiments of the present invention can be implemented using software plus necessary general-purpose hardware platforms. Based on this understanding, the technical solutions in the embodiments of the present invention, or the parts that contribute to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods of various embodiments or certain parts of the embodiments of the present invention.

[0194] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application.

[0195] For ease of explanation, the above description has been provided in conjunction with specific embodiments. However, the above exemplary discussion is not intended to be exhaustive or to limit the embodiments to the specific forms disclosed above. Various modifications and variations can be obtained based on the above teachings. The selection and description of the above embodiments are for the purpose of better explaining the principles and practical applications, thereby enabling those skilled in the art to better utilize the described embodiments and various different variations of embodiments suitable for specific use considerations.

Claims

1. A display device, characterized in that, include: The monitor is configured to display the user interface; The controller is configured as follows: In response to a user's input of a target webpage loading command based on a browser application, the display is controlled to show the target webpage on the browser interface, wherein the target webpage plays video media in an embedded playback mode, and the browser application is registered with a content provider; Using a preset subtitle extraction method, subtitle data corresponding to the current playback progress of the video media asset is extracted from the target webpage. In response to the video media asset switching from the embedded playback mode to the independent playback mode, the display is controlled to show the media asset screen of the video media asset in the player interface. The independent playback mode is the mode of playing the video media asset through a player other than the browser application. The player is registered with a content parser that interacts with the content provider. Based on the content parser of the player, obtain the subtitle data sent by the browser application through the content provider; The display is controlled to show the subtitle data in the media asset screen of the player interface.

2. The display device according to claim 1, characterized in that, The subtitle extraction method includes a first extraction method, in which the controller extracts subtitle data corresponding to the current playback progress of the video media asset from the target webpage using a preset subtitle extraction method, specifically configured as follows: Based on the first extraction method, the target webpage is decomposed into multiple object nodes; Calculate the caption prediction score of the object node, which is used to evaluate the probability that the object node is caption data; Object nodes whose predicted subtitle score is greater than or equal to a preset score threshold are marked as subtitle data corresponding to the current playback progress of the video media asset.

3. The display device according to claim 2, characterized in that, The controller performs the calculation of the subtitle prediction score for the object node, specifically configured as follows: The content change frequency score, subtitle feature score, and semantic rule score of the object node are calculated. The content change frequency score is used to evaluate the frequency of the object node's changes with the video media asset. The subtitle feature score is used to evaluate the layout features of the text content contained in the object node. The semantic rule score is used to evaluate the degree to which the text in the object node conforms to the general language expression logic. Based on a preset weight ratio, the content change frequency score, the subtitle feature score, and the semantic rule score are weighted and summed to obtain the subtitle prediction score.

4. The display device according to claim 1, characterized in that, The subtitle extraction method includes a second extraction method, wherein the controller executes a preset subtitle extraction method to extract subtitle data corresponding to the current playback progress of the video media asset from the target webpage, specifically configured as follows: Based on the second extraction method, extract the list of child nodes of the video media asset from the target webpage; Detect the subtitle track elements added to the child node list; When a subtitle track element is detected to be added to the child node list, the subtitle file address of the video media asset is read based on the source attribute corresponding to the subtitle track element; Extract subtitle data corresponding to the current playback progress of the video media asset from the subtitle file address.

5. The display device according to claim 1, characterized in that, The subtitle extraction method includes a third extraction method, in which the controller extracts subtitle data corresponding to the current playback progress of the video media asset from the target webpage using a preset subtitle extraction method, specifically configured as follows: Based on the third extraction method, monitor the subtitle track addition method corresponding to the video media asset; If the subtitle track addition method is called, record the newly created subtitle track object; Upon detecting a change in subtitle prompts in the subtitle track object, obtain the set of active subtitle prompts for the subtitle track object; Extract subtitle data corresponding to the current playback progress of the video media asset from the set of active subtitle prompts.

6. The display device according to claim 1, characterized in that, After the controller executes the content parser based on the player to obtain the subtitle data sent by the browser application through the content provider, it is further configured to: The content provider of the browser application provides a subtitle track query interface and a subtitle prompt query interface to the content parser of the player. In response to the track query request initiated by the content parser through the subtitle track query interface, the content provider returns a list of subtitle track information to the content parser. The list of subtitle track information includes multiple subtitle track information, and the subtitle track information includes track identifier and source type. In response to a subtitle prompt query request initiated by the content parser through the subtitle prompt query interface, the content provider returns subtitle prompt information within a target time window to the content parser. The subtitle prompt information includes at least the subtitle start time, subtitle end time, and subtitle text content. The target time window is a time interval defined centered on the current playback progress of the video media asset.

7. The display device according to claim 6, characterized in that, The controller is also configured to: An authorization string is set between the browser application and the player so that the content parser can create query addresses corresponding to the subtitle track query interface and the subtitle prompt query interface based on the authorization string. Alternatively, when the browser application launches the player via the independent playback mode, the authorization string is passed to the player so that the content parser creates the query address based on the received authorization string.

8. The display device according to claim 1, characterized in that, When multiple subtitle data extracted by different subtitle extraction methods are detected simultaneously in the target webpage, the controller is further configured to: Map multiple sets of subtitle data to a unified timeline; Detect whether there is time window overlap on the unified timeline for subtitle data corresponding to different subtitle extraction methods; When the overlap of the time windows meets the preset overlap conditions, the subtitle data with overlapping time windows are deduplicated based on the preset priority order, wherein the priority order represents the order of confidence of the subtitle data corresponding to different subtitle extraction methods.

9. The display device according to claim 8, characterized in that, The controller is also configured to: If the subtitle data does not carry a language identifier, a character set range detection is performed on the subtitle data, whereby the language identifier is used to indicate the language of the subtitle data. If the subtitle data is detected to contain text characters of a preset type, the subtitle data is subjected to language recognition using a preset language model to obtain the language recognition result; The language recognition result is labeled in the language identifier field of the subtitle data.

10. A method for extracting subtitles based on a browser application, characterized in that, include: In response to a user's input of a loading command for a target webpage based on a browser application, the display is controlled to show the target webpage on the browser interface, wherein the target webpage plays video media in an embedded playback mode, and the browser application is registered with a content provider; Using a preset subtitle extraction method, subtitle data corresponding to the current playback progress of the video media asset is extracted from the target webpage. In response to the video media asset switching from the embedded playback mode to the independent playback mode, the display is controlled to show the media asset screen of the video media asset in the player interface. The independent playback mode is the mode of playing the video media asset through a player other than the browser application. The player is registered with a content parser that interacts with the content provider. Based on the content parser of the player, obtain the subtitle data sent by the browser application through the content provider; The display is controlled to show the subtitle data in the media asset screen of the player interface.

Images