Display equipment and subtitle display method based on split-screen playing

Abstract

The invention provides a display device and a subtitle display method based on split-screen playing, and the method comprises the steps: displaying a first window and a second window on a user interface in response to a split-screen control instruction, obtaining first media asset data and second media asset data under the condition that the first window is selected from the two split-screen windows, and displaying the first media asset data and the second media asset data in the split-screen control instruction; and semantically understanding the media asset audio of the second media asset data to obtain first subtitle data. And controlling the display to display a media resource picture corresponding to the first media resource data in the first window, displaying a media resource picture corresponding to the second media resource data and the first subtitle data in the second window, and controlling the audio output device to play a media resource audio of the first media resource data. Through semantic understanding of the media asset audio in the second window, a user can understand the picture meaning of the first window through the media asset audio of the first media asset data and understand the picture meaning of the second media asset data played in the second window according to the first subtitle data, and the user watching experience during split-screen playing is improved.

Description

Technical Field

[0001] The present application relates to the technical field of display devices, and in particular to a display device and a subtitle display method based on split-screen playback. Background Art

[0002] The display device can display multiple split-screen windows on the same screen through the split-screen technology, so as to display different media data in different split-screen windows. However, when different media data are played in multiple split-screen windows, the audio parts of the multiple media data will affect each other, causing the user to be unable to hear the audio part of the media data clearly.

[0003] To this end, the display device can divide the split-screen window into a main window and a sub-window. During the split-screen display process, the display device plays the audio part of the media data corresponding to the main window, and the sub-window only displays the picture without playing the audio part, thereby avoiding the playback conflict of the media audio. However, in this way, the user can only watch the sub-window, but cannot know the audio part of the media data played by the sub-window and understand the meaning of the sub-window picture, which reduces the user's viewing experience. Summary of the invention

[0004] This is to solve the problem that when the display device plays different media in split screen, only the media audio in the main window is displayed, resulting in the user being unable to understand the meaning of the picture based on the audio part in the secondary window.

[0005] In a first aspect, the present application provides a display device, including:

[0006] a display configured to display a user interface;

[0007] an audio output device configured to play audio data;

[0008] The controller is configured as:

[0009] In response to the split-screen control instruction, displaying a first window and a second window on the user interface;

[0010] When the first window is selected in the two split-screen windows, first media data and second media data are acquired, wherein the first media data is the media data played in the first window, and the second media data is the media data played in the second window;

[0011] Acquire first subtitle data, where the first subtitle data is obtained after semantic understanding processing of the second media asset data;

[0012] Controlling the display to display a media picture corresponding to the first media data in the first window, and to display a media picture corresponding to the second media data and the first subtitle data in the second window;

[0013] When the display is displaying the media picture corresponding to the first media data, the audio output device is controlled to play the media audio of the first media data.

[0014] In the above technical solution, when the display device plays different media data in split screen, the media picture of the first media data is displayed in the first window, and the media audio of the first media data is played, and the media picture of the second media data and the first subtitle data obtained by semantically understanding the media audio corresponding to the second media data are displayed in the second window, so that the user can understand the picture meaning of the second media data displayed in the second window through the first subtitle data, thereby improving the picture experience of split screen playback.

[0015] In some embodiments, when the number of windows of the second window is one, the controller is further configured to:

[0016] In response to a window switching instruction input by a user, semantically understand the media asset audio of the first media asset data to obtain second subtitle data;

[0017] The display is controlled to display the second subtitle data in the first window, and the audio output device is controlled to stop playing the media audio of the first media data and play the media audio of the second media data.

[0018] In the above technical solution, when the user interface includes a second window, the user can switch the second window to the main window through the window switching instruction, and the display device can stop playing the media audio of the first media data in the first window, and the display device plays the media audio of the second media data. At the same time, the second subtitle data is generated according to the media audio of the first media data, so that the user can understand the meaning of the picture of the first media data in the first window through the second subtitle data.

[0019] In some embodiments, before the controller executes the step of acquiring the first subtitle data, the controller is further configured to:

[0020] Detecting media asset subtitle data of the second media asset data;

[0021] If the media asset subtitle data is detected, the first media asset data is generated according to the media asset subtitle data.

[0022] In the above technical solution, before performing semantic understanding on the media asset audio, the display device detects whether the second media asset data contains media asset subtitle data. If the media asset subtitle data is contained, the first subtitle data can be directly displayed in the second window, thereby saving the time of performing semantic understanding on the media asset audio and improving the subtitle display efficiency.

[0023] In some embodiments, the controller performs the step of acquiring the first subtitle data, which is specifically configured as follows:

[0024] Get the target language information input by the user;

[0025] Acquire speech text data, where the speech text data is obtained by performing semantic understanding on the media asset audio corresponding to the second media asset data;

[0026] If the language information of the speech text data is different from the target language information, language translation is performed on the speech text data according to the target language information to obtain first subtitle data.

[0027] In the above technical solution, when the display device displays the first subtitle data in the second window, the user can select the corresponding target language information. The display device can perform semantic understanding on the media audio of the second media data, and perform language translation on the obtained voice text data according to the target language information to obtain subtitle data in different languages ​​to adapt to users whose native languages ​​are different.

[0028] In some embodiments, a memory is further included, and the memory is configured to store a semantic understanding model. The controller performs the step of acquiring speech text data, which is specifically configured as follows:

[0029] Inputting the media asset audio of the second media asset data into the semantic understanding model to extract speech features and semantic features of the media asset audio of the second media asset data;

[0030] The speech text data is generated according to the speech feature and the semantic feature.

[0031] In the above technical solution, when the display device semantically understands the second media data, it can output the media audio of the second media data to the semantic understanding model. The semantic understanding model can extract the speech features and semantic features of the media audio, thereby generating speech text data based on the speech features and semantic features to achieve text extraction.

[0032] In some embodiments, after the controller executes the step of displaying the media asset picture corresponding to the first media asset data in the first window, the controller is further configured to:

[0033] In response to a first scaling instruction input by a user based on the first window, scaling the first window in proportion;

[0034] Calculate a remaining display area according to the display area of ​​the user interface and the first window after scaling;

[0035] The second window is scaled proportionally to fill the remaining display area.

[0036] In the above technical solution, when the display device scales the first window, the second window will be adaptively scaled according to the scaling of the first window. The remaining display area is calculated by the display area of ​​the user interface and the scaled first window, and the second window is scaled proportionally to fill the remaining display area, thereby improving the display area utilization rate of the user interface.

[0037] In some embodiments, the controller performs a step of proportionally scaling the second window to fill the remaining display area, which is specifically configured as follows:

[0038] Calculating a display ratio of the second window to the remaining display area;

[0039] The second window is proportionally scaled according to the display ratio, and the first subtitle data is proportionally scaled according to the display ratio during the proportional scaling of the second window.

[0040] In the above technical solution, the first subtitle data can be scaled according to the scaling of the second window, so that the picture content in the second window is scaled synchronously with the first subtitle data, thereby improving the scaling synchronization rate between the second window and the first subtitle data.

[0041] In some embodiments, the step of performing scaling of the first subtitle data in proportion to the display ratio, the controller further comprises:

[0042] Calculating the display area of ​​the second window;

[0043] If the display area is less than or equal to the preset area threshold, semantically summarizing the text content corresponding to the first subtitle data by using the semantic understanding model to obtain subtitle introduction data;

[0044] The display is controlled to cancel displaying the first subtitle data, and the subtitle introduction data is displayed in the second window.

[0045] In the above technical solution, when the display area of ​​the second window is smaller than the preset area threshold, the display of the subtitle data will block the picture content of the second window. Therefore, the display device can perform semantic generalization on the first subtitle data, thereby simplifying the data content of the first subtitle data and reducing the occlusion of the second window by the first subtitle data.

[0046] In some embodiments, the controller performs the step of displaying the media asset picture corresponding to the second media asset data and the first subtitle data in the second window, which is specifically configured as follows:

[0047] Detecting a voice timestamp in the media asset audio of the second media asset data, where the voice timestamp is a timestamp of the voice data in the media asset audio;

[0048] The first subtitle data is displayed in the second window according to the speech timestamp.

[0049] In the above technical solution, the voice timestamp in the media audio can be detected to obtain the playback time of the voice data in the second media data, so as to display the first subtitle data corresponding to the voice content according to the voice timestamp, maintain the display synchronization of the voice timestamp and the first subtitle data, and ensure the audio and video synchronization effect of the second media data.

[0050] In a second aspect, some embodiments of the present application provide a subtitle display method based on split-screen playback, which is applied to the display device described in the first aspect, wherein the display device includes a display configured to display a user interface; an audio output device configured to play audio data; and a controller; the method includes:

[0051] In response to the split-screen control instruction, displaying a first window and a second window on the user interface;

[0052] When the first window is selected in the two split-screen windows, first media data and second media data are acquired, wherein the first media data is the media data played in the first window, and the second media data is the media data played in the second window;

[0053] Acquire first subtitle data, where the first subtitle data is obtained after semantic understanding processing of the second media asset data;

[0054] Controlling the display to display a media picture corresponding to the first media data in the first window, and to display a media picture corresponding to the second media data and the first subtitle data in the second window;

[0055] When the display is displaying the media picture corresponding to the first media data, the audio output device is controlled to play the media audio of the first media data.

[0056] It can be seen from the above technical solutions that the present application provides a display device and a subtitle display method based on split-screen playback, wherein the method displays a first window and a second window in the user interface in response to a split-screen control instruction, and obtains the first media data and the second media data when the first window is selected in the two split-screen windows, and semantically understands the media audio of the second media data to obtain the first subtitle data. The display is controlled to display the media screen corresponding to the first media data in the first window, and the media screen and the first subtitle data corresponding to the second media data in the second window, and the audio output device is controlled to play the media audio of the first media data. The present application obtains the first subtitle data by semantically understanding the media audio in the second window, so that the user can understand the meaning of the screen of the first window through the media audio and screen of the first media data, and understand the meaning of the screen of the second media data played in the second window according to the first subtitle data, thereby improving the user viewing experience during split-screen playback. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0058] Figure 1 A schematic diagram of an operation scenario between a display device and a control device provided in some embodiments of the present application;

[0059] Figure 2 A schematic diagram of the hardware configuration of a display device provided in some embodiments of the present application;

[0060] Figure 3 A schematic diagram of software configuration of a display device provided in some embodiments of the present application;

[0061] Figure 4 This is a schematic diagram of the display effect of the multi-window mode in the embodiment of the present application;

[0062] Figure 5 A schematic diagram of the display effect of switching the multi-window mode by pressing a button in an embodiment of the present application;

[0063] Figure 6 A schematic diagram of the display effect of switching the multi-window mode by switching options in an embodiment of the present application;

[0064] Figure 7 A flowchart of a subtitle display method based on split-screen playback executed by a display device in an embodiment of the present application;

[0065] Figure 8A timing diagram of a subtitle display method based on split-screen playback executed by a display device in an embodiment of the present application;

[0066] Fig. 9 This is an interface display diagram showing multiple windows when the display device is in multi-window mode in an embodiment of the present application;

[0067] Fig.10 This is a flow chart of generating first subtitle data according to target language information in an embodiment of the present application;

[0068] Fig.11 This is a flowchart of executing different translation methods according to target language information in an embodiment of the present application;

[0069] Fig.12 This is a flowchart of performing text conversion according to a semantic understanding model in an embodiment of the present application;

[0070] Fig.13 This is a flow chart of a first embodiment of switching the main window of the split-screen window in an embodiment of the present application;

[0071] Fig.14 A flow chart of a second embodiment of switching the main window of the split-screen window in the embodiment of the present application;

[0072] Fig.15 This is a flow chart of performing zooming on a first window in an embodiment of the present application;

[0073] Fig.16 This is a flow chart of scaling the second window according to the remaining display area in an embodiment of the present application. DETAILED DESCRIPTION

[0074] The following embodiments are described in detail, and examples thereof are shown in the accompanying drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The implementations described in the following embodiments do not represent all implementations consistent with the present application. They are only examples of systems and methods consistent with some aspects of the present application as detailed in the claims.

[0075] It should be noted that the brief description of terms in this application is only for the convenience of understanding the embodiments described below, and is not intended to limit the embodiments of this application. Unless otherwise specified, these terms should be understood according to their ordinary and common meanings.

[0076] The terms "first", "second", "third", etc. in the specification and claims of this application and the above drawings are used to distinguish similar or similar objects or entities, and do not necessarily mean to limit a specific order or sequence, unless otherwise noted. It should be understood that the terms used in this way can be interchangeable under appropriate circumstances.

[0077] The terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover but not exclude inclusion, for example, a product or device comprising a list of components is not necessarily limited to all the components expressly listed but may include other components not expressly listed or inherent to such product or device.

[0078] The term "module" refers to any known or later developed hardware, software, firmware, artificial intelligence, fuzzy logic, or combination of hardware and / or software code that is capable of performing the functions associated with that element.

[0079] In the embodiment of the present application, the display device 200 generally refers to a device with image display and data processing capabilities. For example, the display device 200 includes but is not limited to a smart TV, a mobile terminal, a computer, a monitor, an advertising screen, a wearable device, a virtual reality device, an augmented reality device, etc.

[0080] Figure 1 This is a schematic diagram of an operation scenario between a display device and a control device provided in some embodiments of the present application. Figure 1 As shown in FIG. 1 , a user can operate the display device 200 through touch operation, the mobile terminal 300 and the control device 100. The control device 100 is used to receive operation instructions input by the user and convert the operation instructions into control instructions that can be recognized and responded to by the display device 200. For example, the control device 100 can be a remote controller, a stylus pen, a handle, etc.

[0081] The mobile terminal 300 can be used as a control device for performing human-computer interaction between the user and the display device 200. The mobile terminal 300 can also be used as a communication device for establishing a communication connection with the display device 200 and performing data interaction. In some embodiments, the mobile terminal 300 can install software applications with the display device 200, and achieve connection and communication through a network communication protocol to achieve the purpose of one-to-one control operation and data communication. The audio and video content displayed on the mobile terminal 300 can also be transmitted to the display device 200 to achieve a synchronous display function.

[0082] In some embodiments, the mobile terminal 300 or other electronic devices may also simulate the functions of the control device 100 by running an application program for controlling the display device 200 .

[0083] like Figure 1 As also shown in FIG. 4 , the display device 200 also communicates data with the server 400 through various communication methods. The display device 200 may be allowed to communicate and connect through a local area network (LAN), a wireless local area network (WLAN), and other networks.

[0084] The display device 200 may provide a broadcast receiving television function, and may also additionally provide an intelligent network television function with a computer support function, including but not limited to network television, smart television, Internet Protocol television (IPTV), and the like.

[0085] Figure 2 Some embodiments of the present application provide Figure 1 2 is a block diagram of the hardware configuration of the display device 200.

[0086] In some embodiments, the display device 200 may include at least one of a tuner 210, a communication device 220, a detector 230, a device interface 240, a controller 250, a display 260, an audio output device 270, a memory, a power supply, and a user input interface.

[0087] In some embodiments, the detector 230 is used to collect signals of the external environment or external interaction. For example, the detector 230 includes a light receiver, a sensor for collecting the intensity of ambient light; or, the detector 230 includes an image collector, such as a camera, which can be used to collect external environment scenes, user attributes or user interaction gestures; or, the detector 230 includes a sound collector, such as a microphone, etc., for receiving external sounds.

[0088] In some embodiments, the display 260 includes a display function component for presenting a picture, and a driving component for driving an image display. The display 260 is used to receive an image signal output from the controller 250 for display. For example, the display 260 can be used to display video content, image content, and components of a menu control interface and a user control UI interface.

[0089] In some embodiments, the communication device 220 is a component for communicating with an external device or server 400 according to various communication protocol types. The display device 200 may be provided with a plurality of communication devices 220 according to different supported communication modes. For example, when the display device 200 supports wireless network communication, the display device 200 may be provided with a communication device 220 including a WiFi function. When the display device 200 supports Bluetooth connection communication, the display device 200 needs to be provided with a communication device 220 including a Bluetooth function.

[0090] The communication device 220 can enable the display device 200 to communicate with the external device or server 400 by wireless or wired connection. Among them, the wired connection can connect the display device 200 with the external device through components such as data cables and interfaces. The wireless connection can connect the display device 200 with the external device through wireless signals or wireless networks. The display device 200 can establish a connection relationship with the external device directly, or indirectly establish a connection relationship through a gateway, a router, a connection device, etc.

[0091] In some embodiments, the controller 250 may include at least one of a central processing unit, a video processor, an audio processor, a graphics processor, and a power processor, and a first interface to an nth interface for input / output. The controller 250 controls the operation of the display device and responds to the user's operation through various software control programs stored in the memory. The controller 250 controls the overall operation of the display device 200.

[0092] In some embodiments, the controller 250 and the tuner-demodulator 210 may be located in different separate devices, that is, the tuner-demodulator 210 may also be located in an external device of the main device where the controller 250 is located, such as an external set-top box.

[0093] In some embodiments, the user may input a user command through a graphical user interface (GUI) displayed on the display 260 , and the user input interface receives the user input command through the graphical user interface (GUI).

[0094] In some embodiments, the audio output device 270 may be a local speaker of the display device 200, or may be an external audio output device of the display device 200. In particular, for the external audio output device of the display device 200, the display device 200 may also be provided with an external audio output terminal, and the audio output device may be connected to the display device 200 through the external audio output terminal to output the sound of the display device 200.

[0095] In some embodiments, the user input interface 280 may be used to receive instructions from a user.

[0096] In order to perform user interaction, in some embodiments, the display device 200 may run an operating system. The operating system is a computer program for managing and controlling hardware resources and software resources in the display device 200. The operating system can control the display device to provide a user interface, for example, the operating system can directly control the display device to provide a user interface, or can provide a user interface by running an application program. The operating system also allows the user to interact with the display device 200.

[0097] It should be noted that the operating system may be a native operating system based on a specific operating platform, or a third-party operating system deeply customized based on a specific operating platform, or an independent operating system specially developed for the display device.

[0098] The operating system can be divided into different modules or layers according to the functions implemented, such as Figure 3As shown, in some embodiments, the system is divided into four layers, from top to bottom, namely, the application layer (Applications) layer (referred to as "application layer"), the application framework layer (Application Framework) layer (referred to as "framework layer"), the system library layer and the kernel layer.

[0099] In some embodiments, the application layer is used to provide services and interfaces for applications so that the display device 200 can run applications and interact with users based on the applications. At least one application can be run in the application layer, and these applications can be window programs, system settings programs, clock programs, etc. that come with the operating system; they can also be applications developed by third-party developers. In specific implementations, the application packages in the application layer are not limited to the above examples.

[0100] The framework layer provides application programming interfaces (APIs) and programming frameworks for applications. The application framework layer includes some predefined functions. The application framework layer is equivalent to a processing center that determines the actions that applications in the application layer take. Through the API interface, applications can access system resources and obtain system services during execution.

[0101] like Figure 3 As shown, Figure 3 A software configuration diagram of a display device provided for some embodiments of the present application. In some embodiments, the system of the display device 200 can be divided into three layers, namely, an application layer, a middleware layer, and a hardware layer from top to bottom.

[0102] The application layer mainly includes applications on the TV and the application framework. Among them, applications are mainly applications developed based on the browser, such as HTML5 APPs and native applications.

[0103] The Application Framework is a complete program model that has all the basic functions required by standard application software, such as file access, data exchange, etc., as well as the user interfaces of these functions (toolbars, status bars, menus, dialog boxes).

[0104] Native apps can support online or offline, message push or local resource access.

[0105] The middleware layer includes various TV protocols, multimedia protocols, system components and other middleware. The middleware can use the basic services (functions) provided by the system software to connect various parts of the application system or different applications on the network, and can achieve the purpose of resource sharing and function sharing.

[0106] The hardware layer mainly includes HAL interface, hardware and driver. Among them, HAL interface is the unified interface for all TV chips to connect, and the specific logic is implemented by each chip. Drivers mainly include: audio driver, display driver, Bluetooth driver, camera driver, WIFI driver, USB driver, HDMI driver, sensor driver (such as fingerprint sensor, temperature sensor, pressure sensor, etc.), and power driver, etc.

[0107] It should be noted that the above example is only a simple division of the operating system functions and does not constitute a limitation on the specific operating system form of the display device 200 in the embodiment of the present application. Depending on factors such as the function of the display device and the type of operating system, the number of levels and specific level types contained in the operating system may be expressed in other forms.

[0108] Based on the above display device 200, the user can control the display device 200 to enter the multi-window mode during the use of the display device 200. In the multi-window mode, the display 260 of the display device 200 can simultaneously display two or more split-screen windows in the user interface, and each split-screen window can independently display different content. Figure 4 As shown, after the display device 200 enters the multi-window mode, the display 260 can simultaneously display the first split-screen window and the second split-screen window, wherein the first split-screen window can be used to display the playback screen of media asset A, and the second split-screen window can be used to display the playback screen of media asset B.

[0109] In different usage scenarios, the screen contents displayed in multiple windows are different, and the corresponding screen types can also be different. For example, the playback screen of media resource A is displayed in the first split-screen window, and the setting menu screen is displayed in the second split-screen window. That is, different split-screen windows display different types of interfaces at the same time, the first split-screen window displays the playback interface, and the second split-screen window displays the homepage interface.

[0110] The user can control the display device 200 to enter the multi-window mode through a specific interactive action. Figure 5 As shown, a multi-window mode switching button can be set on the control device 100 supporting the display device 200. When the user presses the multi-window mode switching button, the display device 200 can be controlled to enter the multi-window mode, that is, the display 260 of the display device 200 can be controlled to display multiple split-screen windows. Figure 6As shown, a multi-window mode switching option can also be set in the UI interface of the display device 200. When the user clicks the switching option, the display device 200 can be controlled to enter or exit the multi-window mode.

[0111] When the display device 200 is in the multi-window mode, it can not only display different screen contents through multiple windows at the same time, but also output the audio content corresponding to the screen while displaying the screen content. For example, when the display device 200 plays the media asset A through the first split-screen window, the audio output device 270 can synchronously play the audio of the media asset A, so that the user can listen to the audio corresponding to the media asset A while watching the media asset picture corresponding to the media asset A through the first split-screen window.

[0112] However, in the multi-window mode, when the display device 200 plays the media audio of multiple media data through multiple windows at the same time, different media audios will cause audio interference with each other. For example, when the first split screen window displays the playback screen of media A and the second split screen window displays the playback screen of media B, the audio output device 270 will play the media audio of media A and the media audio of media B at the same time. The media audio of media A will interfere with the media audio of media B, and the media audio of media B will also interfere with the media audio of media A, resulting in the user being unable to hear the sound parts of media A and media B clearly.

[0113] In order to avoid mutual interference of audio parts caused by playing different media data in multi-window mode, the display device can divide the split-screen interface into a main window and a sub-window. Taking the first window as the main window and the second window as the sub-window as an example, during the split-screen display process, the display device 200 can display the playback screen of media A in the first window, and play the media audio corresponding to media A through the audio output device 270. At the same time, the display device 200 displays the playback screen of media B in the second window, and does not play the media audio corresponding to media B through the audio output device 270. At this time, the display device 200 only plays the media audio corresponding to the media data of the main window. Although the conflict of audio playback is avoided, the user can only watch the sub-window displayed by the display device 200, and cannot understand the meaning of the sub-window through the audio part of the corresponding media data in the sub-window, which reduces the user's split-screen viewing experience.

[0114] When the display device plays different media resources in split - screen mode, only the media audio of the media resources played in the main window is displayed, resulting in the user being unable to understand the meaning of the picture based on the audio part of the media data corresponding to the sub - window. For this reason, some embodiments of the present application provide a display device 200, which includes: a display 260, an audio output interface 270, and a controller 250. Among them, the display 260 is used to display a user interface, and when the display device 200 is in a multi - window mode, the displayed user interface may include multiple split - screen windows. The audio output interface 270 can be connected to a local speaker and / or an external audio device, and is used to output audio to form an audio output path. The controller 250 is configured to run a subtitle display method based on split - screen playback. When playing media data in split - screen mode, it performs semantic understanding on the media audio of the media data corresponding to the sub - window, generates subtitle data, and displays the subtitle data in the sub - window, so that the user can understand the meaning of the sub - window through the subtitle data.

[0115] Figure 7 The flowchart of the split - screen playback executed by the display device provided by some embodiments of the present application Figure 8 The timing diagram of the split - screen playback executed by the display device provided by some embodiments of the present application. The display device 200 executes a subtitle display method based on split - screen playback according to the timing relationship as Figure 8 shown. Refer to Figure 7 and Figure 8 , the method includes the following content:

[0116] S100: In response to a split - screen control instruction, display a first window and a second window in the user interface.

[0117] The controller 250 can respond to a split - screen control instruction input by the user and divide at least two split - screen windows in the user interface. During the process of dividing the split - screen windows, the controller 250 can detect the number of split - screen windows, and thus determine the planning method of the split - screen windows according to the number of windows. For example, when the user interface includes two split - screen windows, the split - screen windows can be divided in two ways: left - right division and up - down division. When the user interface includes three split - screen windows, in addition to the horizontal division and vertical division of the split - screen windows, the controller 250 can also adopt a "pin" - shaped division, etc. The split - screen windows can also be floating - displayed on the upper layer of the user interface, that is, the split - screen windows are floating windows, and the display device 200 can determine the planning layout of the split - screen windows by responding to the user's dragging of the floating window through the touch function.

[0118] It should be understood that the present embodiment does not specifically limit the number of split-screen windows divided in the user interface, but since the display area of ​​the split-screen window is inversely correlated with the number of the split-screen windows, that is, in the display area of ​​the display 260, the more the number of split-screen windows, the smaller the display area of ​​each split-screen window. Therefore, the controller 250 should ensure that the display area of ​​the split-screen window is greater than or equal to the minimum window display area, so that the user can clearly see the split-screen images displayed in the split-screen window.

[0119] In order to reduce the conflict of media audio playback of different media data, the controller 250 can determine the first window and the second window in the split-screen window, the first window is the main window, and the second window is the sub-window, that is, the split-screen window other than the first window in the split-screen window. The display device 200 can control the audio output device 270 to play the media audio of one type of media data in the multi-window mode, so the split-screen window only includes one first window. According to the number of split-screen windows, the second window can include one or more, for example, Fig. 9 As shown, when the user interface includes three split-screen windows, one of which is a first window, the remaining two split-screen windows are second windows, and the split-screen windows include two second windows. Fig. 9 The user interface shown is the display interface of the horizontally divided split-screen window in the aforementioned embodiment.

[0120] For another example, when the user interface includes two split-screen windows, one of which is a first window and the remaining split-screen window is a second window, the split-screen windows include a second window.

[0121] In some embodiments, the split-screen control instruction includes a main window mark, and the main window mark is used to specify the first window in the split-screen window. In order to determine the first window in the split-screen window, the controller 250 can obtain the main window mark in the split-screen control instruction, and set the split-screen window specified by the main window mark in the split-screen window as the first window. If the split-screen control instruction does not include the split-screen control instruction, the split-screen window at the top of the split-screen window sequence can be set as the first window according to the preset window sorting rule.

[0122] S200: When the first window is selected in the two split-screen windows, first media asset data and second media asset data are acquired.

[0123] Different media data can be played in different split-screen windows, or the same media data can be played. Taking the case where two split-screen windows are included in the user interface as an example, after dividing the split-screen windows, the user needs to select the first window in the split-screen window and obtain the media data played in the first window. To this end, the user can enter a media selection instruction in the user interface of the split-screen window, and the controller 250 can respond to the selection instruction and play the media data specified by the selection instruction in the split-screen window. The first media data is the media data played in the first window, and the second media data is the media data played in the second window. It should be understood that the media data can be live broadcast data, locally cached audio and video data, audio and video data played through a server, etc.

[0124] In some embodiments, the display device 200 can split the split-screen window in response to the split-screen control instruction when the media data is being played. At this time, the display device 200 can use the media screen of the currently played media data as the split-screen screen of one of the split-screen windows. If the split-screen window is marked with a main window mark, the currently played media data is the first media data.

[0125] S300: Acquire first subtitle data.

[0126] The first subtitle data is obtained by semantically understanding the second media data. After obtaining the second media data, since the second window does not play the media audio, in order to facilitate the user to understand the media picture corresponding to the second media data without listening to the media audio of the second media data, the controller 250 can perform speech recognition on the media audio of the second media data, and perform semantic understanding on the speech recognition result obtained by the speech recognition, thereby generating the first subtitle data.

[0127] It should be noted that the process of performing semantic understanding on the second media asset data can be performed on the server side. For example, the controller 250 can send the second media asset data to the server side, and the server side can perform semantic understanding on the second media asset data, generate first subtitle data, and feed the first subtitle data back to the display device 200. The controller 250 can directly obtain the first subtitle data sent by the server side.

[0128] In some embodiments, when the display device 200 is deployed with a semantic understanding model for performing semantic understanding, the process of performing semantic understanding on the second media asset data can also be performed by the semantic understanding model. At this time, the controller 250 can input the second media asset data into the semantic understanding model to perform speech recognition and semantic understanding on the second media asset data through the semantic understanding model, thereby outputting the first subtitle data. In the embodiments of the present application, other methods of obtaining the first subtitle data can also be included, for example, obtaining the first subtitle data in the subtitle track of the second media asset data, etc. This embodiment does not specifically limit the method of obtaining the first subtitle data.

[0129] In some embodiments, the controller 250 can convert the media audio in the media data into subtitle data in real time or non-real time through speech recognition and natural language processing technology. After the controller 250 receives the second media data, the media audio in the second media data can be parsed. Taking the second media data as live broadcast data with strong real-time performance as an example, the controller 250 can obtain the radio frequency signal of the live broadcast data through an antenna or a wired network. The radio frequency signal is down-converted and analog-to-digital converted (ADC) to obtain a digital intermediate frequency signal, and then undergoes digital signal processing, such as equalization and error correction, to obtain transport stream data (Transport Stream, TS).

[0130] The transport stream data includes multiple data such as video and audio of live data, which can be distinguished by a unique identifier PID (Packet Identifier). In order to parse the media audio from the transport stream data, the controller 250 can filter out the media audio packets from the transport stream data according to the PID of the media audio through a demultiplexing device (Demux), and decode the media audio packets to obtain the media audio of the live data.

[0131] After parsing and obtaining the media audio, the controller 250 may convert the media audio into first subtitle data through speech recognition and natural language processing technology.

[0132] In some embodiments, if the language of the first subtitle data is different from the user's native language, the user cannot directly understand the semantics of the media audio through the first subtitle data. For example, if the second media data is a teaching video in English, after performing voice conversion and semantic understanding, the first subtitle data obtained is English text data. Users whose native language is not English cannot directly understand the content of the second window based on the English subtitle data.

[0133] In order to solve the problem that the subtitle language is different from the user's native language, such as Fig.10As shown, the user can input the target language information of the second window to the display device 200. After obtaining the target language information, the controller 250 can obtain the voice text data, and the voice text data is obtained by performing semantic understanding based on the media audio corresponding to the second media data. The voice text data is text data with the same language as the media audio of the second media data. After obtaining the voice text data, the controller 250 can detect the language information of the voice text data. If the language information of the voice text data is the same as the target language information, it means that the user can directly understand the content of the second window based on the voice text data, and the controller 250 can generate the first subtitle data based on the voice text data. If the language information of the voice text data is different from the target language information, the controller 250 can perform language translation on the voice text data according to the target language information to obtain the first subtitle data.

[0134] For example, the user can input Chinese as the target language information to the display device 200, and the second media data is a segment of English video. The controller 250 can perform semantic understanding on the media audio of the English video to obtain a segment of English text data. The language of the English text data is different from the Chinese language. Therefore, the controller 250 can convert the English text data into Chinese text data that the user can understand according to the Chinese language, thereby generating the first subtitle data.

[0135] It should be noted that the controller 250 can perform language translation on the voice text data of each second media data according to the target language information, and the second media data may include different language information. For example, the user interface includes two second windows, and the controller 250 can obtain two second media data accordingly. If the language information of the two second media data is different from the target language information, and the language information of the two second media data is also different from each other, the controller 250 will perform different language translation operations on different second media data according to the target language information.

[0136] Fig.11 This is an interface diagram of a display device displaying second media data in different languages ​​in an embodiment of the present application. Fig.11 If the speech text data obtained after semantic understanding of one of the second media asset data A is English text data, the controller 250 can perform an English-to-Chinese translation process on the English text data, and translate the English text data into Chinese text data to obtain the first subtitle data A corresponding to the second media asset data A. If the speech text data obtained after semantic understanding of the other second media asset data B is French text data, the controller 250 can perform a French-to-Chinese translation process on the French text data, and translate the French text data into Chinese text data to obtain the first subtitle data B corresponding to the second media asset data B.

[0137] In some embodiments, the controller 250 may also perform audio-to-text processing on the media audio through a neural network model. Based on this, the display device 200 may also include a memory configured to store a semantic understanding model. Fig.12 As shown, in the process of performing semantic understanding on the media audio of the second media data, the controller 250 can input the media audio of the second media data into the semantic understanding model, and the semantic understanding model can perform semantic recognition on the media audio and extract the voice features and semantic features of the media audio. Voice features are features extracted from the media audio for identification and processing, and semantic features are specific meanings or information contained in the media audio. The controller 250 can generate voice text data based on the voice features and semantic features.

[0138] S400: Control the display to display a media picture corresponding to the first media data in the first window, and to display a media picture corresponding to the second media data and the first subtitle data in the second window.

[0139] After generating the first subtitle data, the controller 250 can control the display 260 to display the media picture corresponding to the first media data in the first window, and to display the media picture corresponding to the second media data in the second window, so that the user can simultaneously watch the media pictures displayed in multiple split-screen windows through the user interface.

[0140] In the second window, the controller 250 may further control the display 260 to display the first subtitle data in the second window, so that the user can understand the media picture corresponding to the second media data according to the first subtitle data.

[0141] S500: When the display is displaying a media picture corresponding to the first media data, the audio output device is controlled to play the media audio of the first media data.

[0142] When the display device 200 is in the multi-window mode, the audio output device 270 plays the media audio corresponding to different media data at the same time, which causes the user to be unable to hear the media audio clearly. Based on the above problem, in this embodiment, the controller 250 can control the audio output device 270 to play the media audio of the media data played by the main window, that is, when the display 260 displays the media screen corresponding to the first media, the audio output device 270 is controlled to play the media audio of the first media data, so that the user can understand the media screen corresponding to the first media data through the media audio of the first media data, and because the audio output device 270 does not play or plays the media audio corresponding to the second media data in silence, the audio output device 270 only inputs the media audio corresponding to the first media data, so that the user can understand the meaning of the screen of the first window through the media audio and screen of the first media data, and no other media audio has an auditory impact on the media audio being played. The user can also understand the screen meaning of the second media data played in the second window according to the first subtitle data, thereby improving the user viewing experience during split-screen playback.

[0143] In some embodiments, the user may also input a window switching instruction to the display device 200 to actively switch the main window. Based on this, the controller 250 may respond to the window switching instruction, set the split screen window specified by the window switching instruction as the main window, and switch the original main window to the secondary window. Depending on the number of second windows, the controller 250 may perform the main window switching in the following two cases.

[0144] like Fig.13 As shown, when the number of windows of the second window is one, the user interface only includes a first window, that is, the current main window, and a second window. When the controller 250 receives a window switching instruction, the main window must be switched from the first window to the second window, that is, the second window is switched to the main window, and the first window is switched to the sub-window. To this end, the controller 250 can semantically understand the media audio of the first media data and obtain the second subtitle data. After obtaining the second subtitle data, the controller 250 can control the display 260 to display the second subtitle data in the first window, and control the audio output device 270 to stop playing the media audio of the first media data and instead play the media audio of the second media data. After completing the main window switching, the user can understand the content of the media screen corresponding to the first media data through the second subtitle data, and understand the content of the second media data through the media audio of the second media data and the media screen corresponding to the second media data. After the window switching is performed, the display 260 can continue to display the first subtitle data in the second window.

[0145] When there are multiple second windows, the controller 250 may update the second window specified by the window switching instruction to the first window. Fig.14As shown, the user interface includes a split-screen window A, a split-screen window B, and a split-screen window C, wherein the split-screen window A is a first window, and the split-screen window B and the split-screen window C are second windows respectively. The user can input a window switching instruction to specify the split-screen window B, and the controller 250 can set the split-screen window B as the first window and the split-screen window A as the second window in response to the window switching instruction. After the split-screen window A is set as the second window, the controller 250 can perform semantic understanding on the second media data played by the split-screen window A to obtain the second subtitle data, and control the audio output device 270 to stop playing the media audio of the first media data of the split-screen window A, and instead play the media audio of the second media data of the split-screen window B.

[0146] In some embodiments, the first media asset data or the second media asset data may include media asset subtitle data, which is subtitle data parsed from the transport stream data. The controller 250 may detect the media asset subtitle data in the second media asset data. If the media asset subtitle data is detected, the controller 250 may directly generate the first subtitle data based on the media asset subtitle data, thereby reducing the time for executing the semantic understanding process on the media asset audio and improving the efficiency of subtitle generation.

[0147] In some embodiments, when the language information of the media asset subtitle data is different from the target language information, the controller 250 may display the first subtitle data and the media asset subtitle data in the second window at the same time. In the process of displaying the first subtitle data and the media asset subtitle data, in order to reduce mutual occlusion between the subtitle data, the controller 250 may obtain a first display position of the media asset subtitle data relative to the second window, and control the display 260 to display the first subtitle data at a second display position other than the first display position.

[0148] During the process of displaying the media resource screen in the first window and the second window, the user can also adjust the window size of the first window or the second window. Fig.15 As shown, the user can input a first zoom instruction based on the first window, and the controller 250 can obtain the first zoom ratio of the first window in response to the first zoom instruction, and perform proportional zooming on the first window according to the first zoom ratio. Since the user interface includes a first window and a second window, after the first window is zoomed, the second window will be adaptively zoomed according to the zoom of the first window. For example, when the first window is proportionally enlarged, the display area of ​​the first window increases, while the display area of ​​the user interface is fixed. Therefore, the second window should be reduced according to the enlargement ratio of the first window. For example, when the first window is proportionally reduced, the second window should be enlarged according to the reduction ratio of the first window.

[0149] Therefore, the controller 250 can calculate the remaining display area according to the display area of ​​the user interface and the first window after the scaling is performed. The remaining display area is the display area other than the first window after the scaling is performed. Fig.15 In the interface shown in FIG. 1 , the shaded area and the second window together form a display area. After obtaining the remaining display area, as shown in FIG. Fig.16 As shown, the controller 250 may perform proportional scaling on the second window to fill the remaining display area.

[0150] When scaling a split-screen window, if the scaling ratio is too large, part of the split-screen window will be outside the display area of ​​the user interface, resulting in incomplete display of the screen in the split-screen window. If the scaling ratio is too large, the split-screen window will not be able to completely fill the remaining display area, reducing the efficiency of using the display area.

[0151] In order to solve the above-mentioned problems that occur during the split-screen window scaling process, in some embodiments, the controller 250 can calculate the display ratio of the second window to the remaining display area, and perform proportional scaling on the second window according to the display ratio. In the process of calculating the display ratio, the controller 250 can obtain the first display area area of ​​the remaining display area and the second display area area of ​​the second window, and calculate the display ratio based on the display area areas of the two. For example, if the area of ​​the first display area is 1.5 times the area of ​​the second display area, the controller 250 can set the display ratio of the second window to 1.5.

[0152] The split-screen window may include a horizontal window and a vertical window. The above embodiment is a case where the window display mode of the second window corresponds to the display area shape of the remaining display area. For example, the second window is a horizontal window, and the display area shape of the remaining display area is also a horizontal rectangular display area. If the window orientation of the second window does not correspond to the display area shape of the remaining display area, the controller 250 may calculate the maximum zoom ratio of the remaining second window in the remaining display area according to the window orientation of the second window. The maximum zoom ratio is calculated based on the area boundary of the remaining display area.

[0153] In some embodiments, the controller 250 may scale the first subtitle data in proportion to the display ratio during the scaling of the second window, so that the first subtitle data is scaled according to the size of the second window.

[0154] In some embodiments, the controller 250 can display the first subtitle data according to the original display ratio during the process of scaling the second window. However, when the split-screen window is too small and the display ratio of the subtitle data remains unchanged, the subtitle data will block part of the display area, causing the user to be unable to see the complete media asset screen. To this end, the controller 250 can calculate the display area of ​​the second window. If the display area is less than or equal to the preset area threshold, the controller 250 can semantically summarize the text content of the first subtitle data to obtain subtitle introduction data. In the above process, the controller 250 can input the first subtitle data into the semantic understanding model, extract the semantic result of the first subtitle data through the semantic understanding model, and semantically summarize the semantic result to obtain subtitle introduction data with a word count less than the first subtitle data. After obtaining the subtitle introduction data, the controller 250 can control the display 260 to cancel the display of the first subtitle data, and display the subtitle introduction data in the second window, the original display position of the first subtitle data, thereby reducing the occlusion of the display screen by more subtitle data.

[0155] In some embodiments, the controller 250 can detect the voice timestamp in the media audio when displaying the media picture corresponding to the second media data and the first subtitle data in the second window. The voice timestamp is the timestamp of the voice data in the media audio. The first subtitle data corresponds to the voice lines in the second media data. Therefore, the controller 250 can display the first subtitle data in the second window according to the voice timestamp to improve the synchronization of audio and video when the second media data is played.

[0156] Some embodiments of the present application also provide a subtitle display method based on split-screen playback, which is applied to a display device 200, wherein the display device 200 should at least include a display 260 configured to display a user interface; an audio output device 270 configured to play audio data; and a controller 250. The method includes:

[0157] S100: In response to a split-screen control instruction, display a first window and a second window on the user interface.

[0158] S200: When the first window is selected in the two split-screen windows, first media asset data and second media asset data are acquired.

[0159] The first media data is the media data played in the first window, and the second media data is the media data played in the second window.

[0160] S300: Acquire first subtitle data.

[0161] The first subtitle data is obtained after semantic understanding processing of the second media asset data.

[0162] S400: Control the display to display a media picture corresponding to the first media data in the first window, and to display a media picture corresponding to the second media data and the first subtitle data in the second window.

[0163] S500: When the display is displaying a media picture corresponding to the first media data, the audio output device is controlled to play the media audio of the first media data.

[0164] It can be seen from the above technical solutions that the present application provides a display device and a subtitle display method based on split-screen playback, wherein the method displays a first window and a second window in the user interface in response to a split-screen control instruction, and obtains the first media data and the second media data when the first window is selected in the two split-screen windows, and semantically understands the media audio of the second media data to obtain the first subtitle data. The display is controlled to display the media screen corresponding to the first media data in the first window, and the media screen and the first subtitle data corresponding to the second media data in the second window, and the audio output device is controlled to play the media audio of the first media data. The present application obtains the first subtitle data by semantically understanding the media audio in the second window, so that the user can understand the meaning of the screen of the first window through the media audio and screen of the first media data, and understand the meaning of the screen of the second media data played in the second window according to the first subtitle data, thereby improving the user viewing experience during split-screen playback.

[0165] The same and similar parts between the various embodiments in this specification can be referenced to each other and will not be described again here.

[0166] Those skilled in the art can clearly understand that the technology in the embodiments of the present invention can be implemented by means of software plus a necessary general hardware platform. Based on this understanding, the technical solution in the embodiments of the present invention is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which can be stored in a storage medium such as ROM / RAM, a disk, an optical disk, etc., and includes a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods of various embodiments of the present invention or certain parts of the embodiments.

[0167] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present application.

[0168] For the convenience of explanation, the above description has been made in conjunction with specific embodiments. However, the above exemplary discussion is not intended to be exhaustive or limit the embodiments to the specific forms disclosed above. Based on the above teachings, various modifications and variations can be obtained. The selection and description of the above embodiments are to better explain the principles and practical applications, so that those skilled in the art can better use the embodiments and various different variations of the embodiments suitable for specific use considerations.

Claims

1. A display device, characterized in that: include: a display configured to display a user interface; an audio output device configured to play audio data; The controller is configured as: In response to the split-screen control instruction, displaying a first window and a second window on the user interface; When the first window is selected in the two split-screen windows, first media data and second media data are acquired, wherein the first media data is the media data played in the first window, and the second media data is the media data played in the second window; Acquire first subtitle data, where the first subtitle data is obtained after semantic understanding processing of the second media asset data; Controlling the display to display a media picture corresponding to the first media data in the first window, and to display a media picture corresponding to the second media data and the first subtitle data in the second window; When the display is displaying the media picture corresponding to the first media data, the audio output device is controlled to play the media audio of the first media data.

2. The display device according to claim 1, characterized in that When the number of the second window is one, the controller is further configured to: In response to a window switching instruction input by a user, semantically understand the media asset audio of the first media asset data to obtain second subtitle data; The display is controlled to display the second subtitle data in the first window, and the audio output device is controlled to stop playing the media audio of the first media data and play the media audio of the second media data.

3. The display device according to claim 1, characterized in that Before the controller executes the step of acquiring the first subtitle data, the controller is further configured to: Detecting media asset subtitle data of the second media asset data; If the media asset subtitle data is detected, the first subtitle data is generated according to the media asset subtitle data.

4. The display device according to claim 1, characterized in that The controller performs the step of semantically acquiring the first subtitle data, which is specifically configured as follows: Get the target language information input by the user; Acquire speech text data, where the speech text data is obtained by performing semantic understanding on the media asset audio corresponding to the second media asset data; If the language information of the speech text data is different from the target language information, language translation is performed on the speech text data according to the target language information to obtain first subtitle data.

5. The display device according to claim 4, characterized in that The controller further includes a memory configured to store a semantic understanding model, and the controller performs the step of acquiring speech text data, which is specifically configured as follows: Inputting the media asset audio of the second media asset data into the semantic understanding model to extract speech features and semantic features of the media asset audio of the second media asset data; The speech text data is generated according to the speech feature and the semantic feature.

6. The display device according to claim 4, characterized in that After the controller executes the step of displaying the media asset picture corresponding to the first media asset data in the first window, the controller is further configured to: In response to a first scaling instruction input by a user based on the first window, scaling the first window in proportion; Calculate a remaining display area according to the display area of ​​the user interface and the first window after scaling; The second window is scaled proportionally to fill the remaining display area.

7. The display device according to claim 6, characterized in that The controller performs a step of scaling the second window in equal proportion to fill the remaining display area, and is specifically configured as follows: Calculating a display ratio of the second window to the remaining display area; The second window is proportionally scaled according to the display ratio, and the first subtitle data is proportionally scaled according to the display ratio during the proportional scaling of the second window.

8. The display device according to claim 7, characterized in that The step of performing scaling of the first subtitle data in proportion to the display ratio, the controller further comprises: Calculating the display area of ​​the second window; If the display area is less than or equal to a preset area threshold, semantically summarizing the text content corresponding to the first subtitle data using the semantic understanding model to obtain subtitle introduction data; The display is controlled to cancel displaying the first subtitle data, and the subtitle introduction data is displayed in the second window.

9. The display device according to claim 1, characterized in that: The controller performs the step of displaying the media asset picture corresponding to the second media asset data and the first subtitle data in the second window, and is specifically configured as follows: Detecting a voice timestamp in the media asset audio of the second media asset data, where the voice timestamp is a timestamp of the voice data in the media asset audio; The first subtitle data is displayed in the second window according to the speech timestamp.

10. A subtitle display method based on split-screen playback, characterized in that: The display device as claimed in any one of claims 1 to 9, wherein the display device comprises a display configured to display a user interface; an audio output device configured to play audio data; Controller; the method comprising: In response to the split-screen control instruction, displaying a first window and a second window on the user interface; When the first window is selected in the two split-screen windows, first media data and second media data are acquired, wherein the first media data is the media data played in the first window, and the second media data is the media data played in the second window; Acquire first subtitle data, where the first subtitle data is obtained after semantic understanding processing of the second media asset data; Controlling the display to display a media picture corresponding to the first media data in the first window, and to display a media picture corresponding to the second media data and the first subtitle data in the second window; When the display is displaying the media picture corresponding to the first media data, the audio output device is controlled to play the media audio of the first media data.

Images