Display device and display method

Abstract

The invention is suitable for the technical field of display equipment, and provides display equipment and a display method, and the display equipment comprises a display which is configured to display a menu interface and a video interface; the controller is configured to obtain to-be-displayed video data of the video interface; determining target video data in the video data to be displayed; wherein the target video data is determined according to the detected menu interface and / or according to a target object in the to-be-displayed video data, and the target object refers to an object needing fuzzy processing in the to-be-displayed video data; and performing fuzzy processing on the target video data, merging the target video data after fuzzy processing and the to-be-displayed video data, and sending the merged video data to the video interface for display. According to the invention, when the video is played in the display, the display effect of fuzzy processing can be improved.

Description

Technical Field

[0001] The present application relates to the technical field of display devices, and in particular to a display device and a display method. Background Art

[0002] The display device can display multiple interfaces, including a menu interface (e.g., a menu interface displayed by an on-screen menu adjustment method (OnScreen Display, OSD)) and a video interface (e.g., a video played in real time by the display device), etc. When the display device displays multiple interfaces, the background can be blurred or a specific video area can be emphasized by blurring, thereby improving the display effect.

[0003] At present, blur processing is mainly implemented at the OSD level. On the one hand, various parameters (such as color, resolution, etc.) can be adjusted through the function menu at the OSD level for blur adjustment; on the other hand, OSD information (such as text, etc.) can be generated to block the video content. However, the method of adjusting various parameters through the function menu is relatively rough, and it is impossible to accurately determine the video area that needs to be blurred. In addition, when blur processing is performed by blocking OSD information, the OSD information does not belong to the content of the video, and the display effect of blur processing is poor. Therefore, the above method will result in a poor display effect of blur processing. Summary of the invention

[0004] The present application provides a display device and a display method, which can improve the display effect of blur processing when playing a video on a display.

[0005] In a first aspect, some embodiments of the present application provide a display device, including a display and a controller. The display is configured to display a menu interface and a video interface; the controller is configured to:

[0006] Obtaining video data to be displayed on the video interface;

[0007] Determining target video data in the video data to be displayed; wherein the target video data is determined according to the detected menu interface and / or according to a target object in the video data to be displayed, and the target object refers to an object in the video data to be displayed that needs to be blurred;

[0008] The target video data is fuzzy processed, the fuzzy target video data and the to-be-displayed video data are merged, and the merged video data is sent to the video interface for display.

[0009] Since the target video data is determined based on the detected menu interface and / or based on the target object in the video data to be displayed, it means that the target video data in the video data to be displayed can be determined in a targeted manner, so the accuracy of blurring the target video data can be improved. In addition, by blurring the above-mentioned target video data, merging the blurred target video data and the video data to be displayed, and sending the merged video data to the video interface for display, since the blurring process and the subsequent merging process are directly applied to the video data to be displayed in the video interface, the problem of abrupt display effects and easy delays during blurring at the OSD level can be avoided, effectively improving the display effect of blurring during video playback.

[0010] In a second aspect, some embodiments of the present application further provide a display method, which is applied to the display device provided in the first aspect, and the method includes:

[0011] Get the video data to be displayed on the video interface;

[0012] Determining target video data in the video data to be displayed; wherein the target video data is determined according to a detected menu interface and / or according to a target object in the video data to be displayed, and the target object refers to an object in the video data to be displayed that needs to be blurred;

[0013] The target video data is fuzzy processed, the fuzzy target video data and the to-be-displayed video data are merged, and the merged video data is sent to the video interface for display.

[0014] In a third aspect, a display device is provided, comprising a module for executing the display method in the second aspect.

[0015] In a fourth aspect, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a display device, the display device executes the display method in the second aspect.

[0016] In a fifth aspect, a computer program product is provided, the computer program product comprising: a computer program, when the computer program is executed by a display device, the display device executes the display method in the second aspect.

[0017] It can be understood that the beneficial effects of the second to fifth aspects mentioned above can be found in the relevant description of the first aspect mentioned above, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0019] Figure 1 A schematic diagram of an operation scenario between a display device and a control device provided in some embodiments of the present application;

[0020] Figure 2 A schematic diagram of the hardware configuration of a display device provided in some embodiments of the present application;

[0021] Figure 3 A schematic diagram of the hardware configuration of a control device provided in some embodiments of the present application;

[0022] Figure 4 A schematic diagram of software configuration of a display device provided in some embodiments of the present application;

[0023] Figure 5 A structural block diagram of a display device provided in some embodiments of the present application;

[0024] Figure 6 A schematic flow chart of a display method provided in some embodiments of the present application;

[0025] Figure 7 A schematic diagram of determining target video data according to a location area provided in some embodiments of the present application;

[0026] Figure 8 An interactive schematic diagram of determining a first target video provided in some embodiments of the present application;

[0027] Fig. 9 An interactive schematic diagram of determining a second target video provided in some embodiments of the present application;

[0028] Fig.10 An interactive schematic diagram of fuzzy processing in a controller provided in some embodiments of the present application;

[0029] Fig.11 A schematic diagram of the structure of a display device provided in some embodiments of the present application. DETAILED DESCRIPTION

[0030] The following embodiments are described in detail, and examples thereof are shown in the accompanying drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The implementations described in the following embodiments do not represent all implementations consistent with the present application. They are only examples of systems and methods consistent with some aspects of the present application as detailed in the claims.

[0031] It should be noted that the brief description of terms in this application is only for the convenience of understanding the embodiments described below, and is not intended to limit the embodiments of this application. Unless otherwise specified, these terms should be understood according to their common and usual meanings.

[0032] In the description of the embodiments of the present application, unless otherwise specified, " / " means or, for example, A / B can mean A or B; "and / or" in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships, for example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, in the description of the embodiments of the present application, "multiple" means two or more than two.

[0033] The terms "first", "second", "third", etc. in the specification and claims of this application and the above drawings are used to distinguish similar or similar objects or entities, and do not necessarily mean to limit a specific order or sequence, unless otherwise noted. It should be understood that the terms used in this way can be interchangeable under appropriate circumstances.

[0034] The terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover but not exclude inclusion, for example, a product or device comprising a list of components is not necessarily limited to all the components expressly listed but may include other components not expressly listed or inherent to such product or device.

[0035] The term "module" refers to any known or later developed hardware, software, firmware, artificial intelligence, fuzzy logic, or combination of hardware and / or software code that is capable of performing the functions associated with that element.

[0036] In the embodiment of the present application, the display device 200 generally refers to a device with image display and data processing capabilities. For example, the display device 200 includes but is not limited to a smart TV, a mobile terminal, a computer, a monitor, an advertising screen, a wearable device, a virtual reality device, an augmented reality device, etc.

[0037] Figure 1 This is a schematic diagram of an operation scenario between a display device and a control device provided in some embodiments of the present application. Figure 1As shown in FIG. 1 , the user can operate the display device 200 through touch operation, the mobile terminal 300 and the control device 100. For example, the control device 100 can be a remote controller, a stylus pen, a handle, etc.

[0038] The mobile terminal 300 can be used as a control device for performing human-computer interaction between the user and the display device 200. The mobile terminal 300 can also be used as a communication device for establishing a communication connection with the display device 200 and performing data interaction. In some embodiments, the mobile terminal 300 can install software applications with the display device 200, and achieve connection and communication through a network communication protocol to achieve the purpose of one-to-one control operation and data communication. The audio and video content displayed on the mobile terminal 300 can also be transmitted to the display device 200 to achieve a synchronous display function.

[0039] like Figure 1 As also shown in FIG. 4 , the display device 200 also communicates data with the server 400 through various communication methods. The display device 200 may be allowed to communicate and connect through a local area network (LAN), a wireless local area network (WLAN), and other networks.

[0040] The display device 200 may provide a broadcast receiving television function, and may also additionally provide an intelligent network television function with a computer support function, including but not limited to network television, smart television, Internet Protocol television (IPTV), and the like.

[0041] Figure 2 Some embodiments of the present application provide Figure 1 2 is a block diagram of the hardware configuration of the display device 200.

[0042] In some embodiments, the display device 200 may include at least one of a tuner 210, a communication device 220, a detector 230, a device interface 240, a controller 250, a display 260, an audio output device 270, a memory, a power supply, and a user input interface.

[0043] In some embodiments, the detector 230 is used to collect signals of the external environment or external interaction. For example, the detector 230 includes a light receiver, a sensor for collecting the intensity of ambient light; or, the detector 230 includes an image collector, such as a camera, which can be used to collect external environment scenes, user attributes or user interaction gestures; or, the detector 230 includes a sound collector, such as a microphone, etc., for receiving external sounds.

[0044] In some embodiments, the display 260 includes a display function component for presenting a picture, and a driving component for driving an image display. The display 260 is used to receive an image signal output from the controller 250 for display. For example, the display 260 can be used to display video content, image content, and components of a menu control interface and a user control UI interface.

[0045] In some embodiments, the display 260 can be a liquid crystal display (LCD), an organic light-emitting diode display (OLED), a plasma display (PDP), a vacuum fluorescent display (VFD), a field emission display (FED), or a projection display, and can also be a projection device or a projection screen.

[0046] In some embodiments, the communication device 220 is a component for communicating with an external device or server 400 according to various communication protocol types. The display device 200 may be provided with a plurality of communication devices 220 according to different supported communication modes. For example, when the display device 200 supports wireless network communication, the display device 200 may be provided with a communication device 220 including a WiFi function. When the display device 200 supports Bluetooth connection communication, the display device 200 needs to be provided with a communication device 220 including a Bluetooth function.

[0047] The communication device 220 can enable the display device 200 to communicate with the external device or server 400 by wireless or wired connection. Among them, the wired connection can connect the display device 200 with the external device through components such as data cables and interfaces. The wireless connection can connect the display device 200 with the external device through wireless signals or wireless networks. The display device 200 can establish a connection relationship with the external device directly, or indirectly establish a connection relationship through a gateway, a router, a connection device, etc.

[0048] In some embodiments, the controller 250 may include at least one of a central processing unit, a video processor, an audio processor, a graphics processor, and a power processor, and a first interface to an nth interface for input / output. The controller 250 controls the operation of the display device and responds to the user's operation through various software control programs stored in the memory. The controller 250 controls the overall operation of the display device 200.

[0049] In some embodiments, the controller 250 and the tuner-demodulator 210 may be located in different separate devices, that is, the tuner-demodulator 210 may also be located in an external device of the main device where the controller 250 is located, such as an external set-top box.

[0050] In some embodiments, the user may input a user command through a graphical user interface (GUI) displayed on the display 260 , and the user input interface receives the user input command through the graphical user interface (GUI).

[0051] In some embodiments, the audio output device 270 may be a local speaker of the display device 200, or may be an external audio output device of the display device 200. In particular, for the external audio output device of the display device 200, the display device 200 may also be provided with an external audio output terminal, and the audio output device may be connected to the display device 200 through the external audio output terminal to output the sound of the display device 200.

[0052] In some embodiments, the user input interface 280 may be used to receive instructions from a user.

[0053] Figure 3 Some embodiments of the present application provide Figure 1 The hardware configuration diagram of the control device in the figure. Figure 3 As shown, the control device 100 may include: a controller 110, a communication interface 130, a user input / output interface, a memory, and a power supply.

[0054] The control device 100 is configured to control the display device 200 , and can receive user input operation instructions, and convert the operation instructions into instructions that the display device 200 can recognize and respond to, playing the role of an interactive intermediary between the user and the display device 200 .

[0055] In some embodiments, the control device 100 may be a smart device, for example, the control device 100 may be installed with various applications for controlling the display device 200 according to user needs.

[0056] In some embodiments, Figure 1 As shown, the mobile terminal 300 or other intelligent electronic devices can play a similar function as the control device 100 after installing the application for controlling the display device 200 .

[0057] The controller 110 includes a processor 112, a RAM 113, a ROM 114, a communication interface 130, and a communication bus. The controller 110 is used to control the operation and operation of the control device 100, as well as the communication and cooperation between the internal components and the external and internal data processing functions.

[0058] The communication interface 130 implements communication of control signals and data signals with the display device 200 under the control of the controller 110. The communication interface 130 may include at least one of other near field communication modules such as a WiFi chip 131, a Bluetooth module 132, and an NFC module 133.

[0059] The user input / output interface 140 , wherein the input interface includes at least one of other input interfaces such as a microphone 141 , a touch panel 142 , a sensor 143 , and a button 144 .

[0060] In some embodiments, the control device 100 includes at least one of a communication interface 130 and an input / output interface 140. The control device 100 is configured with a communication interface 130, such as a WiFi, Bluetooth, NFC or other module, and can encode the user input command through the WiFi protocol, Bluetooth protocol, or NFC protocol and send it to the display device 200.

[0061] The memory 190 is used to store various operating programs, data and applications for driving and controlling the control device 100 under the control of the controller. The memory 190 can store various control signal instructions input by the user.

[0062] The power supply 180 is used to provide operating power support for each component of the control device 100 under the control of the controller.

[0063] In order to perform user interaction, in some embodiments, the display device 200 may run an operating system. The operating system is a computer program for managing and controlling hardware resources and software resources in the display device 200. The operating system may provide a user interface (control the display device), allow the user to interact with the display device 200, and support the running of various application programs.

[0064] It should be noted that the operating system may be a native operating system based on a specific operating platform, or a third-party operating system deeply customized based on a specific operating platform. In the embodiment of the present application, the specific operating platform may be a mobile platform, and the operating system may be a mobile operating system based on the mobile platform, such as an Android system, an iOS system, a HarmonyOS, or a third-party operating system customized and developed based on the Android system.

[0065] The operating system can be divided into different modules or layers according to the functions implemented, such as Figure 4As shown, in some embodiments, the system is divided into four layers, from top to bottom, namely, the application layer (Applications) layer (referred to as "application layer"), the application framework layer (Application Framework) layer (referred to as "framework layer"), the system library layer and the kernel layer.

[0066] In some embodiments, the application layer is used to provide services and interfaces for applications so that the display device 200 can run applications and interact with users based on the applications. At least one application can be run in the application layer, and these applications can be system setting programs or clock programs that come with the operating system, etc.; they can also be applications developed by third-party developers. In specific implementations, the application packages in the application layer are not limited to the above examples.

[0067] The framework layer provides application programming interfaces (APIs) and programming frameworks for applications. The application framework layer includes some predefined functions. The application framework layer is equivalent to a processing center that determines the actions that applications in the application layer take. Through the API interface, applications can access system resources and obtain system services during execution.

[0068] like Figure 4 As shown, in the embodiment of the present application, the application framework layer includes a view system, managers, content providers, etc., wherein the view system can design and implement the interface and interaction of the application, and the view system includes lists, grids, text boxes, buttons, etc. The manager includes at least one of the following modules: an activity manager for interacting with all activities running in the system; a location manager for providing system services or applications with access to system location services; a package manager for retrieving various information related to the application package currently installed on the device; a notification manager for controlling the display and clearing of notification messages; and a window manager for managing icons, windows, toolbars, wallpapers, and desktop widgets on the user interface.

[0069] In some embodiments, the activity manager is used to manage the life cycle of each application and the general navigation back function, such as controlling the exit, opening, back, etc. of the application.

[0070] In some embodiments, the system runtime layer can provide support for the framework layer. When the framework layer is used, the operating system will run the instruction library contained in the system runtime layer, such as the C / C++ instruction library or the Android runtime library, to implement the functions to be implemented by the framework layer.

[0071] In some embodiments, the kernel layer is a functional layer between the hardware and software of the display device 200. The kernel layer can implement functions such as hardware abstraction, multitasking, and memory management. Figure 4 As shown, the kernel layer may be configured with hardware drivers, and the drivers included in the kernel layer may be at least one of the following drivers: audio driver, display driver, Bluetooth driver, camera driver, WIFI driver, USB driver, HDMI driver, sensor driver (such as fingerprint sensor, temperature sensor, pressure sensor, etc.), and power driver, etc.

[0072] It should be noted that the above example is only a simple division of the operating system functions and does not constitute a limitation on the specific operating system form of the display device 200 in the embodiment of the present application. Depending on factors such as the function of the display device and the type of operating system, the number of levels and specific level types contained in the operating system may be expressed in other forms.

[0073] In some related technologies, blurring of video data can be achieved through an on-screen menu adjustment method (On Screen Display, OSD), including: adjusting various parameters (for example, color, resolution, etc.) through the function menu at the OSD level for blurring adjustment, and generating OSD information to block the video content to achieve blurring. The blurring at the OSD level can be achieved in the following ways: 1. Take a screenshot of the video content to obtain a bitmap, use a blurring algorithm to process the bitmap, obtain a blurred image and set it as the background of the menu, or identify the content that needs to be blurred in the bitmap and block it by generating OSD information; 2. In mobile operating systems such as the Android system, blurring can be achieved by calling the relevant API interface provided by the system.

[0074] However, the above-mentioned blurring processing methods all require taking screenshots, and then performing algorithmic processing on the screenshot images. When there is no video, users generally fail to notice it. However, when playing the video, the display effect is rather abrupt based solely on the menu adjustment at the OSD level or the blurring processing of the OSD information. At the same time, when the video screen changes rapidly, since the blurring processing is mainly for screenshots, it will consume more computing resources, and there may be large delays or device freezes. Therefore, the display effect of the above-mentioned blurring processing method implemented at the OSD level is poor.

[0075] In order to improve the display effect of blur processing when playing videos on a display, an embodiment of the present application provides a display device and a display method. The method first obtains the video data to be displayed on the video interface, and determines the target video data in the above-mentioned video data to be displayed. Since the target video data is determined according to the detected menu interface and / or according to the target object in the video data to be displayed, it means that the target video data in the video data to be displayed can be determined in a targeted manner, so the accuracy of blur processing on the target video data can be improved. In addition, by blurring the above-mentioned target video data, merging the blurred target video data and the video data to be displayed, and sending the merged video data to the video interface for display, since the blurring process and the subsequent merging process are directly applied to the video data to be displayed in the video interface, the problem of abrupt display effect and easy delay during blurring at the OSD level can be avoided, and the display effect of blurring during video playback is effectively improved.

[0076] In order to facilitate further understanding of the technical solutions in some embodiments of the present application, the technical solutions of the display device and the display method, as well as how the technical solutions solve the above-mentioned technical problems are described in detail below in combination with some specific embodiments and drawings. The various embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. Obviously, the described embodiments are part of the embodiments of the present application, not all of them.

[0077] Figure 5 It shows that some embodiments of the present application provide Figure 1 In the structural block diagram of the display device, for the convenience of explanation, only the parts related to the embodiment of the present application are shown.

[0078] Reference Figure 5 , the display device includes a display 510 and a controller 520. Among them,

[0079] The display 510 is configured to display a menu interface and a video interface.

[0080] It should be understood that the above menu interface and video interface are interfaces of different levels of the display, that is, the contents displayed by the menu interface and the video interface are independent of each other. The above menu interface is used to display the function menu called by the user, such as the media center menu or the setting menu, etc.; the above video interface is used to display video data, such as real-time video playback.

[0081] The controller 520 is configured to: obtain the video data to be displayed of the above-mentioned video interface; determine the target video data in the above-mentioned video data to be displayed; wherein the above-mentioned target video data is determined based on the above-mentioned menu interface detected, and / or based on the target object in the above-mentioned video data to be displayed, and the above-mentioned target object refers to the object that needs to be blurred in the above-mentioned video data to be displayed; blur the above-mentioned target video data, merge the blurred target video data and the above-mentioned video data to be displayed, and send the merged video data to the above-mentioned video interface for display.

[0082] Figure 6 A flow chart of a display method provided by some embodiments of the present application is shown, and is described in detail as follows:

[0083] S610: Obtain video data to be displayed on the video interface.

[0084] The video data to be displayed refers to the video data to be displayed on the video interface. The controller can obtain the video data to be displayed from the server based on the play request. The server can be a local server, a remote server or a cloud server.

[0085] It should be understood that the above-mentioned play request can be a request actively triggered by the user, for example, the above-mentioned play request can be a play request triggered when the user selects video data in an application (also referred to as an application) through an input device such as a remote control or a stylus, or can be a play request triggered when the user selects video data in an application through a touch, gesture, or sound. Of course, the above-mentioned play request can also be a passively triggered request, for example, a play request passively triggered by a controller according to the video data to be played continuously, etc.

[0086] S620, determining target video data in the video data to be displayed; wherein the target video data is determined based on a detected menu interface and / or based on a target object in the video data to be displayed, and the target object refers to an object in the video data to be displayed that needs to be blurred.

[0087] The target video data refers to the video data to be displayed that needs to be blurred, and the target video data is the entire video screen or part of the video screen in the video interface when the video data to be displayed is displayed in the video interface.

[0088] It should be understood that the menu interface is usually displayed above the video interface.

[0089] When the display plays a video, if the user opens a menu interface, in order to improve the display effect, the target video data can be determined according to the menu information of the menu interface, wherein the menu information may include information such as the type and priority of the menu interface. If the priority of the menu interface opened by the user is higher, the entire video screen of the video data to be displayed on the video interface can be used as the target video data; if the priority of the menu interface opened by the user is lower, the partial video screen of the video data to be displayed on the video interface can be used as the target video data. The correspondence between the menu interface and the target video data can be set according to the actual application scenario.

[0090] Optionally, when the display plays a video, there may be video content in the video to be displayed that needs to be hidden or needs to be particularly emphasized, that is, the above-mentioned target object. At this time, the above-mentioned target object can be identified from the video to be displayed, and then the corresponding target video data can be determined according to the object information of the above-mentioned target object. The above-mentioned object information may include object type, object priority, etc. The corresponding method of the above-mentioned target object and target video data can be set according to the actual application scenario. The corresponding method of the above-mentioned target object and target video data can refer to the relevant content of the menu interface in the above embodiment, which will not be repeated here.

[0091] In the embodiment of the present application, by determining the target video data through the menu interface and / or the target object, the video data that needs to be blurred in the video data to be displayed can be determined more accurately, thereby improving the accuracy of the blurring process.

[0092] S630: perform blur processing on the target video data, merge the target video data after blur processing with the video data to be displayed, and send the merged video data to the video interface for display.

[0093] Specifically, the areas in the target video data where the color changes or transitions are more obvious can be identified first to obtain contour areas, and then these contour areas can be blurred to obtain the blurred target video data, and the blurred target video data and the video data to be displayed can be merged, and the merged video data can be displayed through the video interface.

[0094] In the embodiment of the present application, by blurring the target video data, since the target video data is the entire video screen or part of the video screen in the video interface when the video data to be displayed is displayed on the video interface, the video data displayed in the video interface can be blurred in a targeted manner. In addition, the target video data is blurred and merged with the video data to be displayed. Since the blurring and data merging processes act directly on the video interface, the problem of abrupt display effects during blurring at the OSD level can be avoided, as well as the problem of delays caused by the need to take screenshots during blurring at the OSD level, thereby effectively improving the display effects of blurring during video playback.

[0095] In some embodiments, the target video data includes first target video data and second target video data, and the controller determines the target video data in the video data to be displayed, including:

[0096] Encoding the video data to be displayed to obtain an encoded video stream;

[0097] In the case where the menu interface is detected, first target video data is determined; the first target video data is video data determined according to the position area of ​​the menu interface in the encoded video stream;

[0098] and / or,

[0099] The encoded video stream is subjected to target object detection according to a pre-trained target object detection model, and when the target object in the encoded video stream is detected, second target video data is determined; the second target video data is video data determined according to the position area of ​​the target object in the encoded video stream.

[0100] It should be understood that although the menu interface and the video interface are interfaces of different levels, they are both displayed based on the same display (i.e., the same screen). Therefore, in order to accurately determine the video data that needs to be blurred in the video interface (i.e., the above-mentioned target video data), the corresponding target video data can be determined based on the detected position area of ​​the menu interface and / or the position area of ​​the above-mentioned target object.

[0101] Specifically, after the above-mentioned controller obtains the video data to be displayed from the server, the video data to be displayed may contain different types of data at the same time, for example, video data, audio data, subtitle data, etc. Since the blur processing mainly acts on the video data, the video data can be separated first through a separation (Demux) operation, and then the separated video data is encoded using the codec to obtain the above-mentioned encoded video stream, and the first target video data and / or the second target video data are extracted from the above-mentioned encoded video stream.

[0102] Among them, when a menu interface called by the user is detected, the position area of ​​the currently called menu interface in the display can be determined, and then the first target video data in the above-mentioned encoded video stream is determined based on the position area and the above-mentioned menu information, and / or, the above-mentioned encoded video stream is subjected to target object detection using a pre-trained target object detection model to obtain the position area of ​​the target object in the display, and then the video data of the above-mentioned encoded video stream in the position area is determined based on the object information, that is, the above-mentioned second target video data.

[0103] In some examples, the above encoding can be performed by a preset codec (such as a MediaCodec codec). During encoding, the data encoding type, resolution and other parameters are input, and the codec outputs a coded video stream according to the above parameters, wherein the coded video stream can be video data in YUV format. The above pre-trained target object detection model can be a pre-trained neural network model, and the above neural network model can be one of a YOLO recognition model and an SSD model. During the training process, the location and object information (such as category) of the target object can be annotated in the training data (such as an image or video), and then the neural network model (such as a YOLO recognition model) can be trained using the annotated training data. The loss value is calculated based on the output result of the neural network model and the annotated information, and the parameters of the neural network model are adjusted based on the loss value. The above process is repeated until the loss value is less than or equal to the preset loss threshold, or the number of iterations meets the preset number of iterations, and the trained target object detection model is obtained.

[0104] It should be understood that different menu interfaces can be classified in advance according to the above menu information. It is assumed that the menu interfaces are divided into two categories of menu interfaces, wherein the first category of menu interfaces is a menu interface that needs to blur part of the video data in the video interface to highlight the emphasis, and the second category of menu interfaces is a menu interface that needs to blur all the video data in the video interface to highlight the emphasis. Figure 7As shown in 7-1, when it is detected that the menu interface called by the user is located above the video interface, if it is determined according to the menu information that the called menu interface is a first-type menu interface, then the above-mentioned first target video data can be a portion of the video screen 1 corresponding to the position area of ​​the menu interface when the video data to be displayed is displayed on the video interface; if it is determined according to the menu information that the called menu interface is a second-type menu interface, then the above-mentioned first target video data can be the entire video screen 2 of the video interface when the video data to be displayed is displayed on the video interface; of course, it is also possible to determine, based on actual conditions, that the first target video data is the portion of the video screen remaining in the entire video screen 2 of the video interface except for the portion of the video screen 1, which is not limited here.

[0105] Optionally, different target objects may also be classified in advance according to the above object information. It is assumed that the target objects are divided into two categories of target objects, wherein the first category of target objects are objects that need to be blurred for privacy protection, and the second category of target objects are objects that need to be blurred for emphasis. Figure 7 As shown in 7-2, if the target object is determined to be a first-category target object through the above-mentioned pre-trained target detection model, then the above-mentioned second target video data can be the partial video screen 3 corresponding to the position area of ​​the target object when the video data to be displayed is displayed on the video interface; if the target object is determined to be a second-category target object through the above-mentioned pre-trained target detection model, then the above-mentioned second target video data can be the partial video screen remaining in the entire video screen 4 of the video interface except for the partial video screen 3 when the video data to be displayed is displayed on the video interface.

[0106] It should be noted that the above menu interface and the classification of the above target objects can be determined according to actual conditions and are not limited here.

[0107] In the embodiment of the present application, the video data to be blurred in the video data to be displayed, i.e., the target video data, can be accurately determined through the position area of ​​the menu interface and / or the position area of ​​the target object, thereby improving the accuracy of the blurring process. In addition, when determining the target video data, the category of the menu interface or the target object is also considered, thereby improving the flexibility of determining the target video data.

[0108] In some embodiments, assuming that the detected menu interface is a first type of menu interface, the controller determines the first target video data when the menu interface is detected, including:

[0109] When the menu interface is detected, determining the menu area according to the menu position coordinates of the menu interface;

[0110] The video data corresponding to the menu area is extracted from the encoded video stream to obtain the first target video data.

[0111] It should be understood that each menu interface corresponds to an interactive component (such as an Activity). When the user calls the menu interface, the controller will actually start the corresponding interactive component (such as an Activity). When the interactive component (such as an Activity) draws the menu interface, it will specify information such as the display position and display size of the menu interface on the display.

[0112] Specifically, in the case of detecting a menu interface, the display position and display size of the menu interface in the display can be determined by the interactive component, and then the menu position coordinates are determined according to the display position and display size, and the area formed by the menu position coordinates is used as the menu area. When the menu interface is a first-class menu interface, video data corresponding to the menu area can be extracted from the encoded video stream.

[0113] For example, assuming that the size of the entire display is 3840x2160, the display position of the upper left corner of the menu interface called by the interactive component (such as Activity) is determined to be the coordinates (5, 5), and the display size is 100×100 (that is, the width and height are both 100), then the menu position coordinates corresponding to the menu interface are (5, 5), (5, 105), (105, 5), (105, 105). The area formed by the above coordinates is the menu area. The video data corresponding to the menu area is extracted from the encoded video stream (such as video data in YUV format) to obtain the above-mentioned first target video data.

[0114] It should be noted that the method for determining the first target video data of the second type menu interface is similar to that of the first type menu interface. The only difference is that the determined first target video data is the video data after removing the video data corresponding to the menu area in the encoded video stream, which will not be repeated here.

[0115] In the embodiment of the present application, since the menu interface and the video interface are displayed through a display of the same size, the menu area is determined by the menu position coordinates of the menu interface, and the first target video data in the video interface can be accurately determined through the above menu area.

[0116] In some embodiments, when the controller determines the menu area according to the menu position coordinates of the menu interface, the process includes:

[0117] Get the UI elements contained in the above menu interface;

[0118] Determine the menu position coordinates of the menu interface according to the coordinate information of the UI element;

[0119] The position area of ​​the menu interface on the video interface is determined according to the menu position coordinates to obtain the menu area.

[0120] It should be understood that the menu items of the menu interface are composed of different UI elements, and the above UI elements include one or more of the following: text, icons, buttons, navigation bars, page layouts, etc.

[0121] Specifically, the display position of the UI elements corresponding to different menu items in the menu interface and the display size of the UI elements can be determined through the above-mentioned interactive components (such as Activity), and then the display area of ​​each UI element is determined according to the display position of the UI element and the display size of the UI element, and the display areas of all UI elements are combined, and the coordinates corresponding to the combined display areas of the above-mentioned UI elements are determined as the above-mentioned menu position coordinates, and then the area composed of the menu position coordinates is determined as the above-mentioned menu area.

[0122] In the embodiment of the present application, since the menu interface is composed of UI elements, by combining the display areas of all UI elements, it is possible to avoid omission of the menu area and accurately obtain the above-mentioned menu area.

[0123] Reference Figure 8 As shown, in order to determine the interaction diagram of the first target video, the controller first executes step S801 through the player to obtain the video data to be displayed from the server, and then executes step S802 to pass the video data to be displayed to the codec (such as the MediaCodec codec) for encoding; the codec then executes step S803 to return the encoded video stream, such as video data in YUV format; the player executes step S804 to detect the menu interface, and when the menu interface is detected, executes step S805 to request the menu position coordinates from the interaction component, and then the interaction component executes step S806 to request the menu position coordinates; the player executes step S807 to determine the first target video data from the encoded video stream (such as video data in YUV format) according to the menu position coordinates.

[0124] In some embodiments, assuming that the identified target object is a first type of target object, the controller performs target object detection on the encoded video stream according to a pre-trained target object detection model, and when the target object in the encoded video stream is detected, determines the second target video data, including:

[0125] Extracting video features of the encoded video stream using the pre-trained object detection model;

[0126] Matching the video features with a preset feature library to obtain a feature matching result; wherein the preset feature library includes features of the target object that needs to be blurred, and the feature matching result is used to indicate whether the video data to be displayed contains the target object;

[0127] In the case where the feature matching result indicates that the to-be-displayed video data contains the target object, determining the successfully matched video feature as the target object;

[0128] Acquire the object position coordinates of the target object, and determine the object area according to the object position coordinates;

[0129] The video data corresponding to the object area is extracted from the encoded video stream to obtain the second target video data.

[0130] It should be understood that the targets that need to be blurred in the above-mentioned preset feature library may include: license plates, ID cards, vehicles, etc. Among them, license plates and ID cards can be the first type of target objects, that is, objects that need to be blurred for privacy protection, and vehicles can be the second type of target objects, that is, objects that need to be blurred for background and highlighted.

[0131] Specifically, the above-mentioned encoded video stream is used as the input of the above-mentioned pre-trained target object detection model. The above-mentioned pre-trained target object detection model will extract the video features contained in the encoded video stream frame by frame, and then match the video features of each frame with the features of the target object in the preset feature library to obtain a feature matching result. When the feature matching result indicates that the above-mentioned video data to be displayed contains the above-mentioned target object, it means that the match is successful, and the above-mentioned video features that have been successfully matched are determined as the above-mentioned target object, and the object position coordinates of the target object are output. Finally, the video data corresponding to the object area composed of the object position coordinates is extracted from the encoded video stream to obtain the above-mentioned second target video data.

[0132] It should be noted that the method for determining the second target video data of the second type of target object (such as a vehicle, etc.) is similar to that of the first type of target object. The only difference is that the determined second target video data is the video data after removing the video data corresponding to the object area in the encoded video stream, which will not be repeated here.

[0133] In an embodiment of the present application, by directly detecting the target object in the encoded video stream through a pre-trained target detection model, the video data that needs to be blurred can be accurately and quickly determined, thereby improving the accuracy of determining the second target video data.

[0134] Reference Fig. 9As shown, in order to determine the interactive schematic diagram of the second target video, the controller first executes step S901 through the player to obtain the video data to be displayed from the server, and then the player executes step S902 to pass the video data to be displayed into the codec (such as MediaCodec codec) for encoding, and the codec executes step S903 to return the encoded video stream (such as video data in YUV format), and the player executes step S904 to use the encoded video stream (such as video data in YUV format) as the input of the pre-trained target object detection model to obtain the object position coordinates of the detected target object, and finally executes step S905 to determine the second target video data from the encoded video stream (such as video data in YUV format) according to the object position coordinates. For example, if the detected target object is a license plate, the coordinate positions of the four corners of the license plate are obtained to form an object position area, and the second target video data corresponding to the object position area is extracted from the encoded video stream, that is, the video data corresponding to the license plate.

[0135] In some embodiments, when the controller performs blur processing on the target video data, the process includes:

[0136] Perform fuzzy calculation on each pixel in the target video data to obtain a new pixel;

[0137] The target video data after blurring is obtained by combining the new pixel points.

[0138] It should be understood that the above blurring can be achieved by a preset blurring algorithm, which includes one of Gaussian blurring algorithm, mean blurring algorithm, median blurring algorithm, etc. Among them, the Gaussian blurring algorithm performs weighted average on each pixel in the target video data and the pixels in the preset size neighborhood to obtain a new pixel, and the blurring effect is achieved by combining the new pixel. The weight in the Gaussian blurring algorithm is assigned according to the Gaussian distribution function. The closer the pixel is to the center pixel, the greater the weight, and vice versa. The mean blurring algorithm performs mean calculation on each pixel in the image and the pixels in the preset size neighborhood to obtain a new pixel, and the blurring effect is achieved by combining the new pixel. The weights of each pixel in the mean blurring algorithm are the same. The median blurring algorithm sorts each pixel and the pixels in the preset size neighborhood according to the gray value, and then takes the middle value as the new pixel, and the blurring effect is achieved by combining the new pixel.

[0139] Specifically, after obtaining the above-mentioned target video data (including the first target video data and / or the second target video data), the controller can pass the target video data (including the first target video data and / or the second target video data) into the fuzzy processing module, and then the fuzzy processing module will call the preset fuzzy processing algorithm to perform fuzzy calculation on the pixel points in the target video data (including the first target video data and / or the second target video data), and combine the new pixel points to obtain the blurred video data. It should also be noted that when the detected menu interface is a first-class menu interface and the identified target object is a second-class target object, the first target video data corresponding to the first-class menu interface and the second target video data corresponding to the second-class target object may have overlapping data. According to the actual situation, it can be selected to only perform fuzzy processing on the first target video data or the second-class target object, thereby reducing the occupation of computing resources and avoiding freezes, delays and other phenomena.

[0140] In some embodiments, when the controller performs merging the target video data after blurring and the video data to be displayed, the process includes:

[0141] The target video data after blurring is merged with the video data to be displayed according to the position area of ​​the menu interface and / or the position area of ​​the target object.

[0142] Specifically, since the position area of ​​the menu interface and / or the position area of ​​the target object are both position areas determined based on the display, the target video data after blurring and the video data to be displayed can be merged based on the position areas, thereby accurately obtaining the video data to be displayed after blurring. Figure 7-2 For example, after the target video data corresponding to the partial video screen 3 is blurred, it is merged with the to-be-displayed video data corresponding to the entire video screen 4, so that a complete video screen with the partial video screen 3 blurred and the remaining video screens clear can be obtained, that is, the merged video data.

[0143] In some embodiments, the video interface is configured with a playback component for managing video display, and when the controller sends the merged video data to the video interface for display, the controller includes:

[0144] The merged video data is sent to the above-mentioned playback component, so that the above-mentioned video interface displays the above-mentioned merged video data based on the above-mentioned playback component.

[0145] It should be understood that the above-mentioned playback component can be a component for managing video display in different systems. For example, in an Android system, the above-mentioned playback component can be a SurfaceView component. Optionally, assuming that the system of the display device is an Android system, after obtaining the merged video data, the controller can send the merged video data to the SurfaceView component, and then display the merged video data on the video interface through the SurfaceView component.

[0146] Reference Fig.10 As shown in FIG. 1 , it is an interactive schematic diagram of fuzzy processing in a controller. First, the player in the controller executes step S1001 to obtain the encoded video stream (e.g., video data in YUV format) returned by the codec (e.g., MediaCodec codec), and then executes step S1002 to detect the menu interface. When the menu interface is detected, the player executes step S1003 to obtain the menu position coordinates from the interactive component. Then, the player executes step S1004 to determine the first target video data from the encoded video stream (e.g., video data in YUV format) according to the menu position coordinates. At the same time, the player executes step S1005 to detect the target object and the corresponding object position coordinates in the encoded video stream (e.g., video data in YUV format) through a pre-trained target object detection model. The player then executes step S1006 to determine the second target video data from the encoded video stream (e.g., video data in YUV format) according to the object position coordinates; the player executes step S1007 to pass the target video data (including the first target video data and / or the second target video data) to the fuzzy processing module; the fuzzy processing module executes step S1008 to fuzzy the target video data, and the fuzzy processing module executes step S1009 to return the fuzzy target video data; the player executes step S1010 to merge the fuzzy video data and the video data to be displayed, and the player then executes step S1011 to pass the merged video data to the playback component, and finally the playback component executes step S1012 to display the merged video data.

[0147] In some other embodiments, the blurring effect may not meet the user's requirements. After the executor sends the merged video data to the playback component, it is further configured to:

[0148] Receiving a fuzzy processing adjustment instruction; the fuzzy processing adjustment instruction is used to adjust the parameters of the fuzzy processing;

[0149] Performing fuzzy processing on the target video data according to the fuzzy processing adjustment instruction to obtain new fuzzy processed target video data;

[0150] The target video data after the new blur processing is merged with the video data to be displayed, and the merged video data is sent to the playback component for display.

[0151] It should be understood that the above-mentioned fuzzy processing adjustment instruction may include new fuzzy processing parameters.

[0152] Specifically, when receiving a blur processing adjustment instruction, the blur processing module can update the parameters in the preset blur processing algorithm according to the above blur processing adjustment instruction, and then use the updated blur processing algorithm to blur the target video data, and merge the new blurred target video data and the video data to be displayed, and send the merged video data to the playback component for display.

[0153] For example, assuming that the preset blur processing algorithm is a Gaussian blur processing algorithm, the Gaussian convolution kernel, blur radius and other parameters can be adjusted through the blur processing adjustment instruction to achieve different blur effects.

[0154] In the embodiment of the present application, the display effect of the fuzzy processing can be adjusted according to needs through the fuzzy processing adjustment instruction, thereby improving the flexibility of the fuzzy processing.

[0155] Corresponding to the display method described in the above embodiment, Fig.11 A structural block diagram of a display device provided in an embodiment of the present application is shown. For ease of explanation, only the parts related to the embodiment of the present application are shown.

[0156] Reference Fig.11 The device includes: an acquisition module 1110, a video data determination module 1120 and a fuzzy processing module 1130.

[0157] in,

[0158] The acquisition module 1110 is used to acquire video data to be displayed on the video interface.

[0159] The video data determination module 1120 is used to determine the target video data in the video data to be displayed; wherein the target video data is determined based on the detected menu interface and / or based on the target object in the video data to be displayed, and the target object refers to the object that needs to be blurred in the video data to be displayed.

[0160] The fuzzy processing module 1130 is used to perform fuzzy processing on the target video data, merge the fuzzy-processed target video data with the video data to be displayed, and send the merged video data to the video interface for display.

[0161] It should be noted that the information interaction, execution process, etc. between the above-mentioned devices / units are based on the same concept as the method embodiment of the present application. Their specific functions and technical effects can be found in the method embodiment part and will not be repeated here.

[0162] The embodiment of the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a controller, the steps in the above-mentioned method embodiments can be implemented.

[0163] An embodiment of the present application provides a computer program product. When the computer program product is run on a display device, a controller can implement the steps of the display method described in the above embodiments when the controller is executed.

[0164] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the processes in the above-mentioned embodiment method, which can be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may at least include: any entity or device that can carry the computer program code to the camera / electronic device, recording medium, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electric carrier signal, telecommunication signal and software distribution medium. For example, a USB flash drive, a mobile hard disk, a magnetic disk or an optical disk. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electric carrier signals and telecommunication signals.

[0165] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0166] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0167] In the embodiments provided in the present application, it should be understood that the disclosed devices / network equipment and methods can be implemented in other ways. For example, the device / network equipment embodiments described above are merely schematic. For example, the division of the modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0168] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0169] The embodiments described above are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, a person skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.

Claims

1. A display device, characterized in that: include: A display configured to display a menu interface and a video interface; The controller is configured as: Obtaining video data to be displayed on the video interface; Determining target video data in the video data to be displayed; wherein the target video data is determined according to the detected menu interface and / or according to a target object in the video data to be displayed, and the target object refers to an object in the video data to be displayed that needs to be blurred; The target video data is fuzzy processed, the fuzzy target video data and the to-be-displayed video data are merged, and the merged video data is sent to the video interface for display.

2. The display device according to claim 1, characterized in that The target video data includes first target video data and second target video data. When the controller determines the target video data in the video data to be displayed, the controller includes: Encoding the video data to be displayed to obtain an encoded video stream; In the case where the menu interface is detected, first target video data is determined; the first target video data is video data determined according to the position area of ​​the menu interface in the encoded video stream; and / or, The encoded video stream is subjected to target object detection according to a pre-trained target object detection model, and when the target object in the encoded video stream is detected, second target video data is determined; the second target video data is video data determined according to the position area of ​​the target object in the encoded video stream.

3. The display device according to claim 2, characterized in that When the controller detects the menu interface, determining the first target video data includes: When the menu interface is detected, determining a menu area according to the menu position coordinates of the menu interface; The video data corresponding to the menu area is extracted from the encoded video stream to obtain the first target video data.

4. The display device according to claim 3, characterized in that When the controller determines the menu area according to the menu position coordinates of the menu interface, it includes: Obtain the UI elements contained in the menu interface; Determining the menu position coordinates of the menu interface according to the coordinate information of the UI element; The position area of ​​the menu interface on the video interface is determined according to the menu position coordinates to obtain the menu area.

5. The display device according to claim 2, characterized in that The controller performs target object detection on the encoded video stream according to the pre-trained target object detection model, and when the target object in the encoded video stream is detected, determines the second target video data, including: Extracting video features of the encoded video stream using the pre-trained object detection model; Matching the video features with a preset feature library to obtain a feature matching result; wherein the preset feature library includes features of a target object that needs to be blurred, and the feature matching result is used to indicate whether the video data to be displayed contains the target object; In a case where the feature matching result indicates that the to-be-displayed video data contains the target object, determining the successfully matched video feature as the target object; Acquire the object position coordinates of the target object, and determine the object area according to the object position coordinates; The video data corresponding to the object area is extracted from the encoded video stream to obtain the second target video data.

6. The display device according to any one of claims 2 to 5, characterized in that: When the controller performs fuzzy processing on the target video data, it includes: Performing fuzzy calculation on each pixel in the target video data to obtain a new pixel; The target video data after blurring is obtained by combining the new pixel points.

7. The display device according to any one of claims 2 to 5, characterized in that: When the controller performs merging of the target video data after blurring and the video data to be displayed, the method includes: The target video data after blurring is merged with the video data to be displayed according to the position area of ​​the menu interface and / or the position area of ​​the target object.

8. The display device according to any one of claims 2 to 5, characterized in that: The video interface is configured with a playback component for managing video display, and the controller executes sending the merged video data to the video interface for display, including: The merged video data is sent to the playback component, so that the video interface displays the merged video data based on the playback component.

9. The display device according to claim 8, characterized in that After the executor sends the merged video data to the playback component, the executor is further configured to: receiving a fuzzy processing adjustment instruction; the fuzzy processing adjustment instruction is used to adjust the parameters of the fuzzy processing; Performing fuzzy processing on the target video data according to the fuzzy processing adjustment instruction to obtain new fuzzy processed target video data; The target video data after the new blur processing is merged with the video data to be displayed, and the merged video data is sent to the playback component for display.

10. A display method, characterized in that: The display method comprises: Get the video data to be displayed on the video interface; Determine target video data in the video data to be displayed; wherein the target video data is determined according to a detected menu interface and / or according to a target object in the video data to be displayed, and the target object refers to an object in the video data to be displayed that needs to be blurred; The target video data is fuzzy processed, the fuzzy target video data and the to-be-displayed video data are merged, and the merged video data is sent to the video interface for display.

Images