A display device

Abstract

The application discloses a display device, comprising: a camera configured to capture environment image data; a display configured to display a user interface, wherein the user interface is used to present camera-related application content or on-demand media playback content; a controller in communication with the display, the controller is configured to execute the presentation of the user interface: receive a content recommendation instruction based on a current display screen input by a user, determine whether the application corresponding to the current display screen is a camera-related application, and if so, display a first user interface, wherein the first user interface comprises a first image captured by the camera.

Description

[0001] This application is a divisional application of the application with the application date of August 19, 2019, the application number of 201910766057.5, and the invention name of a display device. TECHNICAL FIELD

[0002] The present application relates to display technology, and in particular to a display method and a display device with a camera. BACKGROUND

[0003] At present, display devices can provide users with playing pictures such as audio, video, and pictures, and are widely concerned by users. With the development of big data and artificial intelligence, users' functional requirements for display devices are increasing day by day. For example, if the user's age is determined through face recognition, health popular science programs and resources with a sense of the times such as music and videos can be pushed; or if the user's mood is determined through face recognition, comedy videos and music with a lively rhythm can be pushed. However, the current display devices, especially home devices such as smart TVs, do not have built-in cameras, so they cannot realize the above scenarios.

[0004] Therefore, there is an urgent need for a display device that can present traditional TV pictures while also pushing content recommendation information associated with face feature information, to provide intelligent services for users and improve user experience. SUMMARY

[0005] The embodiments of the present application provide a display method and a display device with a camera to display content recommendation information associated with a face.

[0006] In a first aspect, a display method is provided, applied to a display device, the display device being configured with at least one camera. The method comprises: in response to an information pushing instruction, displaying a first user interface, the first user interface comprising an image display area and an information display area, the image display area displaying a first image acquired by the at least one camera, and the information display area displaying content recommendation information associated with a first face recognized in the first image.

[0007] Optionally, the displaying the first user interface in response to the information push instruction comprises: in response to the information push instruction, performing: acquiring a first image in an image sequence captured by the at least one camera in real time; calling a face recognition module to perform face recognition on the first image, and sending feature information of at least a first face recognized to a server; marking the first face in the first image; receiving content recommendation information associated with at least the first face sent by the server; displaying the first image in which the first face is marked in the image display area, and displaying the content recommendation information associated with the first face in the information display area; wherein the content recommendation information associated with the first face is selected from an information library by the server according to the feature information of the first face.

[0008] Optionally, the information display area comprises a face thumbnail display sub-area and a content recommendation information display sub-area; the face thumbnail display sub-area displays a thumbnail sequence of faces recognized in the image displayed in the image display area, and the thumbnail sequence at least comprises a thumbnail of the first face, and the thumbnail of the first face falls into a focus frame; the content recommendation information display sub-area displays the content recommendation information associated with the first face falling into the focus frame.

[0009] Optionally, the faces recognized in the image displayed in the image display area further comprise at least a second face; at least the first face in the image is marked, and the thumbnail sequence further comprises a thumbnail of the second face.

[0010] Optionally, the method further comprises: in response to a direction key instruction, the thumbnail of the second face in the thumbnail sequence in the face thumbnail display sub-area falls into the focus frame, the information displayed in the content recommendation information display sub-area is updated to the content recommendation information associated with the second face, and the second face in the first image displayed in the image display area is marked.

[0011] Optionally, the response to the direction key instruction, the thumbnail of the second face in the thumbnail sequence in the face thumbnail display sub-area falls into the focus frame, the information displayed in the content recommendation information display sub-area is updated to the content recommendation information associated with the second face, and the second face in the first image displayed in the image display area is marked, comprises: in response to the direction key instruction, performing: falling the thumbnail of the second face in the thumbnail sequence in the face thumbnail display sub-area into the focus frame; acquiring the content recommendation information associated with the second face, and updating the information displayed in the content recommendation information display sub-area to the information associated with the second face; marking the second face in the first image displayed in the image display area.

[0012] Optionally, the obtaining the content recommendation information associated with the second face includes: obtaining the content recommendation information associated with the second face stored locally, wherein the content recommendation information associated with the second face is obtained from a server after the second face is identified and saved locally; or sending the feature information of the second face to the server, and receiving the content recommendation information associated with the second face returned by the server based on the feature information of the second face.

[0013] Optionally, in response to the direction key instruction, the first image in the image display area is further moved and / or zoomed so that the region where the identified second face is located is in a visible region in the image display area.

[0014] Optionally, in the first user interface, the image display area and the information display area are located in different layers, and the layer where the image display area is located is partially covered by the layer where the information display area is located.

[0015] Optionally, before the first user interface is displayed in response to the information push instruction, the method further includes: displaying a second user interface in response to a trigger instruction related to the at least one camera, the second user interface being used to display a sequence of images captured by the at least one camera in real time.

[0016] In a second aspect, a display device is provided, including: a camera configured to capture environment image data; a display configured to display a user interface, wherein the user interface is used to present camera-related application content or on-demand media playback content; and a controller in communication with the display, the controller being configured to execute the presentation of the user interface: receiving a content recommendation instruction based on a current display screen input by a user, determining whether the application corresponding to the current display screen is a camera-related application, and if so, displaying a first user interface, wherein the first user interface includes a first image captured by the camera.

[0017] Optionally, the controller is further configured to receive a content recommendation instruction based on a current display screen input by a user, determine whether the application corresponding to the current display screen is a camera-related application, and if not, display a second user interface, wherein the second user interface includes a screenshot image of the current display screen.

[0018] Optionally, the camera-related application is an application that can obtain the current first image through a picture storage address.

[0019] Optionally, the controller is configured to, in response to the received user input of the content recommendation instruction based on the current display screen, determine whether the application corresponding to the current display screen is a camera-related application, and if so, display a first user interface, wherein the first user interface comprises the first image captured by the camera, and the face recognition module is called to perform face recognition on the first image, and the feature information of at least a first face recognized is sent to the server; receive the content recommendation information associated with at least the first face sent by the server; and transmit the content recommendation information associated with at least the first face to the information display area of the display for presentation.

[0020] Optionally, the controller is configured to: acquire a first image in an image sequence captured by the at least one camera in real time; call a face recognition module to perform face recognition on the first image, and send the feature information of at least a first face recognized to the server through the communicator; identify the first face in the first image; receive the content recommendation information associated with at least the first face sent by the server through the communicator; display the first image in which the first face is identified in the image display area, and display the content recommendation information associated with the first face in the information display area; and wherein the content recommendation information associated with the first face is selected from an information library by the server according to the feature information of the first face.

[0021] Optionally, the information display area comprises a face thumbnail display sub-area and a content recommendation information display sub-area; the face thumbnail display sub-area displays a thumbnail sequence of the recognized face in the image displayed by the image display area, and the thumbnail sequence at least comprises a thumbnail of the first face, and the thumbnail of the first face falls into a focus box; and the content recommendation information display sub-area displays the content recommendation information associated with the first face falling into the focus box.

[0022] Optionally, the recognized face in the image displayed by the image display area further comprises at least a second face, and at least the first face in the image is identified, and the thumbnail sequence further comprises a thumbnail of the second face.

[0023] Optionally, the controller is further configured to execute the computer instructions stored in the memory to perform: in response to a direction key instruction, instruct the thumbnail of the second face in the thumbnail sequence in the face thumbnail display sub-area to fall into the focus box, update the information in the content recommendation information display sub-area to the content recommendation information associated with the second face, and display the second face in the first image displayed by the image display area to be identified.

[0024] Optionally, the controller is specifically configured to: drop a thumbnail of the second face in the thumbnail sequence in a focus frame in the face thumbnail display sub-region; acquire content recommendation information associated with the second face, and update the information displayed in the content recommendation information display sub-region to information associated with the second face; and identify the second face in the first image displayed in the image display region.

[0025] Optionally, the controller is further configured to execute the computer instructions stored in the memory to perform: acquiring the content recommendation information associated with the second face stored in the memory, wherein the content recommendation information associated with the second face is acquired from a server after the second face is identified and saved locally; or sending feature information of the second face to a server through the communicator, and receiving content recommendation information associated with the second face returned by the server based on the feature information of the second face through the communicator.

[0026] Optionally, the controller is further configured to execute the computer instructions stored in the memory to perform: moving and / or zooming the first image in the image display region so that the region where the identified second face is located is located in a visible region in the image display region.

[0027] Optionally, in the first user interface, the image display region and the information display region are located in different layers, and the layer where the image display region is located is partially covered by the layer where the information display region is located.

[0028] Optionally, the controller is further configured to execute the computer instructions stored in the memory to perform: in response to a trigger instruction related to the at least one camera, causing the display to display a second user interface before displaying the first user interface in response to an information push instruction, the second user interface being used to display a sequence of images captured in real time by the at least one camera.

[0029] In the above embodiment of the present application, the display device comprises: a camera configured to collect environmental image data; a display configured to display a user interface, wherein the user interface is used to present camera-related application content or on-demand media playback content; a controller in communication with the display, the controller being configured to execute the presentation of the user interface: receive a content recommendation instruction based on the current display screen input by the user, determine whether the application corresponding to the current display screen is a camera-related application, and if so, display a first user interface, wherein the first user interface comprises a first image obtained by the camera. In this way, the display device uses the configured camera to capture an image, identifies the facial information of the user in the image, and thus recommends content recommendation information associated with the face to the user according to the facial information of the user, thereby providing intelligent services for the user and improving the user experience. BRIEF DESCRIPTION OF DRAWINGS

[0030] Figure 1 An operation scenario between the display device and the control device in an embodiment of the present application;

[0031] Figure 2a A configuration block diagram of a software system in the display device in an embodiment of the present application;

[0032] Figure 2b A schematic diagram of various application programs executable by the display device in an embodiment of the present application;

[0033] Figure 3 A schematic diagram of a user interface of the display device in an embodiment of the present application;

[0034] Figure 4 A schematic diagram of a user interface of the display device in an embodiment of the present application;

[0035] Figure 5a A schematic diagram of a user interface for displaying a camera shooting area screen in an embodiment of the present application;

[0036] Figure 5b A schematic diagram of another user interface for displaying a camera shooting area screen in an embodiment of the present application;

[0037] Figure 6 A flowchart of a display method provided in an embodiment of the present application;

[0038] Figure 7a A user interface for displaying a push resource provided in an embodiment of the present application;

[0039] Figure 7b A user interface for displaying a push resource provided in an embodiment of the present application;

[0040] Figure 7c A user interface for displaying a push resource is provided for embodiments of the present application;

[0041] Figure 7d A user interface for displaying a push resource is provided for embodiments of the present application;

[0042] Figure 7e A user interface for displaying a push resource is provided for embodiments of the present application;

[0043] Figure 7f A user interface for displaying a push resource is provided for embodiments of the present application;

[0044] Figure 8a A user interface for displaying a push resource is provided for embodiments of the present application;

[0045] Figure 8b A user interface for displaying a push resource is provided for embodiments of the present application;

[0046] Figure 9 A flow diagram of a display method is provided for embodiments of the present application;

[0047] Figure 10 A flow diagram of a display method is provided for embodiments of the present application;

[0048] Figure 11 A flow diagram of a display method is provided for embodiments of the present application;

[0049] Figure 12 A structural diagram of a display device is provided for embodiments of the present application. DETAILED DESCRIPTION

[0050] In order to make the objects, technical solutions and advantages of the exemplary embodiments of the present application clearer, the technical solutions in the exemplary embodiments of the present application will be described below in connection with the accompanying drawings. Obviously, the described exemplary embodiments are only a part of the embodiments of the present application, but not all embodiments of the present application.

[0051] The concepts involved in the present application will be described below in connection with the accompanying drawings. It should be noted that the following descriptions of the concepts are only to make the content of the present application easier to understand, and do not limit the scope of protection of the present application.

[0052] The term "module" used in the embodiments of the present application can refer to any known or later developed hardware, software, firmware, artificial intelligence, fuzzy logic, or a combination of hardware or / and software code capable of performing functions related to the element.

[0053] The term "remote controller" used in the embodiments of the present application refers to a component of an electronic device (such as the display device disclosed in the present application), which can generally wirelessly control the electronic device within a short distance range. The component can generally be connected with the electronic device using at least one of the communication modes such as infrared, radio frequency (RF) signal, Bluetooth, etc., and can also include functional modules such as WiFi, wireless universal serial bus (USB), Bluetooth, motion sensor, etc. For example, a handheld touch remote controller is a touch screen user interface that replaces most of the physical built-in hard keys in a general remote control device.

[0054] The term "gesture" used in the embodiments of the present application refers to a user behavior for expressing an intended idea, action, purpose or result through a change in a hand shape or a hand movement, etc.

[0055] The term "hardware system" used in the embodiments of the present application can refer to an entity component with functions of calculation, control, storage, input and output, which is composed of mechanical, optical, electrical and magnetic devices such as integrated circuit (IC), printed circuit board (PCB), etc. In the embodiments of the present application, the hardware system is generally also referred to as a motherboard or a chip.

[0056] Figure 1 The schematic diagram of an operation scenario between a display device and a control device according to an embodiment is exemplarily shown in FIG. 1. As shown in FIG. 1, a user can operate the display device 200 through the control device (100A, 100B). Figure 1

[0057] The control device 100 can be a remote controller 100A, which can communicate with the display device 200 through infrared protocol communication, Bluetooth protocol communication, ZigBee protocol communication or other short-distance communication modes, to control the display device 200 through wireless or other wired modes. The user can input user instructions through the way of pressing keys on the remote controller, the way of voice input, the way of control panel input, etc., to control the display device 200. For example, the user can input corresponding control instructions through the volume up / down key, the channel control key, the up / down / left / right movement key, the voice input key, the menu key, the power on / off key, etc. on the remote controller, to realize the function of controlling the display device 200.

[0058] ​The control device can also be a smart device, a mobile terminal 100B, a tablet computer, a laptop computer, etc. It can communicate with the display device 200 via a Local Area Network (LAN), Wide Area Network (WAN), Wireless Local Area Network (WLAN), or other networks, and control the display device 200 through an application corresponding to the display device 200. For example, an application running on a smart device can control the display device 200; this application can provide various controls to the user through an intuitive user interface (UI) on the screen associated with the display device 200.

[0059] For example, both the mobile terminal 100B and the display device 200 can install software applications, enabling them to connect and communicate via network communication protocols, thereby achieving one-to-one control operations and data communication. For instance, the mobile terminal 100B and the display device 200 can establish a control command protocol, synchronizing the remote control keyboard to the mobile terminal 100B. By controlling the user interface on the mobile terminal 100B, the display device 200 can be controlled. Alternatively, audio and video content displayed on the mobile terminal 100B can be transmitted to the display device 200 for synchronized display.

[0060] like Figure 1 As shown, the display device 200 can also communicate with the server 300 via various communication methods. In various embodiments of this application, the display device 200 may be allowed to communicate with the server 300 via a local area network, a wireless local area network, or other networks. The server 300 may provide the display device 200 with various content and interactive features.

[0061] For example, display device 200 interacts by sending and receiving information and electronic program guides (EPGs), receiving software updates, or accessing remotely stored digital media libraries. Server 300 can be a group or multiple groups, and can be one or more types of servers. Other network services such as video-on-demand and advertising services are provided through server 300.

[0062] Display device 200 can be a liquid crystal display (LCD), an organic light emitting diode (OLED) display, a projection display device, or a smart TV. The specific type, size, and resolution of the display device are not limited. Those skilled in the art will understand that display device 200 can be modified in terms of performance and configuration as needed.

[0063] The display device 200 can provide not only a broadcast receiving television function but also an intelligent network television function that additionally provides a computer support function. Examples include a network television, a smart television, an Internet Protocol Television (IPTV), and the like.

[0064] As Figure 1 As described above, the display device 200 can be connected or provided with a camera for displaying a picture captured by the camera on a display interface of the display device or another display device to implement interactive chatting between users, face recognition, and the like. Specifically, the picture captured by the camera can be displayed in full screen, half screen, or any optional area on the display device.

[0065] As an optional connection method, the camera is connected to the display rear shell through a connection plate and is fixedly installed at an upper middle portion of the display rear shell. As an installable method, the camera can be fixedly installed at any position of the display rear shell as long as an image capturing area of the camera is not blocked by the rear shell, for example, the image capturing area is in the same direction as a display direction of the display device.

[0066] As another optional connection method, the camera is connected to the display rear shell through a connection plate or another conceivable connector and is liftable. A lifting motor is installed on the connector. When the user wants to use the camera or an application program wants to use the camera, the camera is lifted above the display. When the camera is not needed, the camera can be embedded behind the rear shell to protect the camera from damage.

[0067] As an embodiment, the camera used in the present application can be 16 million pixels to achieve the purpose of ultra-high definition display. In actual use, a camera with a higher or lower number of pixels than 16 million pixels can also be used.

[0068] For example, the user can open the camera locally. In response, the display device displays a sequence of images captured in real time. Visually, this function or application can be referred to as a "mirror". The user can obtain an image in the sequence of images captured in real time by the camera through a photographing operation. The user can obtain a sequence of images captured in real time by the camera within a set time period through a recording operation. Further, the mirror application can also provide an information push function for the user, that is, the camera obtains facial information of the user, and pushes video, pictures, music, books, and other resources to the user based on the facial information.

[0069] Figure 2a FIG. 1 shows a configuration block diagram of a software system in the display device 200 according to an exemplary embodiment.

[0070] As Figure 2aAs shown in FIG. 29, the operating system 2911, including execution operation software for processing various basic system services and for implementing hardware-related tasks, serves as an intermediary for data processing between application programs and hardware components.

[0071] In some embodiments, part of the operating system kernel can contain a series of software for managing display device hardware resources and providing services for other programs or software code.

[0072] In other embodiments, part of the operating system kernel can contain one or more device drivers, which can be a set of software code in the operating system that helps to operate or control the devices or hardware associated with the display device. The driver can contain code for operating video, audio and / or other multimedia components. Examples include display, camera, Flash, WiFi and audio drivers.

[0073] Among them, the accessibility module 2911-1 is used to modify or access the application program to achieve the accessibility of the application program and the operability of its display content.

[0074] The communication module 2911-2 is used to connect with other peripherals via relevant communication interfaces and communication networks.

[0075] The user interface module 2911-3 is used to provide objects for displaying user interfaces for various application programs to access, which can achieve user operability.

[0076] The control application 2911-4 is used to control process management, including runtime applications, etc.

[0077] The event transmission system 2914 can be implemented in the operating system 2911 or in the application program 2912. In some embodiments, it is implemented in the operating system 2911 and in the application program 2912, which is used to listen to various user input events, and according to the definition of various events, the recognition results of various events or sub-events are responded to, and a set or multiple sets of predefined operation processing programs are implemented.

[0078] Among them, the event listening module 2914-1 is used to listen to user input interface input events or sub-events.

[0079] The event recognition module 2914-1 is used to define various user input interface input events, recognize various events or sub-events, and transmit them to the processing for executing a set or multiple sets of processing programs corresponding to them.

[0080] Among them, the event or sub-event refers to the input detected by one or more sensors in the display device 200 and the input of the external control device (such as the control device 100, etc.). For example, the voice input sub-event, the gesture recognition gesture input sub-event, and the sub-event input by the remote control key instruction of the control device, etc. For example, one or more sub-events in the remote controller include various forms, including but not limited to one or a combination of key presses up / down / left / right, confirm key, key press, etc. And non-physical key operations such as moving, pressing, releasing, etc.

[0081] The interface layout management module 2913 directly or indirectly receives the various user input events or sub-events monitored by the event transmission system 2914, and is used to update the layout of the user interface, including but not limited to the position of each control or sub-control in the interface, and the size or position of the container, hierarchy, and various execution operations related to the interface layout.

[0082] As shown in Figure 2b The application layer of the display device includes various applications that can be executed on the display device 200.

[0083] The application layer 2912 can include but is not limited to one or more applications, such as a video on demand application, an application center, a game application, a live TV application, a media center application, etc. It should be noted that what applications are included in the application layer 2912 is determined according to the operating system and other designs, and the present application does not need to make specific limitations and divisions on the applications included in the application layer 2912.

[0084] The live TV application can provide live TV through different signal sources. For example, the live TV application can use input from cable TV, radio, satellite service, or other types of live TV services to provide TV signals. In addition, the live TV application can display video of the live TV signal on the display device 200.

[0085] The video on demand application can provide video from different storage sources. Unlike the live TV application, the video on demand provides video display from some storage source. For example, the video on demand can come from a server-side cloud storage or from a local hard disk storage containing stored video programs.

[0086] The media center application can provide various multimedia content playing applications. For example, the media center can provide services for users to access various image or audio services through the media center application, which is different from live TV or video on demand.

[0087] The application center can provide storage for various applications. An application can be a game, an application, or some other application that is related to the computer system or other device but can be run in the display device. The application center can obtain these applications from different sources, store them in the local storage, and then make them available for running on the display device 200.

[0088] Optionally, to facilitate user use, reduce the load of the CPU and improve data processing capability, the display device 200 can use a dual-hardware system architecture. One of the dual-hardware system architectures is referred to as a first hardware system, and the other is referred to as a second hardware system. The first hardware system and the second hardware system can each have an independent operating system installed, so that there are two independent but associated subsystems in the display device 200. In this way, a portion of the functions can be implemented by the first hardware system, and another portion of the functions can be implemented by the second hardware system. The two hardware systems are responsible for processing different data, and can also communicate with each other to implement a certain function. For example, the first hardware system is used to implement the traditional television function, and the second hardware system is used to implement the mirror function described above. When an instruction related to the mirror function is received, the first hardware system decodes and processes the instruction, and if the instruction is related to the mirror function that the second hardware system is responsible for, the processed instruction is sent to the second hardware system so that the second hardware system responds to the instruction.

[0089] It should be noted that the embodiments of the present application can be implemented based on a single hardware system or a dual hardware system, and the present application does not limit this.

[0090] Figure 3 An exemplary schematic diagram of a user interface in the display device 200 according to an exemplary embodiment is shown in FIG. 2. As shown in FIG. 2, the user interface includes a plurality of view display areas, such as a first view display area 201 and a second view display area 202, where the playback screen includes one or more different items arranged in a layout. The user interface also includes a selector that indicates a selected item, and the position of the selector can be moved by user input to change the selection of different items. Figure 3

[0091] It should be noted that the plurality of view display areas can present display screens of different levels. For example, the first view display area can present video chat item content, and can present application layer item content (such as web video, VOD display, application screen, etc.).

[0092] ​Optionally, there is a priority difference between the different view display areas, and the display priority of the view display areas is different between the view display areas with different priorities. For example, the priority of the system layer is higher than the priority of the application layer, and when the user uses the acquisition selector and the screen switching in the application layer, the screen display of the view display area of the system layer is not blocked; and when the size and position of the view display area of the application layer are changed according to the user's selection, the size and position of the view display area of the system layer are not affected.

[0093] The display screen of the same layer can also be presented, and at this time, the selector can switch between the first view display area and the second view display area, and when the size and position of the first view display area change, the size and position of the second view display area can also change.

[0094] The display device generally has a shortcut content pushing option on the remote controller, or the content recommendation can be performed through a voice instruction. For example, when a video is playing on the display screen, the display screen displays a user interface including the currently displayed content and a content recommendation area through triggering by clicking the remote controller, touching the screen, or gestures. The content recommendation area includes a plurality of option bar sets, and the plurality of option bar sets include a display current screen screenshot picture or a picture obtained by a camera, a recommended content related to the image recognition content of the picture, and a user control instruction display bar of a user control instruction input interface associated with the function operation of the picture.

[0095] In some implementations, a function key corresponding to the device can be provided in the control device.

[0096] Taking the remote controller as an example, the shortcut key related to content recommendation is used to acquire the screen on the current display interface, to display on the display and send to the content recommendation server, and to receive the recommended content sent by the content recommendation server and display in the content recommendation area of the display screen.

[0097] When performing content recommendation, the display device needs to send a picture to the content recommendation server, and the content recommendation server performs person recognition or commodity recognition according to the picture, and performs corresponding content recommendation according to the recognized person or commodity.

[0098] In some examples, when content recognition is performed in the home page or video playing of the display device, the display screen of the current display device is generally intercepted as a picture sent to the content recommendation server. The screenshot picture and the recommended content sent by the content recommendation server are displayed on the display as a second user interface. Alternatively, the second user interface includes the recommended content and can also include the screenshot picture or a thumbnail of the screenshot picture as a second image.

[0099] Specifically, when the display plays content, the user triggers the content recommendation button on the remote control or triggers the content recommendation related voice command, performs a screenshot operation, captures the current screen, and obtains a screenshot file. Alternatively, the display device can transmit the original screenshot file to the content recommendation server, or the display device can compress the screenshot file (e.g., WebP format compression) to obtain a compressed file, and then send the compressed file to the content recommendation server.

[0100] After the content recommendation server receives the screenshot file or the screenshot compressed file sent by the display device, the content recommendation server performs character or object recognition and feeds back the recognition result to the display device.

[0101] The display device receives the recommended content sent by the content recommendation server, displays the recommended content in the content recommendation area, and can receive the user's selection in the content recommendation area to enter the recommended content for the user to experience.

[0102] In other examples, the display device runs under a mirror application, and when the user wants to perform content recognition based on the content obtained by the camera, the camera can directly obtain the picture as the picture to be sent to the content recommendation server in the camera use state, and the resource consumption of the screenshot operation is overcome.

[0103] Therefore, under the mirror application, the display device receives the content recommendation button input by the user on the remote control or the content recommendation related voice command, obtains the image file currently displayed by the camera, and displays the image obtained by the camera and the recommended content sent by the content recommendation server on the display as a second user interface. The second user interface includes the recommended content and can also include the picture obtained by the camera or a thumbnail of the picture obtained by the camera as a first image.

[0104] Specifically, when the display displays the preview picture obtained by the camera under the mirror application, the user triggers the content recommendation button on the remote control or triggers the content recommendation related voice command, performs a photographing operation, and obtains the image file currently displayed by the camera. Alternatively, the display device can transmit the original image file to the content recommendation server, or the display device can compress the image file (e.g., WebP format compression) to obtain a compressed file, and then send the compressed file to the content recommendation server.

[0105] After the content recommendation server receives the image file or the image compressed file sent by the display device, the content recommendation server performs character or object recognition and feeds back the recognition result to the display device.

[0106] The display device receives the recommended content sent by the content recommendation server, displays in the content recommendation area, and can receive the selection of the user in the content recommendation area to enter the recommended content for the user to experience.

[0107] As Figure 4 In the third example, the display device can play video content and display preview pictures and take pictures under the mirror application. The user triggers the content recommendation button on the remote control or triggers the content recommendation related voice instruction, and the display device determines the way to obtain the picture to be sent to the content recommendation server by judging the currently displayed application.

[0108] S1101, receiving the content recommendation instruction based on the current display picture input by the user.

[0109] S1102, judging whether the application corresponding to the current display picture is a camera related application.

[0110] Specifically, the application package name of the currently displayed application is judged through ActivityManager (an application management basic class provided by Android system, which is a general way and technical method), and the file storage location corresponding to the data packet flag bit in the application package name is found.

[0111] S1103, if yes, the picture file displayed by the current application is obtained.

[0112] If there is a picture file obtained by the camera stored in the corresponding storage location, it means that the currently displayed application is a mirror application. The picture file at the current time in the corresponding storage location is obtained as the first image. The first user interface including the first image obtained by the camera is displayed on the display. Optionally, the picture file at the current time in the corresponding storage location can also be compressed, and the compressed picture file is taken as the first image.

[0113] S1104, if no, the screenshot operation is performed to obtain the screenshot picture of the current screen picture.

[0114] If there is a picture file obtained by the camera stored in the corresponding storage location, it means that the currently displayed application is not a mirror application. The display picture of the current display is intercepted as a screenshot image. The second user interface including the screenshot image is displayed on the display. Optionally, the screenshot image can also be compressed, and the compressed screenshot image is taken as the image sent to the content recommendation server.

[0115] S1105, the display device sends the first image or the screenshot image to the content recommendation server.

[0116] S1106, receiving the recommended content sent by the content recommendation server and displaying in the content recommendation area.

[0117] Based on Figure 1 The operation scenario diagram between the display device 200 and the control device 100 is shown. The display device 200 can provide various applications, some of which require the ability to display some information associated with the application on the user interface. For example, the user interface displays the picture of the camera shooting area. When the user operates the control device to select the mirror application, the display device 200 opens the camera and displays the picture in the camera shooting area on the user interface.

[0118] Taking the user interface displaying the picture of the camera shooting area as an example, Figure 5a The content of the user interface is exemplarily shown. The user interface can include a display area 401.

[0119] The display area 401 is used to display the image sequence captured by the camera in real time. Specifically, when the user opens the mirror application through the control device, the display device opens at least one camera configured by the display device, and displays the picture in the camera shooting area in the display area 401 of the user interface, which can display the content in the camera shooting area in real time.

[0120] Among them, at least one camera configured by the display device refers to that the display device is configured with a plurality of cameras with different shooting areas; for example, the display device is configured with a front camera, which is used to shoot the picture in the area facing the display screen of the display device. The clarity of the picture displayed by the display area 401 is related to the resolution of the display device and the pixels of the camera, which is not limited in the present application.

[0121] The display device is instructed to acquire content recommendation information associated with the face in the current camera shooting area according to the face in the current camera shooting area, and display the picture in the current camera shooting area and the content recommendation information. Accordingly, the corresponding prompt information can be displayed on the user interface to prompt the user to perform the corresponding operation to complete the information pushing function. For example, the prompt information "press the confirmation key to identify your little mood" can be displayed above the display area 401. For another example, the direction key is used to adjust the direction of the camera so that the image sequence captured by the camera in real time displayed by the display area 401 meets the user's requirements, and the back key is used to indicate to return to the system home interface or the last user interface.

[0122] The relationship information associated with the human face refers to resources obtained according to feature information of the human face, and the feature information includes but is not limited to the following information: emotion, age, gender, whether wearing glasses, charm value, and position information of the human face in an image; the emotion includes happy, sad, irritable, calm, etc., the charm value refers to a numerical value calculated according to facial features, skin state, etc., and the higher the charm value, the more beautiful the corresponding human face.

[0123] Figure 5b Another user interface for displaying a camera shooting area picture is exemplarily shown. As shown in the figure, the user interface includes a display area 401 and a function option area 402 located below the display area 401.

[0124] The function option area 402 is used to display some function options related to the display area 401. The function option area 402 can include a setting function option 402a, an information push function option 402b, and a return function option 402c.

[0125] The setting function option 402a is used to provide settings for the content displayed in the display area 401. The user can obtain multiple setting functions by clicking the setting function option 402a through a control device, such as setting the picture resolution of the display area 401, setting the direction of the camera, setting the focal length, setting the size of the display area 401, etc.

[0126] The information push function option 402b is used to display an image in a sequence of images shot by the camera in real time and content recommendation information associated with a human face in the image. In one case, the display device, in response to the information push function option, displays a prompt message in the display area 401 if no human face is identified in the image obtained from the sequence of images shot by the camera in real time, the prompt message is used to inform the user that the content recommendation information associated with the human face cannot be obtained, so that the user makes the display device display at least one human face in the image displayed in the display area 401 in response to the information push function option 402b through the settings provided by the setting function option 402a or other ways.

[0127] The return function option 402c returns to the system home interface or returns to the last user interface.

[0128] Optionally, the function option area 402 can also include a recording function option (not shown in the figure), which is used to record the picture shot by the camera in real time within a set time length.

[0129] Optionally, the function option area 402 can further include a beautifying function option (not shown in the figure), which is used for beautifying the image displayed on the display area 401. The beautifying function option can provide one-key beautifying, or multiple beautifying function options such as filters, slimming, whitening, etc., which are not limited in the present application.

[0130] Optionally, the function option area 402 can further include a photographing function option (not shown in the figure), which is used for obtaining an image from the image sequence captured by the camera in real time, in which case the display device does not need to perform face recognition on the image, i.e., the image can not include a face. In one case, the photographing function option can implement the function implemented by the information push function option, i.e., the function option area 402 only provides the photographing function option without providing the information push function option; in a specific implementation, the user can click the photographing function option through the control device, in response to which the display area 401 displays an image in the image sequence captured by the camera and the content recommendation information associated with a face in the image, in which case the display device needs to perform face recognition on the captured image.

[0131] It should be noted that the function option area can be located below the display area, or on the left side of the display area, etc., which are not limited in the present application.

[0132] It should be noted that the multiple options in the function option area can be arranged and displayed in one dimension, or in two dimensions, which are not limited in the present application.

[0133] It should be noted that the multiple options provided by the function option area are only examples, which are not limited in the present application.

[0134] Based on the above user interface, Figure 6 A flowchart of a display method provided by an embodiment of the present application is shown.

[0135] As shown in the figure, the flowchart includes:

[0136] S501: In response to a trigger instruction related to at least one camera, a camera preview screen is displayed, which is used for displaying an image sequence captured by the at least one camera in real time.

[0137] The user sends a trigger instruction related to at least one camera configured by the display device to the display device through the control device, such as sending an instruction for opening the mirror application. After the display device receives the trigger instruction, it responds by opening the at least one camera and displaying the user interface as shown in Figure 5a or the user interface as shown in Figure 5b .

[0138] S502: in response to the information pushing instruction, display a first user interface, the first user interface including an image display area and an information display area.

[0139] The user clicks the confirmation key on the control device or pushes the information pushing in the second user interface function option area through the control device, in response to which the first user interface for displaying the pushed resource is displayed. The first user interface includes an image display area and an information display area, the image display area displays at least one image captured by the camera, and the information display area displays content recommendation information associated with the first face recognized in the image.

[0140] Referring to Figure 7a A schematic diagram of a first user interface for displaying a pushed resource provided by an embodiment of the present application.

[0141] As shown in the figure, the first user interface includes an image display area 601 and an information display area 602. The information display area 602 includes a content recommendation information display sub-area 602a, and the layer where the image display area 601 is located is partially covered by the layer where the information display area 602 is located.

[0142] The image display area 601 is used to display one image in a sequence of images captured by at least one camera in real time, and the image includes the first face. Optionally, the image display area 601 can be provided with window processing options (not shown in the figure) to realize moving, zooming, rotating, and other processing of the image displayed by the image display area 601; or corresponding function buttons can be provided on the control device, for example, the left direction key is used to move the image horizontally to the left, the right direction key is used to move the image horizontally to the right, the volume up key is used to enlarge the image, the volume down key is used to reduce the image, and so on.

[0143] The information display area 602 includes a content recommendation information display sub-area 602a, which is used to display content recommendation information associated with the first face in the image displayed by the image display area, i.e., to display the pushed resource obtained according to the feature information of the first face. The user clicks the pushed resource through the control device, in response to which the display device displays the display screen corresponding to the pushed resource.

[0144] Referring to Figure 7b Another schematic diagram of a first user interface for displaying a pushed resource provided by an embodiment of the present application.

[0145] As shown in the figure, the first user interface includes an image display area 601 and an information display area 602. The information display area 602 includes a face thumbnail display sub-area 602b and a content recommendation information display sub-area 602a. The layer where the image display area 601 is located is partially covered by the layer where the information display area 602 is located.

[0146] The face thumbnail display sub-area 602b is configured to display a first face in the image display area. The first face in the thumbnail is consistent with the first face identified in the image displayed in the image display area 601.

[0147] Referring to Figure 7c Another schematic diagram of a first user interface for displaying a push resource is provided for the embodiments of the present application.

[0148] As shown in the figure, the first user interface includes an image display area 601 and an information display area 602. The information display area 602 includes a content recommendation information display sub-area 602a. The layer where the image display area 601 is located is partially covered by the layer where the information display area 602 is located.

[0149] The faces identified in the image display area 601 include a first face and a second face. The first face is identified by a dashed box. The push resource displayed in the content recommendation information display sub-area 602a is the content recommendation information associated with the marked face in the image display area 601, i.e., the content recommendation information associated with the first face. In this case, the user can switch the dashed box in the image display area 601 by using a control device, so that the second face falls into the dashed box. The content recommendation information display sub-area 602a will update and display the content recommendation information associated with the identified second face.

[0150] It should be noted that the identification method can be in the form of a dashed box, an arrow, or an enlarged target face. The present application does not limit the identification method.

[0151] Figure 7d Another schematic diagram of a first user interface for displaying a push resource is provided for the embodiments of the present application.

[0152] As shown in the figure, the first user interface includes an image display area 601 and an information display area 602. The information display area 602 includes a face thumbnail display sub-area 602b and a content recommendation information display sub-area 602a. The layer where the image display area 601 is located is partially covered by the layer where the information display area 602 is located. The faces identified in the image display area 601 include a first face and a second face.

[0153] The face thumbnail display sub-region 602b is used to display a thumbnail sequence of the identified faces in the image displayed in the image display region 601, i.e. the thumbnail of the first face and the thumbnail of the second face. The push resource displayed in the content recommendation information display sub-region 602a is the content recommendation information associated with the face in the face thumbnail display sub-region 602b falling into the focus frame (the focus frame is represented by a solid line box in the figure), i.e. the content recommendation information associated with the first face. The user can move the focus frame in the face thumbnail display sub-region 602b by the control device, so that the thumbnail of the second face falls into the focus frame, and the content recommendation information display sub-region 602a will update and display the content recommendation information associated with the second face, and the image display region 601 will identify the second face with a dashed line box.

[0154] In one case, all the identified faces in the image display region 601 can be identified, and another identification method is used to identify the face corresponding to the push resource displayed in the information display region 602. In this way, when the user finds that not all the faces in the image are identified, the user can adjust the clarity of the image by the control device to let the display device identify the faces in the image again. For example, all the identified faces in the image are identified with a dashed line box, and in the image display region 601, the dashed line box can be further identified in color, so that the color of the focus frame in the face thumbnail display sub-region 602b is the same as the color of the dashed line box. For another example, all the identified faces in the image are identified with a dashed line box, and in the image display region 601, the dashed line box can be further identified with a magnified dashed line box, so that the face identified by the magnified dashed line box corresponds to the push resource displayed in the content recommendation information display sub-region 601a (this method can be applied to the case where the image displayed in the image display region has more than one identified face, and the information display region is not provided with a face thumbnail display sub-region).

[0155] Based on the first user interface shown in Figure 7a , Figure 7b , Figure 7c and Figure 7d , in the information display region 602, a face feature information display region can also be configured, which is used to display the feature information of the face corresponding to the push resource in the content recommendation information display sub-region 602a.

[0156] Optionally, a window processing option can be provided in the information display region, or a corresponding function button can be provided in the control device, and the setting method is similar to the setting method of the image display region, which will not be described again.

[0157] It should be noted that in the first user interface, the image display area can be partially covered by the information display area, the image display area can also be displayed side by side with the information display area, and the image display area can also be displayed side by side with the information display area, and the application does not limit the arrangement order of the face thumbnail display sub-area, the content recommendation information display sub-area, and the face feature information display area.

[0158] It should be noted that in the information display area, the arrangement order of the face thumbnail display sub-area, the content recommendation information display sub-area, and the face feature information display area is not limited by the application.

[0159] Optionally, in S502, in response to the information pushing instruction, the following is performed: obtaining a first image in an image sequence captured in real time by at least one camera; calling a face recognition module to perform face recognition on the first image, and sending feature information of at least a first face recognized to a server; identifying the first face in the first image; receiving content recommendation information associated with at least the first face sent by the server; displaying the first image in which the first face is identified in the image display area, and displaying the content recommendation information associated with the first face in the information display area; wherein the content recommendation information associated with the first face is selected from an information library by the server according to the feature information of the first face.

[0160] Optionally, the information display area includes a face thumbnail display sub-area and a content recommendation information display sub-area; the face thumbnail display sub-area displays a thumbnail sequence of faces recognized in the image displayed in the image display area, and the thumbnail sequence at least includes a thumbnail of the first face, and the thumbnail of the first face falls into the focus box; the content recommendation information display sub-area displays content recommendation information associated with the first face falling into the focus box.

[0161] Optionally, the face recognized in the image displayed in the image display area also includes at least a second face, and at least the first face in the image is identified, and the thumbnail sequence also includes a thumbnail of the second face.

[0162] Optionally, in response to the direction key instruction, the thumbnail of the second face in the thumbnail sequence in the face thumbnail display sub-area falls into the focus box, the information in the content recommendation information display sub-area is updated to the content recommendation information associated with the second face, and the second face is identified in the first image displayed in the image display area.

[0163] Optionally, in response to the direction key instruction, the following is performed: the thumbnail of the second face in the thumbnail sequence in the face thumbnail display sub-area falls into the focus box; obtaining content recommendation information associated with the second face, and updating the information displayed in the content recommendation information display sub-area to the information associated with the second face; and identifying the second face in the first image displayed in the image display area.

[0164] Optionally, the content recommendation information associated with the second face is acquired, including: acquiring the locally stored content recommendation information associated with the second face, wherein the content recommendation information associated with the second face is acquired from the server after the second face is recognized and saved locally; or sending the feature information of the second face to the server, and receiving the content recommendation information associated with the second face returned by the server based on the feature information of the second face.

[0165] Optionally, in response to the direction key instruction, the first image in the image display area is also moved and / or zoomed, so that the region where the identified second face is located is in the visible region in the image display area.

[0166] The steps involved in the method are described in detail below with the display device being a smart TV, the operating system being an Android system, and the smart TV being configured with a camera. Figure 6

[0167] Firstly, the terms involved in the operating system are explained.

[0168] ImageView, a control, an image display module, is used to identify the face in the image, and to move, zoom in, zoom out, and rotate the displayed image.

[0169] Activity, an application, is used to display the acquired image through the ImageView on the display screen of the smart TV.

[0170] Layout, a layout display module, is used to display and manage the content displayed by the display window.

[0171] Referring to Figure 9 , a flowchart of a display method provided by the embodiment of the application.

[0172] As shown in the figure, the flowchart includes:

[0173] S701~S702: receiving a trigger instruction related to the camera, and in response, starting the camera.

[0174] Taking the control device as a remote controller, the user interface of the smart TV in the working mode state at least displays an application (mirror application) associated with the resource of the pushed face feature information, and the user clicks the mirror application through the confirmation button of the remote controller, that is, sends a trigger instruction for starting the mirror application to the smart TV, in response, the smart TV starts the camera, and displays the user interface as shown in Figure 5a or Figure 5b .

[0175] ​In one possible implementation, for a smart TV in a standby mode, a trigger instruction for starting the mirror application can be set as a power-on wake-up instruction on the remote controller, so that the smart TV in the standby mode can respond to the trigger instruction; specifically, the smart TV powers on in response to the trigger instruction, and enters a working mode from the standby mode, and when entering the working mode, does not display a system home page but directly starts the camera to display a user interface as shown in Figure 5a or Figure 5b .

[0176] For example, a mirror application button can be set on the remote controller, and when a user clicks the mirror application button, for a smart TV in a working mode, whether the mirror application is displayed on the current user interface of the smart TV or not, the smart TV starts the camera and displays a user interface as shown in Figure 5a or Figure 5b ; for a smart TV in a standby mode, the mainboard of the smart TV powers on in response to the trigger instruction, and enters a working mode from the standby mode, and when entering the working mode (e.g., after a power-on video is played), starts the camera and displays a sequence of images captured by the camera in real time, such as a user interface as shown in Figure 5a or Figure 5b .

[0177] S703: receiving an information push instruction, and in response, performing steps S704-S709.

[0178] In S702, the smart TV displays a user interface as shown in Figure 5a or Figure 5b ; for example, when the smart TV displays a user interface as shown in Figure 5a , a user clicks a function key corresponding to a display area on the remote controller, such as a confirmation button, to send an information push instruction to the smart TV, and in response, the smart TV performs steps S704-S709; for example, when the smart TV displays a user interface as shown in Figure 5b , a user clicks information push in a function option area on the remote controller to send an information push instruction to the smart TV, and in response, the smart TV performs steps S704-S709; of course, when the smart TV displays a user interface as shown in Figure 5b , an information push instruction can also be sent to the smart TV by clicking a function key corresponding to a display area on the remote controller, and the specific response process is similar to that of Figure 5a , which will not be described herein again.

[0179] S704: in response to the information push instruction, obtaining a first image in a sequence of images captured by the camera in real time.

[0180] In S704, the camera shoots the picture in the shooting area in real time, and obtains the first image from the image sequence.

[0181] S705: calling the face recognition module to obtain the feature information of the recognized face.

[0182] The face recognition module is called to perform face recognition on the first image. When the face recognition module recognizes at least one face in the first image, the face recognition module obtains the feature information of the recognized face, which includes but is not limited to the following information: emotion, age, gender, whether wearing glasses, charm value, and the position information of the face in the image; wherein the emotion includes happy, sad, irritable, calm, etc., the charm value is a numerical value calculated according to the facial features, skin state, etc., and the higher the charm value, the more beautiful the corresponding face.

[0183] After S705 is executed, S706 and S707 are simultaneously executed according to the feature information of the recognized face.

[0184] S706: calling the image display module (ImageView) to identify the target face.

[0185] Specifically, after obtaining the feature information of the recognized face, the image display module can be directly called, or a new application (Activity) can be started to call the image display module. After the image display module is called, the recognized face in the image displayed by the display area is identified according to the position information of the face in the image. There are two identification methods: one is to identify all the recognized faces in the image, wherein the identification method of the target face is different from that of other faces, for example, all the recognized faces in the image are identified by a face recognition frame, but the color of the face recognition frame used to identify the target face is different from that used to identify other faces; the other is to identify only the target face.

[0186] When there is more than one recognized face, the target face is the face at the predefined position, for example, the first recognized face from left to right in the display area is the target face, or one face is selected from the recognized faces, which is closest to the center of the display area in the image.

[0187] S707-S708: sending the feature information of the recognized face to the server and receiving the push resource sent by the server.

[0188] In S707, the feature information of the face recognized by the face recognition module is sent to the server. Specifically, when more than one face is recognized in the image, only the feature information of the target face can be sent to the server, or the feature information of all the recognized faces can be sent to the server.

[0189] After receiving the feature information of the face, the server calculates the recommended resource type for the feature information of the face by algorithm, selects the push resource from the information library according to the determined resource type, and sends the selected push resource to the smart TV.

[0190] For example, for recommending resources related to emotions, the proportion of emotions in each feature information is greater than that of other feature information when calculating the recommended resource type; for example, when the emotion is sadness, comedy videos, cheerful music, and funny pictures, etc. can be recommended, and when the emotion is calm, videos, books, games, etc. that require attention can be recommended; in this way, different resources are recommended according to different emotions, which can soothe pessimistic emotions, provide thoughtful intelligent services, and improve user experience.

[0191] For example, for recommending resources related to emotional age, the proportion of age in each feature information is greater than that of other feature information when calculating the recommended resource type; for example, when the age is older, health programs, videos, music, novels, etc. with a sense of age can be recommended, and wisdom games can also be recommended; for young women, beauty shopping programs, popular videos, music, novels, etc. can be recommended; in this way, different resources are recommended according to different ages, which can meet the entertainment needs of different age groups, provide thoughtful intelligent services, and improve user experience.

[0192] For example, for young people wearing glasses, videos, pictures, etc. related to vision care can be recommended; for young women or young men with high charm value, videos related to fashion and reality show programs, etc. can be recommended.

[0193] S709: calling a layout display module (Layout) to display a user interface including an image display area and an information display area.

[0194] When the target face recognized in the first image is marked and the push resource associated with the target face sent by the server is received, the layout display module is called, which is used to display the marked image in the image display area and display the push resource associated with the target face in the content recommendation information display sub-area of the information display area.

[0195] Specifically, if there is only one face recognized in the first image, the layout display module is called to display, for example, Figure 7athe user interface shown; if more than one face is identified in the first image, a layout display module is invoked to display a layout corresponding to Figure 7b a similar user interface.

[0196] Optionally, in a specific implementation, after obtaining the first image in the image sequence captured by the camera in real time (S704), a face recognition module is invoked to perform face recognition on the first image, and it is determined whether a face can be identified from the first image. If yes, S705 is performed; if no, the following steps can be performed: displaying prompt information, receiving an instruction related to the display area, adjusting the picture displayed in the display area according to the instruction, and then performing the content shown in S703-S704.

[0197] The prompt information is used to inform the user that no face is identified in the image. According to the prompt information, the user can move the position or adjust the direction of the camera to make the picture displayed in the display area include at least one face. If the picture displayed in the display area includes at least one face, the pixels of the picture displayed in the display area can be set and adjusted by using the display area setting button on the remote controller or the setting function option in the function option area shown in Figure 5b , so that the face recognition module can identify the face.

[0198] Referring to Figure 10 , a flowchart of a display method provided by an embodiment of the present application is shown.

[0199] As shown in the figure, the flowchart includes the following steps:

[0200] S801-S802: receiving a trigger instruction related to the camera, and in response, starting the camera.

[0201] S803: receiving an information push instruction, and in response, performing the steps shown in S804-S810.

[0202] S804: in response to the information push instruction, obtaining a first image in an image sequence captured by the camera in real time.

[0203] S805: invoking a face recognition module to obtain feature information of an identified face.

[0204] After S805 is performed, S806, S807, and S809 are simultaneously performed according to the feature information of the identified face.

[0205] S806: invoking an image display module (ImageView) to identify the target face.

[0206] S807-S808: sending the feature information of the identified face to a server, and receiving a push resource sent by the server.

[0207] The content specifically executed in S801-S808 is consistent with the content specifically executed in S701-S708, which will not be described here. Figure 9

[0208] S809: Generating a face thumbnail according to the position information of the face in the image.

[0209] Specifically, according to the position information of the face in the image, the recognized face is subjected to cutout processing, and the face image obtained by cutout is compressed into a thumbnail, thereby generating a face thumbnail. If there are multiple faces recognized in the image, a corresponding number of face thumbnails are generated.

[0210] S810: Calling a layout display module (Layout) to display a user interface including an information display area of the image display area.

[0211] When the target face recognized in the first image is marked, the face thumbnail corresponding to the face recognized in the image is generated, and the push resource associated with the target face received from the service is sent, the layout display module is called, which is used to display the marked image in the image display area, display the generated face thumbnail in the face thumbnail display sub-area of the information display area, and display the push resource associated with the target face in the content recommendation information display sub-area of the information display area.

[0212] Specifically, if there is only one face recognized in the first image, the layout display module is called to display a user interface as shown in Figure 7c If there is more than one face recognized in the first image, the layout display module is called to display a similar user interface as shown in Figure 7d

[0213] Optionally, in specific implementation, after obtaining the first image in the image sequence captured by the camera in real time (S804), the face recognition module can be called to perform face recognition on the first image, and it is determined whether a face can be recognized from the first image. If yes, S805 is executed; if no, the following steps can be performed: displaying prompt information, receiving an instruction related to the display area, adjusting the picture displayed in the display area according to the instruction, and then executing the content shown in S803-S804.

[0214] The prompt information is used to inform the user that no face is recognized in the image. According to the prompt information, the user can make the picture displayed in the display area include at least one face by moving the position or adjusting the direction of the camera. If the picture displayed in the display area includes at least one face, the display area setting button on the remote controller or the like can be used to adjust the display area. Figure 5b ​​The setting function option in the function option area shown in the middle adjusts and sets the pixels of the picture displayed by the display area, so that the face recognition module can recognize the face.

[0215] In the above embodiments of the present application, the display device is configured with at least one camera, and in response to the received information push instruction, the display device displays a first user interface, wherein the first user interface includes an image display area and an information display area, the image display area displays a first image obtained by the at least one camera, and the information display area displays content recommendation information associated with a first face recognized in the first image. In this way, the display device realizes the use of the configured camera to take an image, recognizes the facial information of the user in the image, and thus recommends content recommendation information associated with the face to the user according to the facial information of the user, provides intelligent service for the user, and improves user experience.

[0216] In addition, taking the "mirror" application scenario as an example, in the process of enabling the television camera to take a picture or take a preview, the facial feature recognition algorithm is used to obtain the emotional, age, gender and other characteristic attributes of the person currently displayed on the screen. Then the above obtained facial feature attributes are transmitted to the background server, and based on the above characteristic information, related video content is recommended, returned to the terminal device, and displayed by the terminal. At this time, multiple people may be recognized, and the user is allowed to select different people through the remote control key operation to select the selected person's avatar preview (by displaying a focus box in different styles on the face area of different people to highlight the prompt of the currently selected person) and update the display of the current user recognition and recommendation information. When selecting a person, the current selected person's avatar area may be covered by the information display area 602 (such as Figure 7e In the middle, the right second face is blocked by the information display area 602, and the user cannot directly view the specific appearance in the screen. Figure 7f In the middle, the right second face is blocked by the information display area 602, and the user cannot directly view the specific appearance in the screen.

[0217] In view of the above situation, a person's avatar scaling and translation processing is needed to center and enlarge the blocked or small edge person.

[0218] S1501, judge the size of the minimum value Min[x, y] of the head portrait P area of the focus point and the custom threshold DM.

[0219] The specific judgment method is: Min[x, y] / DM<1 (Min[x, y] and DM ratio is less than 1). Wherein, the screen coordinate system is as follows Figure 11The screen coordinate system and the object recognition box position information labeling example are shown. The custom threshold DM is a controllable parameter, which is used to calibrate the small size of the recognized portrait image displayed to the user. For example, in order to enable the user to clearly view the selected face image, the minimum side length of the displayed portrait area is expected to be no less than DM=300 pixels. The minimum length and width of the P area Min[x, y] is the minimum side length in the X axis direction and the Y axis direction of the selected portrait area image, for example, the length along the X axis is 300 pixels and the length along the Y axis is 200 pixels, and Min[x, y]=200 pixels.

[0220] In S1502, if the minimum length and width of the P area Min[x, y] is less than the custom threshold DM, the displayed image is enlarged.

[0221] Specifically, the displayed image is enlarged by DM / Min[x, y] times. The displayed image is the first image.

[0222] If Min[x, y] / DM>=1, it is considered that the user can clearly observe the selected area, and no scaling processing is required.

[0223] In S1503, the distance between the center point of the P focus area (such as the enlarged P focus area) and the screen center point coordinates X axis and Y axis is calculated.

[0224] Specifically, the distance HX and VY between the center point of the P focus area (such as the enlarged P focus area) and the screen center point coordinates X axis and Y axis is calculated.

[0225] In S1504, the image is translated along the X axis and the Y axis by HX and VY respectively, so that the P focus area is located in the screen center area (such as Figure 8a ).

[0226] Optionally, as shown in FIG. 8a, when the position of the moved area in the picture is too close to the frame, so that if the selected area is in the picture, the picture edge may be missed, and the picture edge is still a certain distance away from the content recommendation information display sub-area 602a, no centering processing is performed, but the horizontal coordinate of the picture edge after the screen is enlarged and the screen horizontal coordinate are obtained, and the difference between the two is moved. For example, the right edge coordinate of the picture top right vertex is the same as the screen coordinate (X1, Y1), and after the picture is enlarged, the picture edge coordinate becomes (X1, Y1), and then the left bottom is translated (|X2-X1|, |Y2-Y1|).

[0227] Based on the same concept, the display device provided by the embodiment of the present application is configured with at least one camera, which is used to realize the display method shown in FIGS. 7-10.

[0228] AsFigure 12 Fig. 3 shows a structural schematic diagram of a display device according to an embodiment of the present application.

[0229] As shown in the figure, the display device 900 comprises a controller 901, a memory 902, a display 903 and a communicator 904.

[0230] The memory 902 is configured to store computer instructions.

[0231] The controller 901 is configured to execute the computer instructions stored in the memory, and perform the following: in response to an information push instruction, causing the display to display a first user interface, the first user interface comprising an image display area and an information display area, the image display area displaying an image captured by the at least one camera, and the information display area displaying content recommendation information associated with a first face recognized in the image.

[0232] Optionally, the controller 901 is specifically configured to: acquire a first image in a sequence of images captured by the at least one camera in real time; call a face recognition module to perform face recognition on the first image, and send feature information of at least a first face recognized to a server through the communicator 904; identify the first face in the first image; receive content recommendation information associated with at least the first face sent by the server through the communicator 904; display the first image in which the first face is identified in the image display area, and display the content recommendation information associated with the first face in the information display area; wherein the content recommendation information associated with the first face is selected from an information database by the server according to the feature information of the first face.

[0233] Optionally, the information display area comprises a face thumbnail display sub-area and a content recommendation information display sub-area; the face thumbnail display sub-area displays a sequence of thumbnails of faces recognized in the image displayed in the image display area, the sequence of thumbnails at least comprising a thumbnail of the first face, the thumbnail of the first face falling within a focus box; and the content recommendation information display sub-area displays the content recommendation information associated with the first face falling within the focus box.

[0234] Optionally, the image displayed in the image display area further comprises at least a second face, and at least the first face in the image is identified, and the sequence of thumbnails further comprises a thumbnail of the second face.

[0235] Optionally, the controller 901 is further configured to execute the computer instructions stored in the memory 902 to perform the following operation: in response to the direction key instruction, indicating that the thumbnail of the second face in the thumbnail sequence falls into the focus frame in the face thumbnail display sub-region, updating the information displayed in the content recommendation information display sub-region to the content recommendation information associated with the second face, and marking the second face in the first image displayed in the image display region.

[0236] Optionally, the controller 901 is specifically configured to: drop the thumbnail of the second face in the thumbnail sequence into the focus frame in the face thumbnail display sub-region; obtain the content recommendation information associated with the second face, and update the information displayed in the content recommendation information display sub-region to the information associated with the second face; and mark the second face in the first image displayed in the image display region.

[0237] Optionally, the controller 901 is further configured to execute the computer instructions stored in the memory to perform the following operation: obtaining the content recommendation information associated with the second face stored in the memory 902, wherein the content recommendation information associated with the second face is obtained from a server after the second face is identified and saved locally; or sending the feature information of the second face to the server through the communicator 904, and receiving the content recommendation information associated with the second face returned by the server based on the feature information of the second face through the communicator.

[0238] Optionally, the controller 901 is further configured to execute the computer instructions stored in the memory to perform the following operation: moving and / or zooming the first image in the image display region so that the region where the second face is located is located in the visible region in the image display region.

[0239] Optionally, in the first user interface, the image display region and the information display region are located in different layers, and the layer where the image display region is located is partially covered by the layer where the information display region is located.

[0240] Optionally, the controller 901 is further configured to execute the computer instructions stored in the memory to perform the following operation: before displaying the first user interface in response to the information push instruction, in response to a trigger instruction related to the at least one camera, causing the display to display a second user interface, and the second user interface is used to display the image sequence captured by the at least one camera in real time.

[0241] All other embodiments obtained by persons of ordinary skill in the art based on the example embodiments shown in the present application without creative efforts fall within the scope of the present application. In addition, although the disclosure in the present application is introduced according to one or more examples, it should be understood that each aspect of the disclosure can also constitute a complete technical solution independently.

[0242] It should be understood that the terms "first", "second", "third" and the like in the description and claims of the present application and the above drawings are used to distinguish similar objects, and do not necessarily indicate a specific order or sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, for example, those given in the embodiment illustrations or descriptions of the present application can be implemented in an order other than that given.

[0243] It should be understood that the terms "and / or" in the description and claims of the present application and the above drawings describe the association relationship of the associated objects, and can represent three relationships, for example, A and / or B can represent three cases of A alone, A and B together, and B alone. The character " / " generally represents an "or" relationship between the associated objects before and after it.

[0244] In addition, the terms "include" and "have" and any variations thereof are intended to cover but not exclusive inclusion, for example, a product or device containing a series of components does not have to be limited to those components clearly listed, but can include other components not clearly listed or inherent to the product or device.

[0245] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the above embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions recorded in the above embodiments, or make equivalent replacement for part or all of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solution deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A display device, characterized in that, include: A camera configured to capture environmental image data; A display configured to display a first user interface, wherein the first user interface includes an image display area and an information display area, the information display area floating above the image display area, the image display area displaying a first image associated with environmental image data, the first image including at least a first face, the information display area including a face thumbnail display sub-area and a content recommendation information display sub-area, the face thumbnail display sub-area being used to display a sequence of thumbnails of faces identified in the first image, the face thumbnail display sub-area also including a focus indicating the selection of a face thumbnail, used to identify the thumbnail of the first face, the focus being movable in the information display area by user input to select thumbnails of different faces; the content recommendation information display sub-area being used to display content recommendation information associated with the face falling into the focus frame; A controller that communicates with the display is configured to execute the presentation of the user interface: If the face identified in the first image also includes at least a second face, and the thumbnail sequence also includes a thumbnail of the second face, in response to the directional key instruction that switches the focus from the thumbnail of the first face to the thumbnail of the second face, the content recommendation information display sub-region is updated with content recommendation information associated with the second face; If the position of the second face in the image display area is within the coverage area of ​​the information display area, then determine the minimum length and width of the second face P region Min[x,y] and the size of the custom threshold DM, and calculate the distance between the center point of the P region and the center point of the screen; If Min[x,y] is greater than or equal to DM, then the first image is translated along the X-axis and Y-axis respectively, so that the area where the second face is located is located in the visible area of ​​the image display area; If Min[x,y] is less than DM, then the first image is enlarged and translated along the X and Y axes respectively, so that the area where the second face is located is in the visible area of ​​the image display area.

2. The display device according to claim 1, characterized in that, The controller is also configured to: When the focus of the information display area is on the thumbnail of the first face, the first face in the first image is identified; In response to a directional key command that switches the focus from a thumbnail of the first face to a thumbnail of the second face, the second face in the first image is identified.

3. The display device according to claim 1, characterized in that, The controller is also configured to: In response to the directional key command that switches the focus from the thumbnail of the first face to the thumbnail of the second face, the information display area is updated with content recommendation information associated with the second face, specifically including: In response to the directional key command that switches the focus from the thumbnail of the first face to the thumbnail of the second face, Send the feature information of the second face in the first image to the content recommendation server; Receive content recommendation information associated with the second face sent by the content recommendation server; The content recommendation information associated with the second face is transmitted to the information display area of ​​the display for presentation.

4. The display device according to claim 1, characterized in that, The controller is also configured to: In response to a user's instruction to switch from a first face thumbnail to a second face thumbnail, the thumbnail of the second face in the thumbnail sequence in the face thumbnail display sub-region falls into the focus box, the information in the content recommendation information display sub-region is updated to association information associated with the second face, and the second face is identified in the first image displayed in the image display area.

5. A method for recommending content based on the currently displayed screen, applied to a display device, comprising: Upon receiving a command to activate the camera, a preview screen associated with the environmental image data acquired from the camera is displayed on the monitor; In response to a user's input command to obtain content recommendations based on the current display screen, a first user interface is presented on the display. The first user interface includes an image display area and an information display area, with the information display area floating above the image display area. The image display area displays a first image associated with the environmental image data, which includes a first face and a second face. The information display area includes a face thumbnail display sub-area and a content recommendation information display sub-area. The face thumbnail display sub-area displays thumbnails of the first face and the second face and indicates the focus of the selected thumbnail of the first face. The content recommendation information display sub-area displays content recommendation information associated with the first face. In response to the directional key command that switches the focus from the thumbnail of the first face to the thumbnail of the second face, the content recommendation information display sub-area is updated to content recommendation information associated with the second face; If the position of the second face in the image display area is within the coverage area of ​​the information display area, then determine the minimum length and width of the second face P region Min[x,y] and the size of the custom threshold DM, and calculate the distance between the center point of the P region and the center point of the screen; If Min[x,y] is greater than or equal to DM, then the first image is translated along the X-axis and Y-axis respectively, so that the area where the second face is located is located in the visible area of ​​the image display area; If Min[x,y] is less than DM, then the first image is enlarged and translated along the X and Y axes respectively, so that the area where the second face is located is in the visible area of ​​the image display area.

6. The method according to claim 5, characterized in that, In response to a directional key command that switches the focus from a thumbnail of the first face to a thumbnail of the second face, the second face in the first image is identified.

7. The method according to claim 5, characterized in that, In response to the directional key command that switches the focus from the thumbnail of the first face to the thumbnail of the second face, the information display area is updated with content recommendation information associated with the second face, specifically including: In response to the directional key command that switches the focus from the thumbnail of the first face to the thumbnail of the second face, feature information of the second face in the first image is sent to the content recommendation server; Receive content recommendation information associated with the second face sent by the content recommendation server; The content recommendation information associated with the second face is transmitted to the information display area of ​​the display for presentation.

Images