Display device and image recognition method

Abstract

This application provides a display device and an image recognition method in some embodiments. The method can acquire the display content of a user interface in response to a feature target recognition command. The display content includes video content and / or image content. When the category tags of the display content include feature target identifiers, multiple screenshots are continuously taken of the display content within a preset time period to generate multiple screenshots of the feature targets. These multiple screenshots are stitched together to form a panoramic image of the feature targets, and the panoramic image is then recognized to generate feature target information. This solves the problem of low recall rate in feature target recognition and improves the user experience.

Description

Technical Field

[0001] This application relates to the field of display device technology, and in particular to a display device and an image recognition method. Background Technology

[0002] Display devices refer to electronic devices capable of outputting specific display images, such as smart TVs, communication terminals, smart advertising screens, and projectors. Taking smart TVs as an example, smart TVs are television products based on Internet application technology, possessing open operating systems and chips, and having open application platforms. They enable two-way human-computer interaction and integrate multiple functions such as audio-visual, entertainment, and data to meet diverse and personalized user needs.

[0003] Display devices can also perform intelligent image recognition on their displayed images. For example, while viewing, users can take screenshots of featured targets such as animals, plants, scenic spots, cars, or brands to identify them and obtain detailed information about those targets. However, some featured targets require panoramic recognition to obtain their information. Therefore, if the displayed image only includes partial features of the target when the user sends a recognition command to the display device, the target information cannot be identified, resulting in a low recall rate and a reduced user experience. Summary of the Invention

[0004] This application provides a display device and an image recognition method to solve the problem of low recall rate for feature target recognition in display devices.

[0005] In a first aspect, this application provides a display device, including a display and a controller. The controller is configured to perform the following program steps:

[0006] In response to a feature target recognition instruction, the display content of the user interface is obtained, the display content including video content and / or image content;

[0007] Read the category tags of the displayed content;

[0008] If the category label of the displayed content includes a feature target identifier, then screenshots of the displayed content will be taken continuously within a preset time to generate multiple screenshots.

[0009] The multiple screenshots are stitched together to form a panoramic image;

[0010] Identify feature targets in the panoramic image to generate feature target information, and control the display to show the feature target information.

[0011] Secondly, this application also provides an image recognition method, comprising:

[0012] In response to a feature target recognition command, the user interface display content is obtained, including video content and / or image content;

[0013] Read the category tags of the displayed content;

[0014] If the category label of the displayed content includes a feature target identifier, then screenshots of the displayed content will be taken continuously within a preset time to generate multiple screenshots.

[0015] The multiple screenshots are stitched together to form a panoramic image;

[0016] Identify feature targets in the panoramic image to generate feature target information, and control the display to show the feature target information.

[0017] As can be seen from the above technical solutions, the display device and image recognition method provided in this application can respond to a feature target recognition instruction and obtain the display content of the user interface. The display content includes video content and / or image content. When the feature target identifier is included in the category label of the display content, the method can continuously take multiple screenshots of the display content within a preset time period, generating multiple screenshots of the feature target. These multiple screenshots are then stitched together to form a panoramic image of the feature target, and the panoramic image is recognized to generate feature target information, thereby solving the problem of low recall rate in feature target recognition and improving user experience. Attached Figure Description

[0018] To more clearly illustrate the technical solution of this application, the drawings used in the embodiments will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 This is a schematic diagram illustrating an operational scenario between a display device and a control device provided in some embodiments of this application;

[0020] Figure 2 This is a schematic diagram of the hardware configuration of a display device provided in some embodiments of this application;

[0021] Figure 3 This is a schematic diagram of the hardware configuration of the control device provided in some embodiments of this application;

[0022] Figure 4 This is a schematic diagram of the software configuration of a display device provided in some embodiments of this application;

[0023] Figure 5 A schematic diagram of the icon control interface of a display device application provided in some embodiments of this application;

[0024] Figure 6 A schematic diagram of the recommended interface for category tags provided in some embodiments of this application;

[0025] Figure 7 This is a schematic diagram of the interface for recognizing feature target information provided in some embodiments of this application;

[0026] Figure 8 A schematic diagram of the prompt information interface of the detection screenshot tool provided in some embodiments of this application;

[0027] Figure 9 This application provides schematic diagrams illustrating the effect of feature target recognition when viewing media assets, based on some embodiments of the present application.

[0028] Figure 10 This application provides schematic diagrams illustrating the effect of non-feature target recognition when viewing media assets, based on some embodiments of the present application.

[0029] Figure 11 This application provides a schematic diagram of the effect of a screenshot for some embodiments;

[0030] Figure 12 This is a schematic diagram illustrating the effect of another screenshot provided in some embodiments of this application;

[0031] Figure 13 This application provides a schematic diagram of the effect of a panoramic image for some embodiments;

[0032] Figure 14 This is a schematic diagram illustrating the process of stitching screenshots according to some embodiments of this application;

[0033] Figure 15 A schematic diagram illustrating an unnatural transition in a panoramic image effect provided for some embodiments of this application;

[0034] Figure 16 A schematic diagram illustrating the feature target recognition process provided in some embodiments of this application;

[0035] Figure 17 This is a flowchart illustrating an image recognition method provided in some embodiments of this application. Detailed Implementation

[0036] To make the objectives and implementation methods of this application clearer, the exemplary implementation methods of this application will be clearly and completely described below with reference to the accompanying drawings of the exemplary embodiments of this application. Obviously, the exemplary embodiments described are only some embodiments of this application, and not all embodiments.

[0037] It should be noted that the brief descriptions of terms in this application are only for the convenience of understanding the embodiments described below, and are not intended to limit the embodiments of this application. Unless otherwise stated, these terms should be understood in their ordinary and common meaning.

[0038] The terms "first," "second," "third," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar or related objects or entities, and do not necessarily imply a specific order or sequence, unless otherwise specified. It should be understood that such terms are interchangeable where appropriate.

[0039] The terms “comprising” and “having”, and any variations thereof, are intended to cover but not exclude inclusion, for example, a product or device that includes a range of components is not necessarily limited to all of the components that are clearly listed, but may include other components that are not clearly listed or that are inherent to such product or device.

[0040] The display device provided in this application can have various implementation forms, such as a television, a laser projection device, a monitor, an electronic bulletin board, an electronic table, etc.

[0041] Figure 1 This is a schematic diagram illustrating the operational scenario between the display device and the control unit according to the embodiment. Figure 1 As shown, the user can operate the display device 200 through the terminal device 300 or the control device 100.

[0042] In some embodiments, the control device 100 may be a remote control. Communication between the remote control and the display device includes infrared protocol communication, Bluetooth protocol communication, and other short-range communication methods, controlling the display device 200 wirelessly or via wired means. Users can control the display device 200 by inputting user commands through buttons on the remote control, voice input, control panel input, etc.

[0043] In some embodiments, a terminal device 300 (such as a mobile phone, tablet computer, computer, laptop computer, etc.) may also be used to control the display device 200. For example, an application running on the terminal device 300 may be used to control the display device 200.

[0044] In some embodiments, the display device 200 may receive instructions not through the terminal device 300 or control device 100 described above, but through touch or gestures.

[0045] In some embodiments, the display device 200 can also be controlled in ways other than the control device 100 and the terminal device 300. For example, it can be controlled by directly receiving the user's voice commands through a module configured inside the display device 200 for acquiring voice commands, or it can be controlled by receiving the user's voice commands through a voice terminal device set outside the display device 200.

[0046] In some embodiments, the display device 200 also communicates with the server 400. The display device 200 may communicate via a local area network (LAN), wireless local area network (WLAN), and other networks. The server 400 may provide various content and interactive features to the display device 200. The server 400 may be a cluster or multiple clusters, and may include one or more types of servers.

[0047] like Figure 2 As shown, the display device 200 may include at least one of the following: a tuner 210, a communicator 220, a detector 230, an external device interface 240, a controller 250, a display 260, an audio output interface 270, a memory, a power supply, and a user interface.

[0048] In some embodiments, controller 250 may include a processor, a video processor, an audio processor, a graphics processor, RAM, ROM, and a first to an nth interface for input / output.

[0049] The display 260 may include the following components: a display screen component for displaying images; a driving component for driving image display; a component for receiving image signals output from the controller 250 and displaying video content, image content, and a menu control interface, as well as a user control UI interface, etc.

[0050] The display 260 can be an LCD display, an OLED display, or a projection display, and can also be a projection device and a projection screen.

[0051] The communicator 220 is a component used to communicate with external devices or servers according to various communication protocol types. For example, the communicator may include at least one of the following: a Wi-Fi module, a Bluetooth module, a wired Ethernet module, other network communication protocol chips or near-field communication protocol chips, and an infrared receiver. The display device 200 can establish the transmission and reception of control signals and data signals with the external control device 100 or the server 400 through the communicator 220.

[0052] The user interface can be used to receive control signals from the control device 100 (such as an infrared remote control).

[0053] Detector 230 is used to collect signals from the external environment or to interact with the external environment. For example, detector 230 includes a light receiver, a sensor for collecting ambient light intensity; or, detector 230 includes an image acquisition device, such as a camera, which can be used to collect external environmental scenes, user attributes, or user interaction gestures; or, detector 230 includes a sound acquisition device, such as a microphone, for receiving external sounds.

[0054] The external device interface 240 may include, but is not limited to, one or more of the following: High Definition Multimedia Interface (HDMI), analog or high-definition component input interface (component), composite video input interface (CVBS), USB input interface (USB), RGB port, etc. It may also be a composite input / output interface formed by multiple interfaces mentioned above.

[0055] The tuner / demodulator 210 receives broadcast television signals via wired or wireless means, and demodulates audio and video signals, such as EPG data signals, from multiple wireless or wired broadcast television signals. In some embodiments, the controller 250 and the tuner / demodulator 210 may be located in different separate devices, that is, the tuner / demodulator 210 may also be located in an external device of the main device where the controller 250 is located, such as an external set-top box.

[0056] The controller 250 controls the operation of the display device and responds to user operations through various software control programs stored in the memory. The controller 250 controls the overall operation of the display device 200. For example, in response to receiving a user command to select a UI object to display on the monitor 260, the controller 250 can execute operations related to the object selected by the user command.

[0057] In some embodiments, the controller 250 includes at least one of a central processing unit (CPU), a video processor, an audio processor, a graphics processing unit (GPU), RAM (random access memory), ROM (read-only memory), a first to an nth interface for input / output, a communication bus, etc.

[0058] Figure 3 A hardware configuration block diagram of a control device 100 according to some embodiments is shown. For example... Figure 3As shown, the control device 100 includes a control processor 110, a communication interface 130, a user input / output interface 140, a memory, and a power supply. The control device 100 can receive user input operation commands and convert the operation commands into commands that the display device 200 can recognize and respond to, thus acting as an intermediary for interaction between the user and the display device 200.

[0059] like Figure 4 As shown, the display device system is divided into three layers, from top to bottom: the application layer, the middleware layer, and the hardware layer.

[0060] The application layer mainly includes commonly used applications on TVs, as well as the application framework. The commonly used applications are mainly browser-based applications, such as HTML5 apps, and native apps.

[0061] An application framework is a complete program model that has all the basic functions required by standard application software, such as file access, data exchange, etc., as well as the user interface for these functions (toolbar, status bar, menu, dialog box).

[0062] In this embodiment, the application framework layer includes managers, content providers, etc., wherein the managers include at least one of the following modules: an Activity Manager for interacting with all activities running in the system; a Location Manager for providing system services or applications with access to system location services; a Package Manager for retrieving various information related to application packages currently installed on the device; a Notification Manager for controlling the display and clearing of notification messages; and a Window Manager for managing icons, windows, toolbars, wallpapers, and desktop widgets on the user interface.

[0063] In some embodiments, the Activity Manager manages the lifecycle of each application and common navigation and back functions, such as controlling application exit, opening, and back actions. The Window Manager manages all window programs, such as obtaining the screen size, determining whether a status bar is present, locking the screen, capturing the screen, controlling display window changes (e.g., shrinking the display window, dithering the display, distorting the display), detecting whether cursor mode is enabled, and controlling the cursor to move to the corresponding position after enabling cursor mode. For example, after detecting that cursor mode is enabled, the Window Manager can call the cursor display parameters and control the display of the cursor on the screen according to the cursor display parameters. It can also move the cursor to the target position on the screen according to the received cursor movement command.

[0064] Native apps can support online or offline access, push notifications, or access to local resources.

[0065] The middleware layer includes various television protocols, multimedia protocols, and system components. Middleware can use the basic services (functions) provided by system software to connect different parts of application systems or different applications on the network, achieving resource sharing and function sharing.

[0066] The hardware layer mainly includes the HAL interface, hardware, and drivers. The HAL interface is a unified interface for all TV chips, with the specific logic implemented by each chip. The drivers mainly include: audio driver, display driver, Bluetooth driver, camera driver, Wi-Fi driver, USB driver, HDMI driver, sensor drivers (such as fingerprint sensors, temperature sensors, pressure sensors, etc.), and power drivers.

[0067] Figure 5 A schematic diagram illustrating the application programs that the display device provided in some embodiments of this application can provide, such as... Figure 5 As shown, the application layer contains at least one application that can display corresponding icon controls on the display, such as: live TV application icon control, video-on-demand application icon control, media center application icon control, application center icon control, game application icon control, etc.

[0068] In some embodiments, a live TV application can provide live TV from different signal sources. For example, the live TV application can provide a TV signal using input from cable television, terrestrial broadcasting, satellite services, or other types of live TV services. Furthermore, the live TV application can display the video of the live TV signal on display device 200.

[0069] In some embodiments, a video-on-demand application may provide video from different storage sources. Unlike live TV applications, video-on-demand provides video display from certain storage sources. For example, video-on-demand may come from a cloud storage server or from local hard drive storage containing pre-stored video programs.

[0070] In some embodiments, a media center application may be an application that provides playback of various multimedia content. For example, a media center may provide services that, unlike live TV or video-on-demand, allow users to access various images or audio through the media center application.

[0071] In some embodiments, the application center may provide a storage for various applications. An application may be a game, an application, or other applications related to a computer system or other device but capable of running on a smart TV. The application center may obtain these applications from various sources, store them in local storage, and then run them on the display device 200.

[0072] The display device 200 can obtain target media assets from the corresponding media asset library through the server 400 via any of the methods described in the above embodiments, and play the target media assets online. That is, in some embodiments, during the playback of a media asset item, the server 400 can obtain media asset data from the corresponding media asset library in real time, and continuously form media asset images through decoding, rendering, and other processing.

[0073] Furthermore, in some embodiments, the server 400 can also monitor the playback process of the media asset project in real time. When the playback of the media asset project is interrupted, the server 400 will automatically record the playback progress of the media asset project to generate a history record. The history record allows the user to resume playback of the media asset project directly according to the recorded playback progress when playing it again, thus meeting the user's need to continue watching.

[0074] To facilitate the acquisition of media assets by the server 400, in some embodiments, the server 400 can also set category tags for each media asset based on its specific content. The category tags can include information reflecting the characteristics of the media asset, such as the media asset name, category, author, and length classification. Furthermore, the server 400 can determine user viewing preferences based on the frequency of these category tags in historical records, and then push media assets with the same category tags to the display device 200 according to the determined viewing preferences.

[0075] Due to the large number and variety of media assets, in some embodiments, the server 400 can also classify the media assets into several major categories, such as movies, TV series, documentaries, children's content, and educational content. Furthermore, since users' viewing preferences are not limited to a single format, the server 400 can further divide the major categories into multiple subcategories, and then further divide the subcategories into multiple smaller categories, in order to accurately recommend or filter media assets of each type for users.

[0076] For example, Server 400 can set three levels of category tags for media assets. First-level category tags include: "Movies," "TV Series," "Variety Shows," "Children's," "Documentaries," etc. Second-level category tags are refinements of the first-level category tags, and third-level category tags are refinements of the second-level category tags. For example... Figure 6 As shown, taking the primary category tag "Movies" as an example, the secondary category tags included in the primary category tag "Movies" include "Romance," "Action," "Thriller," "Science Fiction," and "Attractions," etc.; the secondary category tag "Attractions" includes the tertiary category tags including "Asia," "Africa," "Europe," "South America," "North America," "Oceania," and "Antarctica." Figure 6 The category tags in Media Assets A include at least “Movies”, “Attractions” and “Asia”.

[0077] In some embodiments, when playing media, the display device 200 can also perform image recognition on the content displayed on the display 260 to obtain information about feature targets contained in the currently displayed content. That is, during viewing, the user can send specified control commands to the display device 200 at any time to identify information about feature targets in the displayed content. The display device 200 can obtain various control commands input by the user, among which some control commands used to control the display device 200 to perform image recognition are called feature target recognition commands. The user can input feature target recognition commands on the display device 200 through different interaction methods or based on different user interfaces to trigger the image recognition function of the display device 200.

[0078] There are many ways to send a feature target recognition command to the display device 200. In some embodiments, the feature target recognition command can be sent directly to the display device 200 by the user, or it can be sent by the user through a control device 100, such as a mobile phone or a remote control. For example, the user can send a feature target recognition command to the display device 200 by operating a designated image recognition button on the remote control.

[0079] For example, when the display device 200 has a built-in intelligent voice system, users can also interact with the display device 200 by inputting voice. In some embodiments, users can also send feature target recognition commands to the display device 200 via voice. For instance, a user can say to the display device 200, "Who is this person?", "What brand is this clothing?", "What variety is this flower?", "Where is this?", etc. The display device 200 then performs voice recognition analysis on the above voice commands using a preset voice command recognition model, and then executes the corresponding operation in response to the voice commands. That is, when the display device 200 receives voice commands such as "Who is this person?", "What brand is this clothing?", "What variety is this flower?", "Where is this?", it takes a screenshot of the currently displayed screen to identify feature targets in the current screenshot.

[0080] However, since image recognition requires the identification of static images, in some embodiments, the display device 200 has a screenshot recognition function. That is, when the display device 200 receives a feature target recognition instruction, it takes a screenshot of the content displayed on the monitor 260 and then identifies the feature targets in the screenshot. After recognition, the display device 200 also displays the feature target recognition result on the monitor 260. The recognition result interface may include information about the feature target; alternatively, it may recommend media assets related to the feature target based on category tags for the user to choose from. Furthermore, if the feature target is a location-type feature such as a tourist attraction or shop, the recognition result may also include the location information of the feature target.

[0081] In some embodiments, there may be one or more feature targets; correspondingly, there may be one or more recognition results. Each recognition result may correspond to one feature target, and each feature target may correspond to multiple recognition results. The categories of feature targets may include portraits, objects, animals, plants, buildings, text, scenic spots, and graphics, etc. Feature targets identified in the same screenshot may be of the same category or different categories.

[0082] For example: Figure 7 As shown, the user is watching Variety Show A through display device 200. When Variety Show A plays to... Figure 7 At the indicated time point, only one distinctive target appeared on the display screen of display device 200: celebrity X. If the user wants to know information about celebrity X, they can send a voice command to display device 200 asking "Who is this?" to identify and search for celebrity X. Display device 200 responds to the distinctive target recognition command by taking a screenshot of the display screen and simultaneously searching for and identifying celebrity X within the image, thus obtaining... Figure 7The recognition results interface is shown below. By viewing the recognition results interface, users can not only obtain information about celebrity X, but also select and watch works in which celebrity X has participated.

[0083] Clearly, the display device 200 can perform image recognition on the screen content displayed on the monitor 260 in any viewing scenario. That is, in some embodiments, the display device 200 can receive and respond to the user's input feature target recognition command in all scenarios where the screen is on, such as when playing media assets, playing media assets from an external storage device, playing games, and browsing the homepage, and then perform image recognition on the feature targets in the current screen.

[0084] However, if the image displayed on the monitor 260 only includes partial features of the target when the user issues a feature target recognition command, recognition failure or errors may occur. For example, if the user sends a feature target recognition command to the display device 200 when the image displayed on the monitor 260 only shows half of a landscape, the display device 200 will respond by taking a screenshot of the currently displayed content. In this case, because the appearance features of the landscape in the image are incomplete, the display device 200 may be unable to recognize the landscape information or may recognize multiple similar landscape information, resulting in a low recognition recall and accuracy rate for the display device 200, thus reducing the user experience.

[0085] Based on the above application scenarios, in order to improve the user experience and address the issues of low recall and accuracy in display devices 200, some embodiments of this application provide an image recognition method. The method specifically includes the following:

[0086] S100: In response to the feature target recognition command, acquire the content to be displayed on the user interface.

[0087] The display 260 of the display device 200 is used to display the user interface. Upon receiving a feature target recognition command, the display device 200 acquires the display content of the user interface. The display content includes video content and / or image content. The video content can be video media played by the display device 200, and the image content can be photos played by the display device 200 or the homepage interface, media recommendation interface, etc., displayed on the display device.

[0088] Furthermore, since the feature target recognition command is a command used to control the display device 200 to perform image recognition, and the image recognition function requires the identification and analysis of static images, in some embodiments, the display device 200 also acquires a user-input start command and detects the activation status of the screenshot tool or screenshot service in response to the start command. The activation status is either an on or off state, and the start command is a control command used to turn on the display device. That is, when the display device 200 is powered on, the activation status of the screenshot tool or screenshot service can be automatically detected. If the screenshot tool or screenshot service is on, the step of acquiring the display content of the user interface in response to the feature target recognition command is executed, i.e., step S100 is executed; if the screenshot tool or screenshot service is off, the display 260 is controlled to display a prompt message indicating that image recognition cannot be performed.

[0089] In some embodiments, the prompt message displayed on the display 260 indicating that image recognition is not possible may also include a tab for enabling or diagnosing the screenshot tool or service. When the screenshot tool or service is manually disabled, the display 260 displays the tab for enabling the screenshot tool or service; when the screenshot tool or service is abnormally disabled, the display 260 displays the tab for diagnosing the screenshot tool or service.

[0090] For example, taking the screenshot tool as an example, after the user powers on display device 200, display device 200 will automatically detect the enabled status of the screenshot tool. Figure 8 As shown, after the display device 200 detects that the screenshot tool is closed, the display device 200's monitor 260 displays... Figure 8 The displayed message indicates that the user is in a state where the display device 200 cannot take screenshots and cannot perform image recognition. Meanwhile, Figure 8 The displayed prompt also includes a tab for enabling the screenshot tool, indicating that the screenshot tool on the current display device 200 has been manually disabled by the user. Therefore, the user can click the "Enable Screenshot Tool" tab and, according to the index and prompt displayed on the monitor 260, enable the screenshot tool, thereby activating the image recognition function of the current display device 200.

[0091] S200: Read the category tags of the displayed content.

[0092] After acquiring the display content, the display device 200 can also read the tags of the display content to obtain basic information contained in the display content. For example, Figure 6The category tags for "Media Asset A" include "Movies," "Attractions," and "Asia." Therefore, after reading the category tags of "Media Asset A," the display device 200 obtains that the currently displayed content includes attraction information. Thus, before identifying the feature targets in the displayed content, it can be determined that the feature targets included in the currently displayed content are likely attractions.

[0093] S300: If the category label of the displayed content includes a feature target identifier, then take screenshots of the displayed content continuously within a preset time to generate multiple screenshots.

[0094] After reading that the category label includes the feature target identifier, the display device 200 takes multiple screenshots of the content displayed on the monitor 260 within a preset time period, generating multiple screenshot images so that the display device 200 can obtain more images containing the feature target.

[0095] However, since the display device 200 consumes significant system resources when taking screenshots using a screenshot tool or service, the number of screenshots should be minimized during an image recognition process to avoid increasing the memory capacity of the display device 200. Furthermore, since some features are easily identifiable, detailed information about the features can be obtained simply by identifying their local features, such as categories like plants, animals, or cars, eliminating the need for multiple screenshots. Therefore, in some embodiments, the display device 200 can define the category of the feature target, such as setting the feature target to categories like scenic spots or portraits, and using synonyms like scenic spots, scenery, or landscapes as identifiers. Moreover, the preset time and execution of screenshots within that time in step S300 is also to reduce the system resource consumption caused by the display device 200 during screenshot execution; therefore, a shorter time period can be set.

[0096] For example: the preset time is 50 milliseconds, the feature target is a scenic spot, and the feature target identifier is scenic spot, scenery, or landscape. Figure 9 As shown, the user is watching Figure 9 When referring to the "Media Asset C" shown, what do you want to do? Figure 9 To identify the information displayed on the screen, a feature target recognition command is sent to the display device 200 via remote control. The display device 200 responds to the feature target recognition command by reading the category tags of "Media Asset C," which includes the first-level tag "Documentary," the second-level tag "Attractions," and the third-level tag "Asia." Upon detecting the "Attractions" tag within the category tags, the display device 200 takes multiple screenshots of the content displayed on the monitor 260 within 50 milliseconds, generating multiple screenshot images.

[0097] In some embodiments, if the category labels of the displayed content do not include feature target identifiers, a screenshot is taken only once to obtain the image to be identified. Then, non-feature targets in the image to be identified are identified to generate non-feature target identification information, and the display 260 is controlled to display the non-feature target identification information, thereby effectively saving system resource consumption.

[0098] For example: the feature target is a scenic spot, and the feature target identifier is scenic spot, scenery, or landscape. Figure 10 As shown, the user is watching Figure 10 When referring to the "Media Assets D" shown, I want to... Figure 10 To identify the information displayed in the content, the user speaks a voice command to the display device 200 asking "Who is this?" The display device 200 then responds to the command by reading the category tags of "Media Asset D," which include the primary tag "Movie," the secondary tag "Romance," and the tertiary tag "Suspense." Since the category tags of "Media Asset D" do not include identifiers for scenic spots, objects, or landscapes, the display device 200 takes a screenshot of the currently displayed content to obtain the image to be identified. Simultaneously, it identifies non-featured target information in the image to be identified. Figure 10 The system identifies the girl and displays the identification results on monitor 260. The identification results interface includes the girl's profile and related acting credits.

[0099] Furthermore, in some embodiments, after reading that the category label of the displayed content includes a feature target identifier, the display device 200 also obtains the playback scene of the displayed content. The playback scene can be a static playback scene or a dynamic playback scene. If the playback scene is a dynamic playback scene, screenshots are continuously taken of the displayed content within a preset time period. If the playback scene is a static playback scene, a single screenshot is taken of the displayed content to obtain a feature image. Feature targets in the feature image are then identified, and feature target information is generated.

[0100] Clearly, a static playback scenario is one where the image continuously changes. For example, when a user browses the homepage of display device 200, the homepage may include static image information; in this case, the playback scenario of display device 200 is a static playback scenario. A dynamic playback scenario can be a video playback scenario. For example, when a user watches a TV series on display device 200, the video displayed on monitor 260 continuously changes its image; in this case, the playback scenario of display device 200 is a dynamic playback scenario. Similarly, when a user scrolls through image content on display device 200, the content displayed on monitor 260 also continuously changes; in this case, the playback scenario of display device 200 is also a dynamic playback scenario.

[0101] Since the displayed content remains unchanged during static playback, multiple screenshots will capture the same image. Therefore, after reading the feature target identifier from the category tags of the displayed content, if the playback scenario of the display device 200 is static, multiple screenshots will not be performed; only one screenshot will be taken for image recognition to reduce system resource consumption.

[0102] Furthermore, when the display device 200 plays video, if the video's media asset type is a non-live signal, it will play and display the video resource at a certain frame rate, meaning the video content is composed of frames. If the display device 200 captures the same frame when taking a screenshot, it will result in multiple screenshots containing identical features, leading to a waste of system resources.

[0103] Therefore, in some embodiments, if the displayed content is video content, the media asset type of the video content is also obtained. If the media asset type is a non-live signal, the frame rate information of the video content is obtained, and then the interval time of each frame in the video content is calculated. If the media asset type is a live signal, the interval time is set according to the preset configuration information in the display device 200. Finally, within the preset time, a screenshot is taken of the displayed content every time interval.

[0104] For example: if the preset time is 45 milliseconds, and the video played by display device 200 is a non-live broadcast media resource with a frame rate of 60 frames per second, then the time interval between a single image is 15 milliseconds. Therefore, the interval time is set to 15 milliseconds. Within the 45 milliseconds, display device 200 takes a screenshot of the content displayed on monitor 260 every 15 milliseconds. That is, display device 200 will take three screenshots during this image recognition process, generating a total of three screenshots.

[0105] For example, if the preset time is 45 milliseconds, the video played by display device 200 is a live video signal, and the preset interval time in display device 200 is 10 milliseconds, then display device 200 will take a screenshot of the content displayed on monitor 260 every 10 milliseconds within the 45 milliseconds. That is, display device 200 will take four screenshots during this image recognition process, generating a total of four screenshots.

[0106] Furthermore, in some embodiments, the display device 200 can also set a threshold for the number of times screenshots are taken. When the number of times screenshots are taken of the displayed content equals the threshold, the process of taking screenshots of the displayed content stops. Since the display device 200 displays and plays videos with various frame rates, setting a threshold can prevent the display device 200 from taking too many screenshots, thus avoiding a waste of resources.

[0107] For example, the threshold for the number of screenshots is 2, the preset time is 45 milliseconds, and the interval time is 15 milliseconds. Within the 45 milliseconds, the display device 200 will take a screenshot of the content displayed on the monitor 260 every 15 milliseconds. Since the threshold for the number of screenshots is 2, the display device 200 will immediately stop taking screenshots of the displayed content after the second screenshot. That is, the display device 200 will take two screenshots during this image recognition process, generating a total of two screenshots.

[0108] S400: Stitch together multiple screenshots to form a panoramic image.

[0109] In order to obtain a panoramic image of the feature target, the display device 200 needs to stitch and merge multiple captured screenshots to obtain an image that contains more features of the feature target.

[0110] For example: the characteristic target is a mountain range. Figure 11 and Figure 12 The mountain range shown is captured by display device 200, with each segment showing a different view. Figure 11 and Figure 12 Screenshot of the scene. Figure 11 and Figure 12 The image includes local features at different locations within the mountain range. To accurately identify the mountain range information, the display device 200 will... Figure 11 , Figure 12 The screenshots are stitched together into one image, such as Figure 13 The image shown is a panoramic view of the mountain range.

[0111] However, since multiple screenshots may contain duplicate feature points, directly stitching them together can result in repetitive scenes or blank spaces, failing to directly yield a panoramic image of the target. Therefore, in some embodiments, when stitching multiple screenshots, the display device 200 also extracts feature points from each screenshot. Then, it determines the stitching region among the multiple screenshots based on these feature points. Here, feature points are pixels in each screenshot that contain feature factors, such as SIFT, SURF, Harris corner points, ORB, etc.; the stitching region is a set of feature points in the screenshots that contain the same feature factors. Finally, the screenshots are set to the stitching region based on the feature point set to stitch the multiple screenshots together.

[0112] For example: the feature target is a mountain range, such as Figure 14 As shown, Figure 14 for Figure 11 , Figure 12 The splicing process. Display device 200 during the splicing process. Figure 11 , Figure 12 When stitching together screenshots, first... Figure 11 , Figure 12 Feature points are extracted, and identical feature points are matched. Among these, Figure 14 In a diagram, two feature points pointed to by a double arrow are considered successfully matched feature points. For example... Figure 14 As shown, by using the successfully matched feature point set, we can determine... Figure 11 , Figure 12 The splicing area in the middle can be spliced ​​to create... Figure 13 Panoramic image of the Zhongshan Mountains.

[0113] Furthermore, to accurately stitch the screenshots within the stitching area, in some embodiments, the display device 200 also establishes a registration coordinate system. A transformation matrix is ​​then generated based on the feature point set and the registration coordinate system. This transformation matrix converts the coordinates of pixels in the screenshots into coordinates within the registration coordinate system. Multiple screenshots are then copied into the registration coordinate system according to the converted coordinates.

[0114] For example, the display device 200 can obtain the transformation matrix using the findHomography function, and then use the transformation matrix to transform the coordinates of the pixels in the screenshot. Finally, multiple screenshots are copied into the registration coordinate system according to the transformed coordinates, and a panoramic image of the feature target can be stitched together.

[0115] However, when the image displayed on the display device 200 changes, factors such as different lighting, shooting angle, and focal length during the shooting process may cause variations in the brightness, angle, or size of feature objects in each frame. Therefore, unnatural transitions may occur in the stitched panoramic image.

[0116] In some embodiments, the display device 200 also acquires an overlapping image of multiple screenshots stitched together. Then, it extracts the pixel value information of the overlapping image and adds the pixel values ​​based on a weighted fusion algorithm to adjust the perspective and brightness of the merged screenshots.

[0117] For example: Figure 15 As shown, Figure 15 This is the stitched panoramic image. Due to differences in lighting and color, Figure 15 The transition at the boundaries of a panoramic image is poor. Therefore, the display device 200 can add the pixel values ​​of the overlapping areas after stitching together screenshots according to certain weights. The resulting image can then be reconstructed as a panoramic image. Figure 13 The panoramic image shown has a natural transition.

[0118] It should be noted that the image stitching and fusion method in this application can also employ other image processing methods, such as wavelet transform. This application does not impose any limitations on this method.

[0119] S500: Identify feature targets in the panoramic image to generate feature target information, and control the display to show the feature target information.

[0120] Display device 200 identifies panoramic images by extracting feature targets from the panoramic images, searching and matching the information of the feature targets, and finally displaying the feature target information on display 260. The feature target information is the recognition result after image recognition by display device 200. The recognition result is the same as that described in the above embodiments, and will not be repeated here.

[0121] In some embodiments, the interface for the recognition results may also include other auxiliary information to assist the display device 200 in displaying the recognition results. For example, the interface for the recognition results may also include one or more of the following: confidence level of the recognition result, confidence threshold, whether the confidence level of the recognition result exceeds the confidence threshold, the position of the feature target in the screenshot, display mode parameters of the recognition result, and additional information corresponding to the confidence level. The confidence level of the recognition result refers to the reliability of the recognition result; a higher confidence level indicates a more reliable recognition result, and a lower confidence level indicates a less reliable recognition result. A confidence level exceeding the confidence threshold indicates a reliable recognition result, while a confidence level below the confidence threshold indicates an unreliable recognition result.

[0122] For example, the display device 200 can generate the confidence score of the recognition result based on the resolution of the screenshot; the higher the resolution of the screenshot, the higher the confidence score, and the lower the resolution of the screenshot, the lower the confidence score. Alternatively, it can generate the confidence score of the recognition result based on the length and width of the region where the feature target is located; the larger the length and width values, the higher the confidence score, and the smaller the length and width values, the lower the confidence score. Or, it can also generate the confidence score of the recognition result based on the position or proportion of the feature target in the screenshot.

[0123] In some embodiments, if there are multiple feature targets, the display device 200 displays multiple feature target information prompt windows, that is, multiple recognition result prompt windows are displayed on the display 260.

[0124] For example, there are two feature targets, namely a first feature target and a second feature target. The display device 200 can display a first feature target recognition window on the display 260 based on the coordinate base point (X1, Y1), the height H1, and the width W1 of the area where the first feature target is located in the panoramic image; and then, based on the coordinate base point (X2, Y2), the height H2, and the width W2 of the area where the second feature target is located in the panoramic image, it can display a second feature target recognition window on the display 260.

[0125] Based on the above image recognition method, some embodiments of this application also provide a display device 200, such as... Figure 16 As shown, it includes: a display 260 and a controller 250. (As shown) Figure 17 As shown, the controller 250 is configured to perform the following program steps:

[0126] S100: In response to the feature target recognition instruction, obtain the display content of the user interface, the display content including video content and / or image content;

[0127] S200: Read the category tags of the displayed content;

[0128] S300: If the category label of the displayed content includes a feature target identifier, then take screenshots of the displayed content continuously within a preset time to generate multiple screenshots;

[0129] S400: Stitch together multiple screenshots into a panoramic image;

[0130] S500: Identify feature targets in the panoramic image to generate feature target information, and control the display to show the feature target information.

[0131] As can be seen from the above technical solutions, the display device and image recognition method provided in some embodiments of this application can obtain the display content of the user interface in response to a feature target recognition instruction. The display content includes video content and / or image content. When the feature target identifier is included in the category label of the display content, the method can continuously take multiple screenshots of the display content within a preset time to generate multiple screenshots of the feature target. These multiple screenshots are then stitched together to form a panoramic image of the feature target, and the panoramic image is recognized to generate feature target information, thereby solving the problem of low recall rate in feature target recognition and improving user experience.

[0132] Similar parts between the embodiments provided in this application can be referred to mutually. The specific implementation methods provided above are only a few examples under the overall concept of this application and do not constitute a limitation on the scope of protection of this application. For those skilled in the art, any other implementation methods extended from the solution of this application without creative effort shall fall within the scope of protection of this application.

Claims

1. A display device, characterized in that, include: The monitor is configured to display the user interface; The controller is configured as follows: In response to a feature target recognition instruction, the display content of the user interface is obtained, the display content including video content and / or image content; Read the category tags of the displayed content; If the category label of the displayed content includes a feature target identifier, then screenshots of the displayed content will be taken continuously within a preset time to generate multiple screenshots. The multiple screenshots are stitched together to form a panoramic image; The system identifies feature targets in the panoramic image to generate feature target information and controls the display to show the feature target information.

2. The display device according to claim 1, characterized in that, The controller is also configured to: Obtain a user-input start command, which is used to turn on the display device; In response to the startup command, the activation status of the screenshot tool or screenshot service is detected, wherein the activation status is either on or off. If the screenshot tool or screenshot service is enabled, then the step of obtaining the display content of the user interface in response to the feature target recognition instruction is executed; If the screenshot tool or screenshot service is not enabled, the display will be controlled to show a message indicating that image recognition is not possible.

3. The display device according to claim 1, characterized in that, The controller is also configured to: Obtain the playback scenario of the displayed content, wherein the playback scenario is a static playback scenario or a dynamic playback scenario; If the playback scenario is a dynamic playback scenario, then screenshots will be continuously taken of the displayed content within a preset time period; If the playback scenario is a static playback scenario, a screenshot is taken of the displayed content to obtain a feature image; And identify feature targets in the feature image to generate feature target information.

4. The display device according to claim 1, characterized in that, The controller is also configured to: If the category label of the displayed content does not include a feature target identifier, then a screenshot is taken of the displayed content to obtain the image to be identified; The system identifies non-feature targets in the image to be identified, generates non-feature target identification information, and controls the display to show the non-feature target identification information.

5. The display device according to claim 1, characterized in that, If the displayed content is video content, the controller is further configured to: Obtain the media asset type of the video content; If the media asset type is a non-live signal, then obtain the frame rate information of the video content and calculate the interval time of each frame in the video content; If the media asset type is a live signal, then the interval time is set according to the preset configuration information; Within a preset time period, a screenshot is taken of the displayed content every specified interval.

6. The display device according to claim 5, characterized in that, The controller is also configured to: Set a threshold for the number of times screenshots are taken; When the number of times the displayed content is captured equals the threshold number, the process of capturing the displayed content is stopped.

7. The display device according to claim 1, characterized in that, The controller is configured to stitch together multiple screenshots into a panoramic image. Extract feature points from each of the screenshots, where each feature point is a pixel containing a feature factor in each screenshot; Based on the feature points, a splicing region is determined among multiple screenshots, and the splicing region is a set of feature points containing the same feature factors in the screenshots; The screenshots are set to the stitching area based on the feature point set in order to stitch together multiple screenshots.

8. The display device according to claim 7, characterized in that, The controller is configured to copy multiple screenshots to a specified location and also to: Establish a registration coordinate system; A transformation matrix is ​​generated based on the feature point set and the registration coordinate system. The transformation matrix is ​​used to convert the coordinate points of the pixels in the screenshot into coordinate points in the registration coordinate system. The multiple screenshots are copied into the registration coordinate system according to the converted coordinate points.

9. The display device according to claim 7, characterized in that, The controller is also configured to: Obtain the overlapping portion of the image obtained by stitching together multiple screenshots; Extract the pixel value information of the overlapping portion of the image; The pixel values ​​are added together using a weighted fusion algorithm to adjust the perspective and brightness of the multiple screenshots.

10. An image recognition method, characterized in that, include: In response to a feature target recognition command, the user interface display content is obtained, including video content and / or image content; Read the category tags of the displayed content; If the category label of the displayed content includes a feature target identifier, then screenshots of the displayed content will be taken continuously within a preset time to generate multiple screenshots. The multiple screenshots are stitched together to form a panoramic image; Identify feature targets in the panoramic image to generate feature target information, and control the display to show the feature target information.

Images