Display device and circle selection screenshot method

Abstract

The invention relates to a display device and a circle selection screenshot method. The display device comprises a display and a controller, and the controller is configured to: in a video stream playing process, in response to a received preset pressing instruction, store a first image corresponding to a picture displayed by the display; wherein the preset pressing instruction represents that a preset key pointing to the remote controller is pressed; in the process of receiving a pointing parameter sent by the pointing remote controller and generating a trajectory according to the received pointing parameter, storing a second image corresponding to a display picture of the display; wherein the pointing parameter changes along with the movement of the pointing remote controller; determining a circling area according to the trajectory, and performing screenshot on the first image and the second image according to the circling area to obtain an area image set; and performing image recognition according to the regional image set to display the character information of the recognized character. By adopting the method, the accuracy of the circled screenshot can be ensured when the video picture is displayed on the display.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer, in particular to a display device and a circle selection screenshot method. BACKGROUND

[0002] With the development of devices (such as smart televisions), some devices integrate screenshot (such as circle selection screenshot, full screen screenshot, etc.) functions to meet the diversified information query needs of users for the interface displayed by the device. For example, the device can take a screenshot of part of the display content in the displayed interface, and perform a search based on the part of the image to display the search results for the part of the image to the user.

[0003] However, since the video picture is real-time changing, the processing logic of the circle selection screenshot for a single frame picture in the related technology will reduce the accuracy of the circle selection screenshot when the circle selection screenshot is performed on the video picture. SUMMARY

[0004] The present application provides a display device and a circle selection screenshot method, which can ensure the accuracy of the circle selection screenshot when the display displays the video picture.

[0005] In a first aspect, some embodiments provide a display device, comprising:

[0006] a display and a controller;

[0007] the controller is configured to:

[0008] in a process of playing a video stream, in response to a received preset pressing instruction, store a first image corresponding to a picture displayed by the display; wherein the preset pressing instruction represents that a preset key in a remote controller is pressed;

[0009] in a process of receiving a pointing parameter sent by the remote controller and generating a trajectory line according to the received pointing parameter, store a second image corresponding to the picture displayed by the display; wherein the pointing parameter changes with the movement of the remote controller;

[0010] determine a circle selection region according to the trajectory line, and take a screenshot of the first image and the second image according to the circle selection region to obtain a region image set;

[0011] perform image recognition according to the region image set to display the information of the recognized person.

[0012] The technical scheme, in the process of playing the video stream, on one hand, the first image corresponding to the display screen is stored when the preset pressing instruction is received, and the second image corresponding to the display screen is stored in the process of generating the trajectory line according to the received pointing parameter; then, the first image and the second image are screenshot according to the selected area determined based on the trajectory line, and the regional image set is obtained, which can simultaneously select and screenshot multiple images displayed on the display in the selection operation, ensuring the integrity and accuracy of the selection operation; on the other hand, the image recognition is performed on each regional image in the regional image set, and the identified character information is displayed, improving the comprehensiveness of the displayed character information, and making the displayed character information more in line with the actual needs of the user.

[0013] In some embodiments, the controller is configured to store the second image corresponding to the display screen in the process of receiving the pointing parameter sent by the remote controller and generating the trajectory line according to the received pointing parameter.

[0014] In the process of receiving the pointing parameter sent by the remote controller and generating the trajectory line according to the received pointing parameter, the video screen displayed by the display is screenshot at a set time interval to obtain the second image, and the second image is stored.

[0015] The technical scheme, by setting a time interval to screenshot the video screen displayed by the display to obtain the second image, provides a simple, fast and optional way to obtain the second image.

[0016] In some embodiments, the controller is configured to store the second image corresponding to the display screen in the process of receiving the pointing parameter sent by the remote controller and generating the trajectory line according to the received pointing parameter.

[0017] In the process of receiving the pointing parameter sent by the remote controller and generating the trajectory line according to the received pointing parameter, the video frame is detected, and the key frame is stored as the second image when it is detected that the video frame is a key frame.

[0018] The technical scheme, by directly taking the key frame in the video stream as the second image, on one hand, provides an optional way to obtain the second image; on the other hand, since the key frame is an important video frame in the video stream, it contains more character information, thereby ensuring the reliability of the second image, so that the final obtained character information is more rich and accurate.

[0019] In some embodiments, the controller is configured to display the character information of the identified character.

[0020] display the character information of the recognized character, and image download information; wherein the image download information comprises a to-be-downloaded image and a download portal; the to-be-downloaded image comprises at least one of a region image set, a target video, and an animation generated based on the region image set; the target video is obtained by cropping a screen video according to the selected region, and the screen video is obtained by screen recording a display interface of the display in the process of generating the trajectory line.

[0021] The technical solution has the following advantages. On the one hand, the introduction of the download portal corresponding to the to-be-downloaded image enables the user to download the to-be-downloaded image when selecting the screenshot, in addition to obtaining the character information in the image, thereby enriching the function of selecting the screenshot. On the other hand, the to-be-downloaded image comprises a video related to the entire process of selecting the screenshot, i.e., the target video, which facilitates the user to intuitively understand the entire process of selecting the screenshot, thereby further improving the user experience of selecting the screenshot.

[0022] In some embodiments, the character information of the recognized character is fed back after the server identifies each region image in the region image set and removes duplicate character information of the recognized character in each region image.

[0023] The technical solution has the following advantages. On the one hand, the removal of the duplicate character information ensures the conciseness of the character information displayed to the user. On the other hand, the deletion of the duplicate character information also reduces the storage space occupied by the display device.

[0024] In some embodiments, the character information of the recognized character is fed back after the server removes the character information of the same character with the same confidence and removes the character information of the same character with different confidence, except for the highest confidence.

[0025] The technical solution has the following advantages. On the one hand, the removal of the duplicate character information ensures the reliability and conciseness of the character information displayed. On the other hand, the deletion of the duplicate character information also reduces the storage space occupied by the display device.

[0026] In some embodiments, the controller is further configured to, in the process of receiving the pointing parameter sent by the remote controller and generating the trajectory line according to the received pointing parameter, perform the following operations:

[0027] display the generated trajectory line on an upper layer of the video picture displayed by the display; wherein a region in the upper layer other than the trajectory line is a transparent region.

[0028] The technical solution above, by covering a transparent layer on top of the video picture when performing the circle selection screenshot processing, can display the drawn trajectory line without affecting the video playing.

[0029] In some embodiments, after obtaining the set of region images, the controller is further configured to:

[0030] The target image is displayed in a preset display area of the layer on top of the video picture displayed by the display, and search instruction information is displayed; wherein, the area of the preset display area is smaller than the display area of the video picture, the area of the layer on top of the video picture other than the preset display area is a transparent area, the search instruction information is used to inform the user that the image search is being performed, and the target image is any region image in the set of region images or the set of region images.

[0031] The technical solution above, by displaying the target image in the preset display area of the layer on top of the video picture and the search instruction information, can enable the user to intuitively understand the image search progress and improve the user experience.

[0032] In some embodiments, when the controller performs the determination of the circle selection region according to the trajectory line, it is configured to:

[0033] Select the minimum horizontal coordinate, the maximum horizontal coordinate, the minimum vertical coordinate and the maximum vertical coordinate from the coordinate data of the trajectory points on the trajectory line;

[0034] Combine the minimum horizontal coordinate, the maximum horizontal coordinate, the minimum vertical coordinate and the maximum vertical coordinate horizontally and vertically to obtain at least two vertex coordinates; wherein, the rectangular region represented by the at least two vertex coordinates is the circle selection region.

[0035] The technical solution above, by determining the vertex coordinates of the circle selection region based on the minimum horizontal coordinate, the maximum horizontal coordinate, the minimum vertical coordinate and the maximum vertical coordinate in the coordinate data of the trajectory points, provides an optional way for quickly determining the circle selection region while ensuring that the subsequently obtained region image can contain effective image information. In addition, the region determined based on the trajectory line is presented in a rectangular shape, which is adapted to the way in which the display device presents the page, and thus the region image obtained is more reasonable.

[0036] In a second aspect, some embodiments further provide a circle selection screenshot method applied to a display device, comprising:

[0037] In the process of playing the video stream, in response to a received preset pressing instruction, a first image corresponding to the display picture is stored; wherein, the preset pressing instruction represents that a preset key in the remote controller is pressed.

[0038] In the process of receiving the pointing parameter sent by the pointing remote controller and generating the trajectory line according to the received pointing parameter, the second image corresponding to the display screen is stored; wherein the pointing parameter changes with the movement of the pointing remote controller;

[0039] According to the trajectory line, the circling selection region is determined, and the first image and the second image are screenshot according to the circling selection region to obtain the region image set;

[0040] According to the region image set, image recognition is performed to display the character information of the recognized character.

[0041] In a third aspect, some embodiments further provide a circling screenshot device applied to a display device, comprising:

[0042] The first storage module is configured to store the first image corresponding to the display screen in response to the received preset pressing instruction during the playing of the video stream; wherein the preset pressing instruction represents that a preset key in the pointing remote controller is pressed.

[0043] The second storage module is configured to store the second image corresponding to the display screen in the process of receiving the pointing parameter sent by the pointing remote controller and generating the trajectory line according to the received pointing parameter; wherein the pointing parameter changes with the movement of the pointing remote controller.

[0044] The screenshot module is configured to determine the circling selection region according to the trajectory line, and perform screenshot on the first image and the second image according to the circling selection region to obtain the region image set.

[0045] The character recognition module is configured to perform image recognition according to the region image set to display the character information of the recognized character.

[0046] In a fourth aspect, a computer readable storage medium is provided, and the computer readable storage medium stores a computer program. When the computer program is run by a data display device, the data display device executes the circling screenshot method in the second aspect.

[0047] In a fifth aspect, a computer program product is provided, and the computer program product comprises a computer program. When the computer program is run by a data display device, the data display device executes the circling screenshot method in the second aspect. BRIEF DESCRIPTION OF DRAWINGS

[0048] In order to more clearly illustrate the technical solutions in the some embodiments or the related art, the following will briefly introduce the drawings needed to be used in the description of some embodiments or the related art. Obviously, the drawings in the following description only some embodiments of the present application, and for those skilled in the art, other related drawings can also be obtained without creative labor on the basis of these drawings.

[0049] Figure 1 The schematic diagram of the operation scenario between the display device and the control device provided by some embodiments of the present application;

[0050] Figure 2 The schematic diagram of the hardware configuration of the display device provided by some embodiments of the present application;

[0051] Figure 3 The schematic diagram of the hardware configuration of the control device provided by some embodiments of the present application;

[0052] Figure 4 The schematic diagram of the software configuration of the display device provided by some embodiments of the present application;

[0053] Figure 5 The schematic diagram of the flow of the screenshot method provided by some embodiments of the present application;

[0054] Figure 6 The schematic diagram of the information display page provided by some embodiments of the present application;

[0055] Figure 7 The schematic diagram of the information display page provided by some other embodiments of the present application;

[0056] Figure 8 The schematic diagram of the image search page provided by some embodiments of the present application;

[0057] Figure 9 The schematic diagram of the information display page provided by some other embodiments of the present application;

[0058] Figure 10 The schematic diagram of the flow of determining the rectangular region provided by some embodiments of the present application;

[0059] Figure 11 The schematic diagram of determining the rectangular region corresponding to the regular track line provided by some embodiments of the present application;

[0060] Figure 12 The schematic diagram of determining the rectangular region corresponding to the irregular track line provided by some embodiments of the present application;

[0061] Figure 13 The schematic diagram of determining the rectangular region corresponding to the track line with invalid width provided by some embodiments of the present application;

[0062] Figure 14 The following is a schematic diagram of the interaction between devices in the circle screenshot process provided by some embodiments of the present application.

[0063] Figure 15 The following is a schematic diagram of the structure of the circle screenshot device in some embodiments. DETAILED DESCRIPTION

[0064] The embodiments will be described in detail with reference to the drawings, wherein like reference numerals refer to like elements throughout. The following description is made with reference to the accompanying drawings in which the same or like reference numerals in different drawings represent the same or similar elements. The embodiments described in the following description are not meant to be exhaustive or to be limited to the precise forms disclosed. These embodiments are chosen and described so that others skilled in the art can appreciate and understand the principles and the practices of the present application.

[0065] It should be noted that the brief description of the terms in the present application is only for the convenience of understanding the following described embodiments, and is not intended to limit the embodiments of the present application. Unless otherwise specified, these terms should be understood according to their ordinary and common meanings.

[0066] The terms "first", "second", "third", and the like in the specification and claims of the present application and the above-described drawings are used to distinguish similar or like objects or entities, and do not necessarily mean to limit the specific order or sequence, unless otherwise noted. It should be understood that the terms used in this way can be interchanged under appropriate circumstances.

[0067] The terms "include" and "have" and any variations thereof are intended to cover but not exclusive of the inclusion, for example, a product or device that includes a series of components is not necessarily limited to all the components clearly listed, but can include other components that are not clearly listed or inherent to these products or devices.

[0068] The term "module" refers to any known or later developed hardware, software, firmware, artificial intelligence, fuzzy logic, or a combination of hardware and / or software code capable of performing a function associated with that element.

[0069] In some embodiments, the display device 200 refers to a device with picture display and data processing capabilities. For example, the display device 200 includes but is not limited to a smart television, a mobile terminal, a computer, a monitor, an advertising screen, a wearable device, a virtual reality device, an augmented reality device, etc.

[0070] Figure 1 The following is a schematic diagram of the operation scenario between the display device and the control device provided by some embodiments of the present application. As shown in Figure 1As shown, users can operate the display device 200 via touch operation, mobile terminal 300, and control device 100. For example, control device 100 can be a remote control, stylus, gamepad, etc.

[0071] The mobile terminal 300 can serve as a control device for human-computer interaction between the user and the display device 200. The mobile terminal 300 can also serve as a communication device for establishing a communication connection with the display device 200 for data interaction. In some embodiments, the mobile terminal 300 can install software applications with the display device 200 to establish a connection and communication via network communication protocols, achieving one-to-one control operations and data communication. It can also transmit audio and video content displayed on the mobile terminal 300 to the display device 200 to achieve synchronous display.

[0072] like Figure 1 The diagram also shows that the display device 200 communicates with the server 400 via various communication methods. This allows the display device 200 to communicate via a local area network (LAN), a wireless local area network (WLAN), and other networks.

[0073] Display device 200 can provide broadcast television reception function, and can also be equipped with intelligent network television function that provides computer support, including but not limited to network television, smart television, Internet Protocol Television (IPTV), etc.

[0074] Figure 2 Provided for some embodiments of this application Figure 1 Hardware configuration block diagram of display device 200.

[0075] In some embodiments, the display device 200 may include at least one of a tuner 210, a communication device 220, a detector 230, a device interface 240, a controller 250, a display 260, an audio output device 270, a memory, a power supply, and a user input interface.

[0076] In some embodiments, detector 230 is used to acquire signals from the external environment or to interact with the outside world. For example, detector 230 includes a light receiver, a sensor for acquiring ambient light intensity; or, detector 230 includes an image acquisition device, such as a camera, which can be used to acquire external environmental scenes, user attributes, or user interaction gestures; or, detector 230 includes a sound acquisition device, such as a microphone, for receiving external sounds.

[0077] In some embodiments, the display 260 includes display components for presenting a picture, and driving components for driving the display of the picture. The display 260 is configured to receive and display picture signals output from the controller 250. For example, the display 260 can be configured to display video content, image content, components of a menu control interface, a user control UI interface, and the like.

[0078] In some embodiments, the communication device 220 is a component for communicating with external devices or the server 400 according to various communication protocol types. The display device 200 can be provided with multiple communication devices 220 according to different supported communication manners. For example, when the display device 200 supports wireless network communication, the display device 200 can be provided with a communication device 220 including a Wireless Fidelity (WiFi) function. When the display device 200 supports Bluetooth connection communication, the display device 200 needs to be provided with a communication device 220 including a Bluetooth function.

[0079] The communication device 220 can be configured to connect the display device 200 to external devices or the server 400 through wireless or wired connection. The wired connection can be achieved by connecting the display device 200 to external devices through data lines, interfaces, and the like. The wireless connection can be achieved by connecting the display device 200 to external devices through wireless signals or wireless networks. The display device 200 can be directly connected to external devices, or can be indirectly connected to external devices through gateways, routers, connection devices, and the like.

[0080] In some embodiments, the controller 250 can include at least one of a central processor, a video processor, an audio processor, a graphics processor, a power supply processor, first to nth interfaces for input / output, and the like. The controller 250 controls the operation of the display device and responds to user operations by controlling various software control programs stored in the memory. The controller 250 controls the overall operation of the display device 200.

[0081] In some embodiments, the controller 250 and the tuner demodulator 210 can be located in different split devices, i.e., the tuner demodulator 210 can also be located in an external device of the main device where the controller 250 is located, such as an external set-top box, and the like.

[0082] In some embodiments, the user can input user commands through a Graphical User Interface (GUI) displayed on the display 260, and the user input interface receives the user input commands through the Graphical User Interface (GUI).

[0083] In some embodiments, the audio output device 270 can be a built-in speaker of the display device 200 or an external audio output device connected to the display device 200. For the external audio output device connected to the display device 200, the display device 200 may also be provided with an external audio output terminal, through which the audio output device can be connected to the display device 200 to output sound from the display device 200.

[0084] In some embodiments, the user input interface 280 can be used to receive instructions from user input.

[0085] Figure 3 Provided for some embodiments of this application Figure 1 Hardware configuration block diagram of the central control device. (Example) Figure 3 As shown, the control device 100 may include: a controller 110, a communication interface 130, a user input / output interface, a memory, and a power supply.

[0086] The control device 100 is configured to control the display device 200, and to receive user input operation commands and convert the operation commands into commands that the display device 200 can recognize and respond to, thus acting as an intermediary for interaction between the user and the display device 200.

[0087] In some embodiments, the control device 100 may be an intelligent device. For example, the control device 100 may be equipped with various applications for controlling the display device 200 according to user needs.

[0088] In some embodiments, such as Figure 1 As shown, the mobile terminal 300 or other smart electronic devices can perform similar functions to the control device 100 after installing the application of the control display device 200.

[0089] The controller 110 includes a processor 112, RAM 113, ROM 114, a communication interface 130, and a communication bus. The controller 110 is used to control the operation of the control device 100, as well as the communication and cooperation between internal components and the external and internal data processing functions.

[0090] Under the control of the controller 110, the communication interface 130 enables communication of control signals and data signals with the display device 200. The communication interface 130 may include at least one of other near-field communication modules such as WiFi chip 131, Bluetooth module 132, and NFC module 133.

[0091] User input / output interface 140, wherein the input interface includes at least one of other input interfaces such as microphone 141, touchpad 142, sensor 143, and button 144.

[0092] In some embodiments, the control device 100 comprises at least one of a communication interface 130 and an input / output interface 140. The communication interface 130, such as a WiFi, Bluetooth, NFC, etc. module, is configured to encode and send user input instructions to the display device 200 via a WiFi protocol, a Bluetooth protocol, or an NFC protocol, under the control of the controller.

[0093] The memory 190 is configured to store various running programs, data, and applications for driving and controlling the control device 100 under the control of the controller. The memory 190 can store various control signal instructions input by the user.

[0094] The power supply 180 is configured to provide operating power support for the elements of the control device 100 under the control of the controller.

[0095] To perform user interaction, in some embodiments, the display device 200 can run an operating system. The operating system is a computer program for managing and controlling hardware resources and software resources in the display device 200. The operating system can provide a user interface to allow the user to interact with the display device 200 and support running various application programs.

[0096] It should be noted that the operating system can be a native operating system based on a specific operating platform, a third-party operating system deeply customized based on a specific operating platform, or an independent operating system specially developed for the display device.

[0097] The operating system can be divided into different modules or levels according to the functions implemented, for example, as shown in FIG. 2, in some embodiments, the system is divided into four layers, from top to bottom, the Applications layer (referred to as the "application layer"), the Application Framework layer (referred to as the "framework layer"), the system library layer, and the kernel layer. Figure 4

[0098] In some embodiments, the application layer is used to provide services and interfaces for application programs, so that the display device 200 can run application programs and interact with the user based on the application programs. At least one application program can be run in the application layer, which can be a window (Window) program, a system setting program, or a clock program, etc. provided by the operating system; or an application program developed by a third-party developer. In specific implementation, the application programs in the application layer are not limited to the above examples.

[0099] ​The framework layer provides application programming interfaces (APIs) and a programming framework for applications. The application framework layer includes predefined functions. It acts as a central processing unit, determining the actions taken by applications within the application layer. Through the API, applications can access system resources and obtain system services during execution.

[0100] like Figure 4 As shown, in some embodiments, the application framework layer includes a view system, managers, and content providers. The view system designs and implements the application's interface and interactions, and includes lists, grids, text boxes, buttons, etc. Managers include at least one of the following modules: an Activity Manager for interacting with all running activities in the system; a Location Manager for providing system services or applications with access to system location services; a Package Manager for retrieving various information related to application packages currently installed on the device; a Notification Manager for controlling the display and clearing of notification messages; and a Window Manager for managing icons, windows, toolbars, wallpapers, and desktop widgets on the user interface.

[0101] In some embodiments, the Activity Manager manages the lifecycle of individual applications and common navigation and back functions, such as controlling application exit, opening, and back actions. The Window Manager manages all window programs, such as obtaining the screen size, determining if a status bar is present, locking the screen, capturing the screen, and controlling changes to the display window, such as shrinking the display window, shaking the display, or distorting the display.

[0102] In some embodiments, the system runtime library layer can provide support for the framework layer. When the framework layer is used, the operating system runs the instruction library contained in the system runtime library layer, such as the C / C++ instruction library, to implement the functions to be performed by the framework layer.

[0103] In some embodiments, the kernel layer is a functional layer situated between the hardware and software of the display device 200. The kernel layer can implement functions such as hardware abstraction, multitasking, and memory management. For example, ... Figure 4As shown, the kernel layer can be configured with hardware drivers, and the kernel layer includes at least one of the following drivers: an audio driver, a display driver, a Bluetooth driver, a camera driver, a WIFI driver, a USB driver, an HDMI driver, a sensor driver (such as a fingerprint sensor, a temperature sensor, a pressure sensor, etc.), and a power supply driver, etc.

[0104] It should be noted that the above examples are only a simple division of the operating system functions, and do not constitute a limitation on the specific operating system form of the display device 200 in some embodiments. Depending on the function of the display device, the type of operating system, and other factors, the number and specific types of levels included in the operating system can take other forms.

[0105] With the development of devices (such as smart TVs), some devices integrate screenshot (such as circle selection screenshot, full screen screenshot, etc.) functions to meet the user's diversified information query needs for the interface displayed by the device. For example, the device can take a screenshot of part of the display content in the displayed interface, and search based on the part of the image to display search results for the part of the image to the user.

[0106] However, since the video picture is real-time changing, the processing logic for circle selection screenshot for a single frame picture in the related art will reduce the accuracy of the circle selection screenshot when the video picture is circle selected and screenshot.

[0107] Based on this, in some embodiments, a circle selection screenshot method is provided. The circle selection screenshot method can be implemented by a display device and a mobile terminal (such as a remote controller) in cooperation. In an optional implementation, taking the case where the circle selection screenshot method is applied to the controller in the display device as an example, the display device includes a display and a controller; the display and the controller are in communication connection, which can be wired communication connection or wireless communication connection. For example, Figure 5 As shown, the specific steps include the following steps:

[0108] S510, in response to the received preset pressing instruction, the first image corresponding to the display picture is stored during the playing of the video stream.

[0109] The preset pressing instruction represents that a preset button in the pointing remote controller is pressed. The preset button can be a screenshot control in the pointing remote controller. Further, the screenshot control is a control capable of starting a screenshot function in the display device. The pointing remote controller is a remote controller integrated with a pointing device. When the pointing device is started, the position information of the pointing remote controller can be acquired in real time, the pointing parameter is generated based on the position information of the pointing remote controller, and the pointing parameter is sent to the display device in the form of an electric signal by using an infrared sending / bluetooth sending mode based on a preset parameter sending frequency. After receiving the pointing parameter, the display device can convert the electric signal into a digital signal and display the position indicated by the pointing parameter in the form of a cursor on a current interface displayed by the display device.

[0110] The first image is a picture displayed by the display when the user performs the circle selection operation.

[0111] Optionally, when the user has a circle selection screenshot requirement, a preset pressing instruction can be sent to the display device by pressing a preset button in the pointing remote controller. Correspondingly, in an optional embodiment, after the display device receives the preset pressing instruction, the display device can start a screenshot application, and the screenshot application can call a screenshot function provided by a system of the display device to perform a screenshot on a current interface displayed by the display to obtain the first image. Then, the first image can be cached in a local space.

[0112] It is worth noting that because there is a certain time delay between the user's screenshot requirement time (the time when the user presses the pointing remote controller) and the time when the display device performs the screenshot, the first image obtained by the display device can be inconsistent with the picture displayed by the display when the user performs the circle selection operation. Based on this, in another optional embodiment, during the playing of the video stream, the played historical images (i.e., the interfaces displayed by the display) can be cached in a local space. After the display device receives the preset pressing instruction, the actual screenshot requirement time of the user can be determined based on the current time and a preset image acquisition time delay. Then, the first image can be obtained from the local cache based on the screenshot requirement time.

[0113] The determination manner of the image acquisition time delay can be that, for each user associated with the display device, behavior data of the user when performing the circle selection and screenshot can be collected in advance (which can include action image data of the user pressing a preset button), then the image acquisition time delay corresponding to the user is analyzed according to the behavior data of the user (for example, a time difference between a time when the user presses the preset button and a time when the preset pressing instruction is sent by pointing to the remote controller is taken as the image acquisition time delay), and the image acquisition time delay of the user is stored in the local space in association with the identity information. Further, in the case that any user sends a preset pressing instruction by pointing to the remote controller is detected, the face image of the user can be collected by an image collection device (for example, a camera) to confirm the identity information of the user, and then the image acquisition time delay corresponding to the user is obtained in the local cache based on the identity information of the user.

[0114] For example, in the case that the image acquisition time delay of the user A is 1 second, if the preset pressing instruction sent by pointing to the remote controller is received at the current time, it is determined that the actual screenshot demand time of the user is 1 second before the current time, at this time, the historical image corresponding to the time 1 second before the current time stored in the cache can be taken as the first image.

[0115] S520, in the process of receiving the pointing parameter sent by the pointing remote controller and generating the trajectory line according to the received pointing parameter, the second image corresponding to the display screen of the display is stored.

[0116] The pointing parameter is used to represent the pointing condition of the current interface displayed by the display device by pointing to the remote controller, that is, the pointing parameter changes with the movement of the pointing remote controller, and can include but is not limited to the position information of the pointing remote controller. In some embodiments, the position information contained in the pointing parameter is the position information of the display device in the geodetic coordinate system, therefore, in order to more accurately present the position indicated by the pointing parameter in the current interface, the display device can take the lower left corner of the current interface as the origin, and according to the resolution of the display device (mostly 1920*1080), the position information in the geodetic coordinate system is converted into the horizontal and vertical coordinate information in the screen coordinate system of the display device. In other embodiments, the pointing parameter includes the horizontal and vertical coordinate information in the screen coordinate system of the display device after the coordinate conversion of the pointing remote controller. The display device can directly determine the display position of the cursor according to the received horizontal and vertical coordinate information.

[0117] The trajectory line is the image circle selection trajectory drawn by the user by pointing to the remote controller. The second image is the image corresponding to the screen displayed by the display in the process of generating the trajectory line, and further, the second image can be one frame or multiple frames of images.

[0118] Optionally, after receiving the pointing parameter sent by the pointing remote controller, the trajectory line for the circle selection can be generated by processing the pointing parameter. In an optional embodiment, when the position information contained in the pointing parameter is the position information of the pointing remote controller relative to the display device in the geodetic coordinate system, the display device can convert the position information in the pointing parameter received from the pointing remote controller into the coordinate information of the corresponding trajectory point in the screen coordinate system on the display device by using the preset coordinate conversion manner, and store the coordinate information of the trajectory point in the trajectory array.

[0119] In another optional embodiment, when the pointing parameter includes the horizontal and vertical coordinate information of the pointing remote controller in the screen coordinate system on the display device after the coordinate conversion, the display device can directly use the position information in the pointing parameter received from the pointing remote controller as the coordinate information of the trajectory point, and store the coordinate information of the trajectory point in the trajectory array.

[0120] Then, based on the sending order of each trajectory point in the trajectory array and the coordinate information corresponding to each trajectory point, the coordinate change between each trajectory point is determined. Then, based on the coordinate change between each trajectory point, the trajectory line is generated.

[0121] At the same time, in order to ensure the reliability of the image circle selection, during the generation of the trajectory line, the second image can be determined according to the image displayed by the display, and the second image can be stored. For example, each frame of image displayed by the display can be used as the second image; or, part of the frame images can be extracted from the image displayed by the display as the second image. For example, the image in the image displayed by the display which is greatly different from the adjacent frame image can be used as the second image.

[0122] S530, according to the trajectory line, determining the circle selection region, and according to the circle selection region, taking screenshots of the first image and the second image to obtain a set of region images.

[0123] The so-called circle selection region is used to represent the region to be selected by the user. The so-called region image is the image corresponding to the circle selection region, for example, the image corresponding to the circle selection region in the first image. Further, the set of region images is a set containing each region image.

[0124] Optionally, the circled selection region can be determined according to the horizontal and vertical coordinate parameters of the trajectory points in the trajectory array associated with the trajectory line. In order to ensure the normativity of the circled selection region, a preset shape region can be pre-configured. Further, the preset shape region can be determined according to the horizontal and vertical coordinate parameters of the trajectory points in the trajectory array associated with the trajectory line, and the preset shape region can be taken as the circled selection region. The region circled by the preset shape region corresponds to the region represented by the trajectory line, and can be a circled selection region containing all the trajectory lines. Optionally, the preset shape region can be a rectangular region (including a square region, a rectangular region), a circular region, an elliptical region, a triangular region, etc.

[0125] It is worth noting that only one preset shape region can be configured, or multiple candidate preset shape regions can be configured at the same time. Before determining the preset shape region corresponding to the trajectory line, the most suitable target preset shape region for the trajectory line can be selected from the multiple candidate preset shape regions based on the circled selection trend of the trajectory line, and then the preset shape region corresponding to the region represented by the trajectory line can be determined.

[0126] In addition, no matter what shape of trajectory line is circled by the user, a preset shape region containing all the trajectory lines can be determined based on the trajectory line. Further, since the shape of the trajectory line and whether it is regular are not limited, the above steps can reduce the difficulty of the user to circle the screenshot, improve the user experience, and ensure the smooth progress of the subsequent image search.

[0127] Optionally, after the circled selection region is determined, the regions corresponding to the circled selection region in the first image and the second image can be screenshot in sequence according to the image acquisition order to obtain the region images corresponding to the images. Then, the region images corresponding to the images can be sorted and stored in sequence according to the image acquisition order, and the region image set can be obtained. The region corresponding to the circled selection region in the first image can be screenshot according to the coordinate information of the circled selection region to obtain the region image corresponding to the first image. It should be noted that the screenshot process of the second image is the same as that of the first image, and will not be described here.

[0128] In some embodiments, in the case where the preset shape region is a rectangular region, in order to ensure the adaptability between the region image and the display page, the image framed by the rectangular region in the first image and the second image can be screenshot to obtain an original image, and then the original image can be optimized based on the length-width ratio of the display page to obtain a region image. For example, in the case where the length-width ratio of the display page is 16:9, the original image can be adjusted to a region image with a length-width ratio of 16:9 without modifying the image length, or the original image can be adjusted to a region image with a length-width ratio of 16:9 without modifying the image width.

[0129] S540, performing image recognition according to the region image set to display the person information of the recognized person.

[0130] The person includes a human and an object, and can be an animal, a plant, an actor, a daily product, and other objects that can be displayed in an image form, etc. Further, the person information is information corresponding to the person, for example, a brief introduction of the person, etc.

[0131] In an optional embodiment, the image acquisition path corresponding to the region image set can be generated according to the device identification information of the display device, and then the region image set and the image acquisition path of the region image set are uploaded to the server. In order to improve the efficiency of image uploading, the region image set can be compressed before image uploading, and the compressed image set obtained after compression and the image acquisition path of the region image set are uploaded to the server.

[0132] The image recognition system is integrated in the server, and then the server can call the image recognition system to sequentially recognize each region image after obtaining the region image set, and feed back the recognition result as the person information of the person in the region image set to the display device based on the image acquisition path. For example, the image recognition system can perform segmentation processing on the region image based on the image features in the region image to obtain the person included in each sub-image, and then for each person included in each sub-image, the person can be used as index information to perform consistency matching on the person in the standard person included in the pre-determined image library, and the person information corresponding to the standard person with the highest matching degree is used as the person information of the person. After obtaining the person information of the person included in each sub-image, the person information is fed back to the display device.

[0133] In another optional embodiment, each region image in the region image set can be sequentially searched in the locally stored person information to obtain the person information of the person in each region image.

[0134] Optionally, after obtaining the person information, the display device can determine an information display layer on the current display interface, and display the person information corresponding to each region image on the preset display position in the information display layer. For example, according to the order of each region image, the person information corresponding to each region image can be sequentially displayed on the preset display position in the information display layer at a preset display time interval. Alternatively, the person information corresponding to each region image can be simultaneously displayed on the preset display position in the information display layer according to the order of each region image. For example, referring to Figure 6The information display page shown, after identifying each region image, the actor M and the actor N are identified in turn, at this time, the information display floating layer can directly display the information of the actor M and the information of the actor N corresponding to the person.

[0135] The technical solution, in the process of playing the video stream, on the one hand, by storing the first image corresponding to the display screen in the case of receiving the preset pressing instruction, and storing the second image corresponding to the display screen in the process of generating the trajectory line according to the received pointing parameter; then, according to the region selected by the trajectory line, the first image and the second image are captured to obtain a region image set, which can simultaneously select and capture the multiple frames of images displayed by the display in the selection operation, ensuring the integrity and accuracy of the selection operation; on the other hand, by simultaneously identifying each region image in the region image set and displaying the identified information, the comprehensiveness of the displayed information is improved, and the displayed information is more in line with the actual needs of the user.

[0136] Based on the above embodiment, in the embodiment of the application, the process of obtaining the second image is refined, and an optional implementation manner is that, in the process of receiving the pointing parameter sent by the pointing remote controller and generating the trajectory line according to the received pointing parameter, the video screen displayed by the display is captured according to the set time interval to obtain the second image, and the second image is stored. The so-called set time interval is the time interval of the capturing operation set in advance.

[0137] Optionally, the set time interval can be determined according to the playing speed of the video stream. For example, in the case of fast playing speed (for example, 2 times or 3 times playing speed), the set time interval can be adaptively shortened, and in the case of slow playing speed (for example, 0.5 times playing speed), the set time interval can be adaptively increased.

[0138] Optionally, in the process of receiving the pointing parameter sent by the pointing remote controller and generating the trajectory line according to the received pointing parameter, the video screen displayed by the display can be directly captured according to the set time interval adapted to the playing speed of the video stream to obtain the second image. Then, the captured second image is stored. For example, in the case of set time interval of 1S, if the process of generating the trajectory line according to the received pointing parameter is 10S, 10 frames of images can be captured as the second image.

[0139] The technical solution, by setting the time interval, the video screen displayed by the display is captured to obtain the second image, which provides a simple, fast and optional way to obtain the second image.

[0140] Another optional implementation is that, in the process of receiving the pointing parameter sent by the pointing remote controller and generating the trajectory line according to the received pointing parameter, the played video frame is detected, and in the case that the video frame is detected as a key frame, the key frame is stored as the second image. The so-called key frame is a video frame with a large degree of change in the video stream. For example, a video frame with a large change can be taken as a key frame.

[0141] Optionally, in the process of receiving the pointing parameter sent by the pointing remote controller and generating the trajectory line according to the received pointing parameter, for each video frame in the video stream, the video frame can be detected based on the degree of change between the previous video frame and the video frame, and in the case that the video frame is detected as a key frame, the key frame is stored as the second image.

[0142] For example, the process of detecting whether the video frame is a key frame can be that the screenshot application controls the home application to detect each frame of the played video stream, and in the case that the video frame is detected as a key frame, the key frame is stored as the second image; or, the home application in the display device can send a screenshot instruction to the screenshot application in the case that the identified video frame is a key frame, the screenshot application can take a screenshot of the video frame displayed by the display, and the screenshot is stored as the second image. The home application is the application in the display device that is currently in visual interaction with the user.

[0143] The above technical solution directly takes the key frame in the video stream as the second image, which on the one hand provides an optional way to obtain the second image, and on the other hand, since the key frame is an important video frame in the video stream and contains more rich character information, the reliability of the second image is ensured, so that the final character information is more rich and accurate.

[0144] Further, another optional implementation can be provided, specifically, a monitoring model capable of monitoring the picture richness is pre-configured in the display device. Correspondingly, in the process of receiving the pointing parameter sent by the pointing remote controller and generating the trajectory line according to the received pointing parameter, the detection model can be called to detect the picture richness of the played video frame, and the video frame with a picture richness greater than a preset richness threshold is stored as the second image.

[0145] On the basis of the above embodiments, in the embodiments of the present application, the process of displaying the character information is refined, specifically, the character information of the recognized character and the image download information are displayed.

[0146] The image download information includes at least one of the region image set, the target video, and the animation generated based on the region image set.

[0147] The target video is obtained by cropping the screen video according to the selected region. The screen video is obtained by screen recording the display interface during the process of generating the track line. It can be understood that, in order to intuitively understand the whole process of the selected screenshot by the user, the screen recording function can be started to record the display interface of the display in response to the received preset pressing instruction. The screen recording is maintained during the process of receiving the pointing parameter sent by the remote controller and generating the track line according to the received pointing parameter, so as to obtain the screen video. Then, the screen video can be cropped according to the coordinate information corresponding to the selected region, so as to obtain the target video.

[0148] Optionally, after obtaining the region image set, each region image in the region image set can be processed based on a preset frame length to obtain an animation corresponding to the region image set.

[0149] Optionally, while performing image recognition on the region image set, the screen video obtained by screen recording the display interface can be acquired, the target video can be obtained by cropping the screen video according to the selected region, and the image animation can be obtained by processing each region image in the region image set. Further, after recognizing the person information of the person, the person information of the person, the image to be downloaded, and the corresponding download entrance can be displayed in the information display layer on the current display interface.

[0150] For example, after obtaining the region image set, the target video, and the animation generated based on the region image set, each type of image to be downloaded can be uploaded to a corresponding type of storage location in the server. After storage is completed, the server will feed back the corresponding storage location information to the display device. The display device can generate a corresponding download entrance according to the storage location information corresponding to each type of image to be downloaded, and display each type of image to be downloaded and the corresponding download entrance.

[0151] For example, in the case where the image to be downloaded includes the region image set, the target video, and the animation generated based on the region image set at the same time. Referring to the information display page shown in Figure 7 The region image set A and the download entrance A of the region image set A, the target video B and the download entrance B' of the target video B, and the animation C and the download entrance C' of the animation C can be displayed below the person information 1.

[0152] The technical scheme has the following beneficial effects. On the one hand, by introducing the download entrance corresponding to the to-be-downloaded image, the user can not only obtain the information of the person in the image when performing the circle selection and screenshot, but also download the to-be-downloaded image in the screenshot, thereby enriching the circle selection and screenshot function. On the other hand, the to-be-downloaded image includes a video related to the entire circle selection and screenshot process, that is, the target video, so that the user can intuitively understand the entire circle selection and screenshot process, and the user experience of the circle selection and screenshot is further improved.

[0153] On the basis of the above embodiment, in the embodiment of the present application, the display device can upload the set of region images to the server, and the image recognition system in the server sequentially recognizes each region image to obtain the information of the person corresponding to each region image. Then, the information of each person is de-duplicated, and the de-duplicated information of each person is fed back to the display device. That is, the information of the recognized person displayed by the display device is the information of the person recognized by the server from each region image in the set of region images in sequence, and the information of the person recognized from each region image is de-duplicated and fed back.

[0154] In an optional embodiment, in the case where the same person is recognized from multiple region images, the server can only retain the region image in which the person is first recognized according to the recognition order for each region image in which the same person is recognized. For example, in the case where the actor P is recognized from both region images S1 and S2, if the server first recognizes the actor P from the region image S1, only the image information of the actor P recognized from the region image S1 is retained, and the display device is fed back the image information of the actor P in the region image S1 and the information of the actor P. If the server first recognizes the actor P from the region image S2, only the image information of the actor P recognized from the region image S2 is retained, and the display device is fed back the image information of the actor P in the region image S2 and the information of the actor P.

[0155] The technical scheme has the following beneficial effects. On the one hand, the technical scheme can ensure the simplicity of the information of the person displayed to the user. On the other hand, since the repeated information of the person is deleted, the storage space of the display device can be reduced.

[0156] Further, on the basis of the above embodiment, in the embodiment of the present application, when the server recognizes the person corresponding to the region image, the confidence of the recognized person can be determined at the same time. The confidence is used to represent the similarity between the person in the region image and the corresponding standard person template.

[0157] Correspondingly, in a case where it is determined that the multiple region images recognize the same person, the person information can be processed according to the confidence of each region image that recognizes the same person information. Specifically, in a case where the confidence corresponding to each region image that recognizes the same person is the same, the repeated person information can be directly processed by de-duplication; in a case where the confidence corresponding to each region image that recognizes the same person is different, only the person information recognized by the region image with the highest recognition confidence can be retained. That is, the person information of the recognized person displayed by the display device is fed back by the server after de-duplication of the person information of the same person with the same confidence, and elimination of the person information of the same person with different confidence except for the highest confidence.

[0158] For example, in a case where the actor Q is recognized in the region images S3, S4 and S5, if the recognition confidence of the actor Q in the region image S3 is 80%, the recognition confidence of the actor Q in the region image S4 is 40%, and the recognition confidence of the actor Q in the region image S5 is 70%, only the image information of the actor Q recognized in the region image S3 is retained, and the display device is fed back the image information of the actor Q in the region image S3 and the person information of the actor Q.

[0159] The above technical solution processes the person information by de-duplication and elimination based on the confidence of the recognized person, which can ensure the reliability and simplicity of the person information display, and also reduces the occupation of the storage space of the display device by deleting related person information.

[0160] On the basis of the above embodiment, in the embodiment of the present application, the generation process of the trajectory line is refined, specifically, the generated trajectory line is displayed on the upper layer of the video picture displayed by the display. The area in the upper layer except the trajectory line is a transparent area.

[0161] It can be understood that, in order to avoid the modification of the video picture by the circle selection operation, in a case where the above circle selection and screenshot processing is performed, the display device will overlay an upper layer on the video picture, and perform the trajectory line generation operation in the upper layer, so that the user can directly see the circle selection result while playing the video picture, and the video picture is not modified. In addition, in a case where the user makes a mistake in circle selection, the trajectory line on the upper layer can be cleared to re-perform the circle selection and screenshot.

[0162] Exemplarily, in a case where it is detected that the circle selection screenshot processing is performed, a brush can be initialized, and when a new track point is received, the brush is used to draw position information corresponding to the track point on the upper layer, so as to generate a track line; in a case where it is detected that the circle selection stopping event, the current brush is destroyed, that is, the circle selection operation at this time is ended.

[0163] The technical solution described above, by covering a transparent layer on the video picture when the circle selection screenshot processing is performed, can display the drawn track line without affecting the video playing.

[0164] Further, on the basis of the embodiments described above, in the embodiments of the present application, the process of information display is further refined. Specifically, the target image and the search indication information are displayed in the preset display area of the upper layer of the video picture displayed by the display.

[0165] The so-called preset display area is an area for displaying the screen image and the search indication information in the image search process. For example, the upper left corner of the upper layer is taken as the preset display area. Further, the area of the preset display area is smaller than the display area of the video picture, and the area of the upper layer other than the preset display area is a transparent area, that is, the video picture played in the area other than the preset display area can be normally watched. The so-called search indication information is used to inform the user that the screen image is being searched, for example, it can be the text information "circle selection content recognition" or "the following results are recognized for you". The target image is any region image in the region image set or the region image set.

[0166] Optionally, after the region image set is acquired, in order not to affect the display of the video picture, any region image in the region image set or the region image set and the search indication information generated when the region image is recognized can be displayed in the preset display area of the upper layer of the video picture.

[0167] For example, referring to the image search page schematic diagram shown in Figure 8 In a case where the upper left corner of the upper layer of the video picture is taken as the preset display area, any region image 2 in the region image set and the search indication information 3 "circle selection content recognition..." can be displayed in the upper left corner of the upper layer of the video picture.

[0168] It is worth noting that, in addition to being able to preset a fixed area as the preset display area, in order to ensure the rationality of the preset display area, the preset display area can also be determined based on position information of a preset element in the current video picture which cannot be blocked. For example, the preset display area can be selected from other areas outside the area where the preset element which cannot be blocked is located.

[0169] Optionally, after the person information of the person in the identified region image is identified, the person information of the person can be displayed in a preset display region in the upper layer of the video picture. For example, referring to the information display schematic diagram shown in Figure 9 After the upper left corner of the upper layer of the video picture is taken as the preset display region and the person information of the person in the identified region image is identified, the person information 4 of the person and the search instruction information 5" for you to identify the following results" can be displayed in the case where the upper left corner of the upper layer of the video picture is taken as the preset display region.

[0170] The technical solution described above can enable the user to intuitively understand the image search progress and improve the user experience by displaying the target image and the search instruction information in the preset display region of the upper layer of the video picture.

[0171] On the basis of the above-described embodiments, in the embodiments of the present application, in the case where the selected region is a rectangular region, the process of determining the selected region is refined, as shown in Figure 10 The process specifically includes the following steps:

[0172] S1010, the minimum horizontal coordinate value, the maximum horizontal coordinate value, the minimum vertical coordinate value and the maximum vertical coordinate value are selected from the coordinate data of the track points on the track line.

[0173] The minimum horizontal coordinate value is the minimum horizontal coordinate value in the track line, the maximum horizontal coordinate value is the maximum horizontal coordinate value in the track line, the minimum vertical coordinate value is the minimum vertical coordinate value in the track line, and the maximum vertical coordinate value is the maximum vertical coordinate value in the track line.

[0174] It can be understood that in actual application, there can be a problem that the selected length in a certain direction is short due to non-standard selection by the user, which can result in a small selected region and thus cannot guarantee the accuracy of the image search result.

[0175] Therefore, in order to guarantee the rationality of the determination of the selected region, the width value of the track line in the width direction of the screen and the length value of the track line in the length direction of the screen are compared with the first threshold value respectively, and in the case where the width value is greater than or equal to the first threshold value or the length value is greater than or equal to the first threshold value, the track line is determined to be valid, and in this case, the minimum horizontal coordinate value, the maximum horizontal coordinate value, the minimum vertical coordinate value and the maximum vertical coordinate value are selected from the coordinate data of the track points on the track line.

[0176] The width value is the maximum horizontal coordinate difference between the horizontal coordinate differences of the trajectory points on the trajectory line, and the length value is the maximum vertical coordinate difference between the vertical coordinate differences of the trajectory points on the trajectory line. For example, in the case where the coordinate unit in the current interface is a pixel, for the horizontal coordinates x1, x2 and x3, the difference between x1 and x2 is 5 pixels, the difference between x1 and x3 is 10 pixels, and the difference between x2 and x3 is 5 pixels. At this time, the width value is 10 pixels. For the vertical coordinates y1, y2 and y3, the difference between y1 and y2 is 4 pixels, the difference between y1 and y3 is 10 pixels, and the difference between y2 and y3 is 6 pixels. At this time, the length value is 10 pixels. The first threshold value is a value used to determine whether the trajectory line is valid. For example, the first threshold value can be 100 pixels. In some embodiments, the first threshold value can be set based on the experience of a person skilled in the art, or can be determined based on a large amount of experimental data, and no limitation is made to this.

[0177] In addition, in the case where the width value and the length value are both less than the first threshold value, a screenshot failure information is output. The screenshot failure information is prompt information displayed by the display device when the screenshot fails, and can include but is not limited to information such as the reason for the screenshot failure.

[0178] Optionally, in the case where the width value and the length value of the trajectory line are both less than the first threshold value, it is proved that the area selected by the user is small, at this time the selected area is invalid, and the display device can determine a floating layer on the target image and output the screenshot failure information in the form of text in the floating layer. For example, the text “The selected area is too small, please select again” can be output in the floating layer.

[0179] S1020, the horizontal and vertical coordinates of the at least two vertexes are determined by combining the minimum horizontal coordinate, the maximum horizontal coordinate, the minimum vertical coordinate and the maximum vertical coordinate.

[0180] The rectangular area represented by the at least two vertexes is the selected area.

[0181] Optionally, after the minimum horizontal coordinate, the maximum horizontal coordinate, the minimum vertical coordinate and the maximum vertical coordinate are combined horizontally and vertically, the at least two vertexes can be obtained; then, the rectangular area is determined based on the obtained vertexes. For example, the vertexes (X-min, Y-min), (X-min, Y-max), (X-max, Y-max) and (X-max, Y-max) can be determined based on the minimum horizontal coordinate X-min, the maximum horizontal coordinate X-max, the minimum vertical coordinate Y-min and the maximum vertical coordinate Y-max, so as to determine the rectangular area based on the above vertexes.

[0182] It can be understood that, when only two vertex coordinates are determined, the two vertex coordinates must be diagonal coordinates. For example, when only two vertex coordinates are determined based on the minimum horizontal coordinate X-min, the maximum horizontal coordinate X-max, the minimum vertical coordinate Y-min, and the maximum vertical coordinate Y-max, the two vertex coordinates can be (X-min, Y-min) and (X-max, Y-max), or (X-min, Y-max) and (X-max, Y-max).

[0183] For example, referring to the schematic diagram of determining a rectangular region corresponding to a regular trajectory line shown in FIG. 6, when the width value and the length value of the trajectory line 6 are both greater than the first threshold value, the coordinate points (X-min, Y-min) and (X-max, Y-max) can be directly determined based on the minimum horizontal coordinate X-min, the maximum horizontal coordinate X-max, the minimum vertical coordinate Y-min, and the maximum vertical coordinate Y-max in the trajectory line 6, so as to determine the rectangular region 7 based on the above coordinate points. Figure 11 It can be understood that, when the trajectory line is an irregular curve intersecting with each other, the rectangular region can also be determined based on the extreme values of the horizontal and vertical coordinates in the trajectory points. For example, referring to the schematic diagram of determining a rectangular region corresponding to an irregular trajectory line shown in FIG. 8, when the width value and the length value of the trajectory line 8 are both greater than the first threshold value, the coordinate points (X-min, Y-min) and (X-max, Y-max) can be directly determined based on the minimum horizontal coordinate X-min, the maximum horizontal coordinate X-max, the minimum vertical coordinate Y-min, and the maximum vertical coordinate Y-max in the trajectory line 8, so as to determine the rectangular region 9 based on the above coordinate points.

[0184] Figure 12 It is worth noting that, in order to generate a valid rectangular region when the trajectory line is invalid (any one of the length value and the width value is invalid), the target value can be calculated based on the valid value of the length value and the width value, and the screen aspect ratio of the display device, so as to optimize the rectangular region. For example, when the length value of the trajectory line is invalid, the optimized length value can be calculated based on the width value and the screen aspect ratio. For example, when the length value of the trajectory line is 10 pixels, the width value is 90 pixels, and the screen aspect ratio is 16:9, the optimized length value can be calculated as 160 pixels. When the width value of the trajectory line is invalid, the optimized width value can be calculated based on the length value and the screen aspect ratio. For example, when the width value of the trajectory line is 10 pixels, the length value is 160 pixels, and the screen aspect ratio is 16:9, the optimized width value can be calculated as 90 pixels.

[0185] It is worth noting that, in order to generate a valid rectangular region when the trajectory line is invalid (any one of the length value and the width value is invalid), the target value can be calculated based on the valid value of the length value and the width value, and the screen aspect ratio of the display device, so as to optimize the rectangular region. For example, when the length value of the trajectory line is invalid, the optimized length value can be calculated based on the width value and the screen aspect ratio. For example, when the length value of the trajectory line is 10 pixels, the width value is 90 pixels, and the screen aspect ratio is 16:9, the optimized length value can be calculated as 160 pixels. When the width value of the trajectory line is invalid, the optimized width value can be calculated based on the length value and the screen aspect ratio. For example, when the width value of the trajectory line is 10 pixels, the length value is 160 pixels, and the screen aspect ratio is 16:9, the optimized width value can be calculated as 90 pixels.

[0186] ​Then, according to the circle selection direction of the trajectory line, a corresponding expansion direction is determined, and the circle selection region is expanded according to the corresponding expansion direction to obtain an expanded circle selection region. In the case of a straight trajectory line, the expansion direction cannot be determined, at which time, based on the image richness on both sides of the trajectory line, the side with higher image richness is taken as the expansion direction.

[0187] For example, in the case where the width value is valid and the length value is invalid, an initial circle selection region can be determined based on the current width value and length value; then, based on the circle selection direction of the trajectory line, the length expansion direction of the initial circle selection region is determined, and the length of the initial circle selection region is expanded to a target value in the length expansion direction, thereby obtaining an expanded circle selection region. Alternatively, in the case where the length value is valid and the width value is invalid, an initial circle selection region can be determined based on the current width value and length value; then, based on the circle selection direction of the trajectory line, the width expansion direction of the initial circle selection region is determined, and the width of the initial circle selection region is expanded to a target value in the width expansion direction, thereby obtaining an expanded circle selection region.

[0188] Optionally, after the expanded circle selection region is determined, the minimum horizontal coordinate, the maximum horizontal coordinate, the minimum vertical coordinate, and the maximum vertical coordinate in the expanded circle selection region can be combined horizontally and vertically to obtain at least two vertex coordinates.

[0189] For example, referring to a rectangular region determination schematic diagram corresponding to a trajectory line with invalid width as shown in Figure 13 For example, referring to a rectangular region determination schematic diagram corresponding to a trajectory line with invalid width as shown in

[0190] The above technical solution determines the vertex coordinates of the circle selection region based on the minimum horizontal coordinate, the maximum horizontal coordinate, the minimum vertical coordinate, and the maximum vertical coordinate in the coordinate data of the trajectory point, which provides an optional way for quickly determining the circle selection region while ensuring that the subsequently obtained region image can contain valid image information. In addition, the region determined based on the trajectory line is in the form of a rectangle, which is adapted to the way in which the display device presents the page, thereby making the intercepted region image more reasonable.

[0191] On the basis of the above-mentioned embodiments, in the embodiments of the present application, in order to ensure the rationality of the circled screenshot, the type of the screen displayed by the current interface can also be detected. Optionally, in the case where the current interface displays a video screen, in the process of generating the trajectory line according to the received pointing parameter, the second image corresponding to the display screen is stored, and then the first image and the second image are screenshot according to the determined circled region to obtain the set of regional images. In the case where the current interface displays a static screen, the first image is deleted, and the circled screenshot is performed on the static screen according to the determined circled region to obtain the regional image.

[0192] The so-called current interface is the current display interface of the display, which can display a video screen or a static screen. The so-called video screen is a screen in which the screen image changes in real time, for example, a video playing screen. The so-called static screen is a screen in which the screen image does not change, for example, a homepage or a query page.

[0193] In an optional manner, the type of the screen displayed by the current interface can be determined according to the state identifier of the multimedia player. When the state identifier indicates that the multimedia player is in a playing state, the current interface displays a video screen. When the state identifier indicates that the multimedia player is in a non-playing state, the current interface displays a static screen. For example, when the state identifier is 1, it is determined that the multimedia player is in a playing state, that is, the current interface displays a video screen. When the state identifier is 0, it is determined that the multimedia player is in a non-playing state, that is, the current interface displays a static screen.

[0194] In another optional manner, the type of the screen displayed by the current interface can be determined according to the application state of the homepage application. When the application state indicates that the homepage application is playing media content, the current interface displays a video screen. When the application state indicates that the homepage application is not playing media content, the current interface displays a static screen.

[0195] Optionally, in the case where the type of the screen displayed by the current interface is determined to be a video screen according to the state identifier of the multimedia player / the application state of the homepage application, the first image and the second image are circled and screenshot to obtain the set of regional images, with reference to the above-mentioned steps S510-S530.

[0196] In the case where the type of the screen displayed by the current interface is determined to be a static screen according to the state identifier of the multimedia player / the application state of the homepage application, in order to improve the resolution of the regional image, the first image saved in advance can be deleted, and the circled screenshot is directly performed on the current display interface to obtain the regional image. Then the regional image is uploaded to the server, the image recognition system in the server recognizes the regional image to obtain the information of the person corresponding to the regional image, and the information is fed back.

[0197] The technical solution above selects a corresponding circle selection screenshot mode according to the type (dynamic / static) of a displayed screen of a current interface, thereby not only expanding the use scenarios of the screenshot function of the display device, but also improving user experience.

[0198] In actual application, the display device can support multiple image processing functions, and different image processing functions correspond to different function triggering modes. Specifically, the image processing functions can include screen recording, full-screen screenshot, and circle selection screenshot. Therefore, in order to better distinguish the image processing function triggered by the user, the image processing function used by the user can be determined from the preset multiple functions based on the switch state of the pointing switch of the pointing remote controller.

[0199] Specifically, in response to a preset pressing instruction sent by the pointing remote controller, a state parameter representing the switch state of the pointing switch of the pointing remote controller can be acquired from the locally stored current device information. It should be noted that, in the case where the pointing switch is turned on, the pointing remote controller can send a pointing parameter to the display device regardless of whether the user presses the screenshot control or not. However, the pointing parameter at this time does not constitute a circle selection area, but is presented in the form of a moving cursor.

[0200] Optionally, in the case where the state parameter represents that the pointing switch is in the off state, the pointing remote controller does not send a pointing parameter, that is, the display device cannot execute the circle selection screenshot processing logic. At this time, the image processing mode triggered by the display device can be screen recording / full-screen screenshot. In the case where the state parameter represents that the pointing switch is in the on state, the pointing remote controller can send a pointing parameter. At this time, the image processing mode triggered by the display device can be circle selection screenshot.

[0201] In addition, the image processing function used by the user can also be determined from the preset multiple functions in combination with the duration for which the user presses the screenshot control.

[0202] Specifically, the triggering mode of the full-screen screenshot can be configured as pressing the screenshot control for a short time, that is, in the case where a stop screenshot instruction is listened to and the timing duration does not exceed a preset duration, the image processing mode is determined to be full-screen screenshot. The so-called full-screen screenshot is a screenshot of the whole current interface. The stop screenshot instruction can be an instruction sent when a preset control of the pointing remote controller is lifted.

[0203] The triggering manner of the circle selection screenshot can be configured as pressing the screenshot control for a long time, and the pointing switch is in the open state, that is, in a case where the timing duration exceeds the preset duration and the state parameter indicates that the pointing switch is in the open state, the image processing manner is determined to be the circle selection screenshot. For example, when the user presses the preset control, the pressing parameter is 100, and when the user releases the preset control, the pressing parameter is 85. That is, in a case where the received pressing parameter is 100, it is proved that the user continuously presses the preset control, and in a case where the received pressing parameter changes to 85, it is proved that the stop screenshot instruction is listened to.

[0204] Optionally, in a case where the timing duration exceeds the preset duration and the state parameter indicates that the pointing switch is in the closed state, the display device triggers the screen recording processing logic to record the current interface to obtain a screen recording result. Then, the screen recording result is stored locally for subsequent user processing such as screen recording and sending.

[0205] On the basis of the technical solutions of the above embodiments, a television device is taken as an example for description, wherein the television device includes a television system and a screenshot application. The television system is the whole machine system of the television device, and the screenshot function for the display interface is provided through cooperation of the television system and the screenshot application. Further, an optional embodiment is provided in a case where a video picture is displayed. Referring to FIG. 13, a circle selection screenshot process is shown, and interaction between devices in the circle selection screenshot process is shown. Figure 14

[0206] S1401, in a case where the pointing remote control detects a pressing operation of the user on the preset control, the pointing remote control sends a preset pressing instruction to the television system.

[0207] S1402, after receiving the preset pressing instruction, the television system sends a starting instruction to the screenshot application.

[0208] S1403, after the screenshot application is started, the screenshot application calls a screenshot method provided by the television system to perform screenshot on the display interface of the display to obtain a first image.

[0209] Specifically, the screenshot application sends a screenshot instruction to the television system, the television system performs screenshot on the display interface, and sends the screenshot application the first image.

[0210] S1404, in a case where the screenshot application detects that the pressing duration of the user on the preset control exceeds the preset duration, the screenshot application sends a state query instruction for the state of the pointing switch of the pointing remote control to the television system.

[0211] S1405, the television system feeds back a state parameter indicating the switch state of the pointing switch of the pointing remote control to the screenshot application.

[0212] ​S1406, when the state parameter table indicates that the pointing switch is in an open state and the state identifier of the multimedia player is in a playing state, the screenshot application receives the pointing parameter sent by the pointing remote controller, and generates a trajectory line according to the received pointing parameter. In the process of generating the trajectory line, the screenshot application takes screenshots of the video frames displayed on the display at a set time interval to obtain a second image.

[0213] Specifically, the screenshot application can also control the home application to detect the video frames being played, and store the key frame as the second image in a case where it is detected that the video frame is a key frame.

[0214] S1407, the screenshot application determines a rectangular region as a circled region corresponding to the pointing parameter according to the trajectory line, and sends a circled screenshot instruction containing the coordinate information of the circled region to the television system.

[0215] S1408, the television system takes screenshots of the first image and the second image according to the coordinate information of the rectangular region to obtain a region image set, and sends the region image set and a screenshot acquisition path of the region image set to the screenshot application.

[0216] S1409, the screenshot application uploads the region image set and the screenshot acquisition path to the image recognition system, and displays any region image in the region image set and search instruction information in a preset display region of an upper layer of the display interface.

[0217] S1410, the image recognition system performs image search according to the region image set to determine the person information of the person in the region image set, and performs deduplication processing on the person information of the person in the region image set.

[0218] S1411, the image recognition system feeds back the person information of the person after deduplication to the screenshot application based on the screenshot acquisition path.

[0219] S1412, the screenshot application displays the person information of the person in each region image after deduplication in the preset display region of the upper layer of the display interface.

[0220] Further, the specific processes of S1401-S1412 described above can be referred to the description of the method embodiments described above, which have similar implementation principles and technical effects, and will not be described here.

[0221] Based on the same inventive concept, some embodiments also provide a circled screenshot device for implementing the above-mentioned circled screenshot method. The implementation scheme of the problem solving provided by the device is similar to the implementation scheme described in the above method, so the specific limitations in one or more circled screenshot device embodiments provided below can be referred to the limitations of the circled screenshot method described above, and will not be described here.

[0222] In one exemplary embodiment, as shown in Figure 15 A circle selection screenshot device 1 is provided in a display device, comprising a first storage module 10, a second storage module 20, a screenshot module 30 and a character recognition module 40, wherein:

[0223] The first storage module 10 is configured to store a first image corresponding to a display screen during playing of a video stream in response to a received preset pressing instruction, wherein the preset pressing instruction represents that a preset button on a remote controller is pressed.

[0224] The second storage module 20 is configured to store a second image corresponding to the display screen during receiving of a pointing parameter sent by the remote controller and generating of a trajectory line according to the received pointing parameter.

[0225] The screenshot module 30 is configured to determine a circle selection area according to the trajectory line, and perform screenshot on the first image and the second image according to the circle selection area to obtain a region image set, wherein the pointing parameter changes with movement of the remote controller.

[0226] The character recognition module 40 is configured to perform image recognition according to the region image set to display character information of a recognized character.

[0227] In some optional embodiments, the second storage module 20 is specifically configured to:

[0228] During receiving of the pointing parameter sent by the remote controller and generating of the trajectory line according to the received pointing parameter, the second storage module 20 is configured to perform screenshot on a video screen displayed by the display at a set time interval to obtain the second image, and store the second image.

[0229] In some optional embodiments, the second storage module 20 is further configured to:

[0230] During receiving of the pointing parameter sent by the remote controller and generating of the trajectory line according to the received pointing parameter, the second storage module 20 is configured to detect a played video frame, and store a key frame as the second image in a case that the video frame is detected as the key frame.

[0231] In some optional embodiments, the character recognition module 40 is specifically configured to:

[0232] The character recognition module 40 is configured to display the character information of the recognized character and image download information, wherein the image download information comprises to-be-downloaded images and a download portal, and the to-be-downloaded images comprise at least one of the region image set, a target video and an animation generated based on the region image set; the target video is obtained by cropping a screen video according to the circle selection area, and the screen video is obtained by screen recording a display interface of the display during generating of the trajectory line.

[0233] In some optional embodiments, the character information of the recognized character is fed back after the server identifies each regional image in the regional image set and removes the character information of the recognized character in each regional image.

[0234] In some optional embodiments, the character information of the recognized character is fed back after the server removes the character information of the same character with the same confidence and removes the character information of the same character with different confidence except the highest confidence.

[0235] In some optional embodiments, the circled screenshot device 1 further comprises a trajectory line generation module, wherein the trajectory line generation module is specifically configured to:

[0236] display the generated trajectory line in an upper layer of the video image displayed on the display; wherein the area in the upper layer other than the trajectory line is a transparent area.

[0237] In some optional embodiments, the circled screenshot device 1 further comprises a display module, wherein the display module is specifically configured to:

[0238] display the target image and the search indication information in a preset display area of the upper layer of the video image displayed on the display; wherein the area of the preset display area is smaller than that of the video image, the area in the upper layer other than the preset display area is a transparent area, the search indication information is used to inform the user that the image search is being performed, and the target image is any regional image in the regional image set or the regional image set.

[0239] In some optional embodiments, the screenshot module 30 is specifically configured to:

[0240] select the minimum horizontal coordinate value, the maximum horizontal coordinate value, the minimum vertical coordinate value, and the maximum vertical coordinate value from the coordinate data of the trajectory points on the trajectory line;

[0241] combine the minimum horizontal coordinate value, the maximum horizontal coordinate value, the minimum vertical coordinate value, and the maximum vertical coordinate value horizontally and vertically to obtain at least two vertex coordinates; wherein the rectangular area represented by the at least two vertex coordinates is the circled area.

[0242] The above-mentioned various modules in the circled screenshot device can be realized by software, hardware, and combinations thereof, in whole or in part. The above-mentioned various modules can be embedded in or independent of the processor in the computer device in hardware form, or can be stored in the computer in software form.

[0243] In one embodiment, a computer readable storage medium is provided, which stores a computer program, and the computer program is executed by a processor to implement the steps in the above-mentioned various method embodiments.

[0244] In an embodiment, a computer program product is provided, including a computer program which, when executed by a processor, implements the steps of any of the above method embodiments.

[0245] It should be noted that the data (including but not limited to image data, etc.) involved in the present application are all data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data need to comply with relevant regulations.

[0246] It can be understood by those skilled in the art that all or part of the processes in the above-mentioned embodiments can be completed by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer readable storage medium, and when executed, can include the processes of the above-mentioned embodiments. Any reference to memory, database or other medium used in the embodiments provided by the present application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration but not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc. The database involved in the embodiments provided by the present application can include at least one of a relational database and a non-relational database. The non-relational database can include a distributed database based on a block chain, etc., without being limited thereto. The processor involved in the embodiments provided by the present application can be a general-purpose processor, a central processing unit, a graphics processing unit, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (AI) processor, etc., without being limited thereto.

[0247] Any technical features in the above embodiments can be combined, and for the sake of brevity, not all possible combinations are described above, however, any combination of these technical features is deemed to be within the scope of the present application.

[0248] The above embodiments only express several implementation manners of the present application, and the description is relatively specific and detailed, but it should not be understood as a limitation on the patent scope of the present application. It should be pointed out that, for ordinary skilled persons in the art, several modifications and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the appended claims.

Claims

1. A display device, characterized in that, include: Displays and controllers; The controller is configured as follows: During video playback, in response to a received preset press command, the first image corresponding to the screen displayed on the display is stored; wherein, the preset press command indicates that a preset button in the remote control is pressed; During the process of receiving pointing parameters sent by the pointing remote control and generating a trajectory line based on the received pointing parameters, the second image corresponding to the display screen is stored; wherein, the pointing parameters change with the movement of the pointing remote control; Based on the trajectory line, a selected area is determined, and based on the selected area, screenshots are taken of the first image and the second image to obtain a set of region images; Image recognition is performed based on the region image set to display the person information of the identified person.

2. The display device according to claim 1, characterized in that, When the controller receives pointing parameters sent by the remote control and generates a trajectory line based on the received pointing parameters, and stores the second image corresponding to the display screen, it is configured as follows: During the process of receiving pointing parameters sent by the remote control and generating a trajectory line based on the received pointing parameters, the video screen displayed on the display is captured at set time intervals to obtain a second image, and the second image is stored.

3. The display device according to claim 1, characterized in that, When the controller receives pointing parameters sent by the remote control and generates a trajectory line based on the received pointing parameters, and stores the second image corresponding to the display screen, it is configured as follows: During the process of receiving pointing parameters sent by the remote control and generating a trajectory line based on the received pointing parameters, the video frames being played are detected, and if the video frame is detected to be a keyframe, the keyframe is stored as a second image.

4. The display device according to claim 1, characterized in that, When the controller displays the information of the identified person, it is configured to: The system displays the person information of the identified person, as well as image download information; wherein, the image download information includes the image to be downloaded and the download entry; the image to be downloaded includes at least one of the region image set, the target video, and the animation generated based on the region image set; the target video is obtained by cropping the screen video according to the selected region, and the screen video is obtained by screen recording of the display interface of the monitor during the generation of the trajectory line.

5. The display device according to any one of claims 1-4, characterized in that, The person information of the identified person is fed back by the server successively identifying each region image in the region image set, and deduplicating the person information of the identified person in each region image.

6. The display device according to claim 5, characterized in that, The identified person information is generated by the server after deduplicating the information of the same person with the same confidence level, and removing the information of the same person with different confidence levels except for the one with the highest confidence level.

7. The display device according to claim 1, characterized in that, The controller, in the process of receiving pointing parameters sent by the remote controller and generating a trajectory line based on the received pointing parameters, is also configured to: The generated trajectory lines are displayed in a layer above the video image displayed on the monitor; wherein, the area outside the trajectory lines in the upper layer is a transparent area.

8. The display device according to claim 1, characterized in that, After obtaining the region image set, the controller is further configured to: The target image and search indication information are displayed in a preset display area of ​​the layer above the video screen on the display. The area of ​​the preset display area is smaller than the display area of ​​the video screen. The area outside the preset display area in the upper layer is a transparent area. The search indication information is used to inform the user that an image search is being performed. The target image is any image from the image set or the image set itself.

9. The display device according to claim 1, characterized in that, When the controller executes the step of determining the selected area based on the trajectory line, it is configured to: From the coordinate data of the trajectory points on the trajectory line, select the minimum, maximum, minimum, and maximum values ​​of the horizontal coordinate; The minimum and maximum values ​​of the horizontal coordinate, the minimum and maximum values ​​of the vertical coordinate are combined horizontally and vertically to obtain at least two vertex coordinates; wherein the rectangular area represented by the at least two vertex coordinates is the selected area.

10. A method for selecting and capturing images, characterized in that, Applied to display devices, including: During video playback, in response to a received preset press command, the first image corresponding to the screen displayed on the monitor is stored; wherein, the preset press command indicates that a preset button in the remote control is pressed; During the process of receiving pointing parameters sent by the pointing remote control and generating a trajectory line based on the received pointing parameters, the second image corresponding to the display screen is stored; wherein, the pointing parameters change with the movement of the pointing remote control; Based on the trajectory line, a selected area is determined, and based on the selected area, screenshots are taken of the first image and the second image to obtain a set of region images; Image recognition is performed based on the region image set to display the person information of the identified person.

Images