A display device and an image processing method

Abstract

The present application provides a display device and an image processing method. The method can calculate the ratio of the time taken to detect a frame of image to the time interval between obtaining two adjacent frames of images to obtain the number of queues; create cache queues according to the number of queues; then store the frames of images into the cache queues in sequence according to the order of obtaining the frames of images; then, for any cache queue, perform human key point recognition on the frames of images in the cache queue to obtain human key point images; and then render the human key point images and the frames of images in the cache queue in sequence according to the order of the cache queues; wherein the order of the cache queues is the storage order of the frames of images; thereby avoiding the problem of low picture display smoothness caused by frame drops during picture rendering, improving the matching degree between the human key point images and the frames of images, improving the accuracy of the human key point images used for rendering the frames of images, and avoiding deviation between the picture rendering result and the frames of images.

Description

Technical Field

[0001] This application relates to the field of computer technology, and particularly to a display device and an image processing method. Background Art

[0002] The number of frames transmitted per second (Frames Per Second, FPS) of a screen represents the number of screens played per second; the more screens played per second, the smoother the display effect of the screen. In a processing scenario, image processing includes rendering each frame of image. The rendering process for each frame of image generally includes the following two processing methods:

[0003] Synchronous rendering: Synchronously obtain each frame of image; for the i-th frame of image, perform image detection on the i-th frame of image according to a preset detection algorithm to obtain the i-th frame of human key point image; then render the i-th frame of image based on the i-th frame of human key point image; after rendering the i-th frame of image, obtain the (i + 1)-th frame of image; and so on, rendering each frame of image.

[0004] Asynchronous rendering: Asynchronously obtain each frame of image; for the i-th frame of image, perform image detection on the i-th frame of image according to a preset detection algorithm to obtain the i-th frame of human key point image; then render the current frame of image based on the i-th frame of human key point image; different from the synchronous rendering method, after obtaining the i-th frame of image, according to the time interval between obtaining two adjacent frames of images, obtain the (i + 1)-th frame of image, and display each frame of image according to the time interval between obtaining two adjacent frames of images; and so on, rendering each frame of image; the current frame of image is the currently displayed image frame.

[0005] In the above synchronous rendering method, if the detection time of image detection is long, the time interval between obtaining the i-th frame of image and obtaining the (i + 1)-th frame of image is large, that is, the time interval between obtaining two adjacent frames of images is large, resulting in a reduction in the number of screens played per second, causing stuttering in the rendering and display of the screen and low screen smoothness. In the above asynchronous rendering method, if the detection time of image detection is long, it makes the detection of human key point images slow, which will cause the (i + 1)-th frame of image to be rendered based on the i-th frame of human key point image; when rendering the screen, the matching degree between the human key point image and the image frame is low, that is, the accuracy of the human key point image used for rendering the current frame of image is low, resulting in a deviation between the screen rendering result and the image frame. Summary of the Invention

[0006] This application provides a display device and an image processing method to avoid stuttering during screen rendering, improve the smoothness of screen rendering and display, increase the matching degree between the human key point image and the image frame, improve the accuracy of the human key point image used for rendering the image frame, and avoid deviation between the screen rendering result and the image frame.

[0007] In a first aspect, the present application provides a display device, including:

[0008] A display configured to display an image frame;

[0009] A camera configured to acquire image frames at a preset interval; the preset interval represents the time interval between two adjacent frames acquired by the camera;

[0010] A controller configured to:

[0011] Calculate the ratio of the time taken to detect one frame of image to the time interval between acquiring two adjacent frames to obtain the number of queues;

[0012] Create a cache queue according to the number of queues; the cache queue is used to store image frames;

[0013] Store the image frames into the cache queue in sequence according to the order of acquiring the image frames;

[0014] For any cache queue, perform human key point recognition on the image frames in the cache queue to obtain human key point images;

[0015] Render the human key point images and image frames in the cache queue in sequence according to the order of the cache queue; the order of the cache queue is the storage order of the image frames.

[0016] In a second aspect, the present application further provides an image processing method, which is applied to a display device;

[0017] The method includes:

[0018] Calculate the ratio of the time taken to detect one frame of image to the time interval between acquiring two adjacent frames to obtain the number of queues;

[0019] Create a cache queue according to the number of queues; the cache queue is used to store image frames;

[0020] Store the image frames into the cache queue in sequence according to the order of acquiring the image frames;

[0021] For any cache queue, perform human key point recognition on the image frames in the cache queue to obtain human key point images;

[0022] Render the human key point images and image frames in the cache queue in sequence according to the order of the cache queue; the order of the cache queue is the storage order of the image frames.

[0023] As can be seen from the above technical solutions, before rendering the image frame, the number of queues is determined according to the ratio of the time taken to detect one frame of image to the time interval between obtaining two adjacent frames of images, and a cache queue is created based on the number of queues; if the number of queues is 3, then 3 cache queues are created.

[0024] Then, the obtained image frames are sequentially stored in the cache queues; after the image frames are stored in the cache queues, the display device performs human key point recognition on the image frames stored in any cache queue to obtain human key point images; that is to say, the display device asynchronously performs human key point recognition on the image frames stored in each cache queue; improving the efficiency and real-time performance of image frame detection. Since the display device asynchronously performs human key point recognition on the image frames stored in each cache queue, the time interval between obtaining human key point images by adjacent cache queues is equivalent to the time interval between obtaining two adjacent frames of images.

[0025] When rendering, based on the order of the cache queues, the image frames in the cache queues and the human key point images corresponding to the image frames are sequentially rendered (for example, first render the cache queue n1, and then render the cache queue n2; n1 and n2 are adjacent cache queues); because the time interval between obtaining human key point images by adjacent cache queues is equivalent to the time interval between obtaining two adjacent frames of images, the problem of low smoothness of the displayed image caused by stuttering during image display is avoided during rendering. Since the image frames in the cache queue and the human key point images are in a one-to-one correspondence relationship, the matching degree between the human key point images and the image frames is increased, the accuracy of the human key point images used for image frame rendering is improved, and the deviation between the image rendering result and the image frame is avoided. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] In order to more clearly illustrate the technical solutions of the present application, the drawings required for use in the embodiments will be briefly introduced below. Obviously, for those of ordinary skill in the art, other drawings can also be obtained based on these drawings without creative efforts.

[0027] Figure 1 It is a schematic diagram of the application scenario of the display device in the embodiment of the present application;

[0028] Figure 2 It is a block diagram of the hardware configuration of the display device in the embodiment of the present application;

[0029] Figure 3 It is a block diagram of the software configuration of the display device in the embodiment of the present application;

[0030] Figure 4a It is a schematic diagram of the i-th frame of image in the embodiment of the present application;

[0031] Figure 4bSchematic diagram of a human key point image of the i-th frame in an embodiment of the present application;

[0032] Figure 4c Schematic diagram of the rendering of the i-th frame image in an embodiment of the present application;

[0033] Figure 5a Schematic diagram of an (i + 1)-th frame image in an embodiment of the present application;

[0034] Figure 5b Schematic diagram of a human key point image of the (i + 1)-th frame in an embodiment of the present application;

[0035] Figure 5c Schematic diagram of the rendering of the (i + 1)-th frame image in an embodiment of the present application;

[0036] Figure 6 Schematic diagram of the rendering of a frame in an embodiment of the present application;

[0037] Figure 7 Schematic diagram of an image processing method in an embodiment of the present application;

[0038] Figure 8 Schematic diagram of an image processing method in an embodiment of the present application;

[0039] Figure 9 Schematic diagram of the rendering of a frame in an embodiment of the present application. Detailed implementation manners

[0040] To make the objectives and implementation manners of the present application clearer, the following will clearly and completely describe the exemplary implementation manners of the present application with reference to the accompanying drawings in the exemplary embodiments of the present application. Obviously, the described exemplary embodiments are only a part of the embodiments of the present application, rather than all of the embodiments.

[0041] It should be noted that the brief description of the terms in the present application is only for facilitating the understanding of the following described implementation manners, rather than intending to limit the implementation manners of the present application. Unless otherwise specified, these terms should be understood according to their ordinary and common meanings.

[0042] The terms "first", "second", "third", etc. in the description, claims and the above accompanying drawings of the present application are used to distinguish similar or like objects or entities, and do not necessarily mean to limit a specific order or sequence, unless otherwise noted. It should be understood that such terms can be interchanged under appropriate circumstances.

[0043] The terms "comprising" and "having" and any variations thereof are intended to cover inclusion without exclusivity. For example, a product or device comprising a series of components need not be limited to all the components clearly listed, but may include other components not clearly listed or inherent to such products or devices.

[0044] Figure 1 This is a schematic diagram of an application scenario shown according to some embodiments of the present application. This schematic diagram is intended to show a type of scenario in which there are multiple display devices and a server that can communicate with the display devices. These display devices include, but are not limited to, devices having data transceiver and processing functions, image display functions, and / or sound output functions. In Figure 1 The shown scenario includes a mobile device 100, a display device 200, and a server 300.

[0045] In some embodiments, the display device 200 (such as a smart TV and the mobile device 100) can display images captured by its own camera or images sent by the server 300.

[0046] Based on the technology of the Internet of Everything, communication connections can be established between multiple display devices in the above scenario. For example, communication can occur between the display device 200 and the mobile device 100 to project the images captured by the mobile device 100 onto the smart TV 200 for display.

[0047] The communication protocols for implementing the above Internet of Everything can include local area network protocols, wide area network protocols, and short-range wireless communication protocols not restricted by the network. Among them, the local area network protocols include, but are not limited to, the HSP communication protocol; the wide area network includes, but is not limited to, the Artificial Intelligence & Internet of Things (AIOT) protocol, and the short-range wireless communication protocols include, but are not limited to, the Bluetooth transmission protocol and the infrared transmission protocol.

[0048] Based on the differences in the types of the foregoing communication protocols, the communication protocol channels of the display device can be divided into local area network protocol channels based on the local area network, wide area network protocol channels based on the wide area network, and other protocol channels. The other protocol channels include, for example, the Bluetooth protocol channel and the infrared protocol channel. The display devices in the above scenario can support one or more of the foregoing protocol channels.

[0049] The display device 200 can establish a communication connection with the server 300 to interact with the server 300, such as providing various contents and interactive information to the display device. The display device is allowed to communicate and connect through a Local Area Network (LAN), a Wireless Local Area Network (WLAN), and other networks. The server 300 can be a cluster or multiple clusters, and can include one or more types of servers.

[0050] It should be noted that Figure 1 In the same scene shown, other display devices can also be included, including but not limited to external devices such as all-in-one touch devices, projection devices, tablet computers, computers, and laptop computers. The number of devices of the same type of terminal is not limited here.

[0051] Figure 2 The hardware configuration block diagram of the display device according to an exemplary embodiment is shown.

[0052] In some embodiments, the display device 200 includes at least one of a tuner demodulator 210, a communicator 220, a detector 230, an external device interface 240, a controller 250, a display 260, an audio output interface 270, a memory, a power supply, and a user interface.

[0053] In some embodiments, the controller includes a processor, a video processor, an audio processor, a graphics processor, a RAM, a ROM, and first to nth interfaces for input / output.

[0054] In some embodiments, the display 260 includes a display screen component for presenting a picture, and a driving component for driving image display, a component for receiving an image signal output from the controller, and for displaying video content, image content, and a menu control interface, as well as a user control UI interface.

[0055] In some embodiments, the display 260 can be a liquid crystal display, an OLED display, and a projection display, and can also be a projection device and a projection screen.

[0056] In some embodiments, the communicator 220 is a component for communicating with external devices or the server 300 according to various communication protocol types. For example: the communicator can include at least one of a Wifi module, a Bluetooth module, a wired Ethernet module, other network communication protocol chips or near field communication protocol chips, and an infrared receiver. The display device can establish the sending and receiving of control signals and data signals with an external control device or the server 300 through the communicator 220.

[0057] In some embodiments, the user interface can be used to receive control signals from a control device (such as an infrared remote control, etc.).

[0058] In some embodiments, the detector 230 is used to collect signals from the external environment or for external interactions. For example, the detector 230 includes a light receiver, which is a sensor for collecting the intensity of ambient light; alternatively, the detector 230 includes an image collector, such as a camera, which can be used to collect external environment scenes, user attributes, or user interaction gestures. Or, the detector 230 includes a sound collector, such as a microphone, etc., for receiving external sounds.

[0059] In some embodiments, the external device interface 240 can include, but is not limited to, the following: any one or more of a high-definition multimedia interface (HDMI), an analog or digital high-definition component input interface (component), a composite video input interface (CVBS), a USB input interface (USB), an RGB port, etc. It can also be a composite input / output interface formed by the above multiple interfaces.

[0060] In some embodiments, the tuner demodulator 210 receives broadcast television signals through wired or wireless reception, and demodulates audio and video signals, such as EPG data signals, from multiple wireless or wired broadcast television signals.

[0061] In some embodiments, the controller 250 and the tuner demodulator 210 can be located in different separate devices, that is, the tuner demodulator 210 can also be in an external device of the main device where the controller 250 is located, such as an external set-top box, etc.

[0062] In some embodiments, the controller 250 controls the operation of the display device and responds to user operations through various software control programs stored in the memory. The controller 250 controls the overall operation of the display device. For example: in response to receiving a user command for selecting a UI object to be displayed on the display 260, the controller 250 can perform operations related to the object selected by the user command.

[0063] In some embodiments, the object can be any one of the selectable objects, such as a hyperlink, an icon, or other operable controls. Operations related to the selected object include: operations such as displaying a page, document, image, etc. connected to the hyperlink, or performing an operation corresponding to the program of the icon.

[0064] In some embodiments, the controller includes at least one of a Central Processing Unit (CPU), a video processor, an audio processor, a Graphics Processing Unit (GPU), a Random Access Memory (RAM), a Read-Only Memory (ROM), a first interface to an nth interface for input / output, a communication bus (Bus), etc.

[0065] The CPU processor is used to execute the operating system and application program instructions stored in the memory, and to execute various application programs, data, and content according to various interaction instructions received from external inputs, so as to finally display and play various audio and video contents. The CPU processor may include multiple processors. For example, it includes a main processor and one or more sub-processors.

[0066] In some embodiments, the graphics processor is used to generate various graphic objects, such as icons, operation menus, and graphic displays of user input instructions, etc. The graphics processor includes an arithmetic unit that performs operations by receiving various interaction instructions input by the user and displays various objects according to display attributes; it also includes a renderer that renders various objects obtained based on the arithmetic unit, and the rendered objects are used to be displayed on the display.

[0067] In some embodiments, the video processor is used to receive an external video signal and perform video processing such as decompression, decoding, scaling, noise reduction, frame rate conversion, resolution conversion, and image synthesis according to the standard codec protocol of the input signal, and can obtain a signal that can be directly displayed or played on a display device.

[0068] In some embodiments, the video processor includes a demultiplexing module, a video decoding module, an image synthesis module, a frame rate conversion module, a display formatting module, etc. Among them, the demultiplexing module is used to demultiplex the input audio and video data stream. The video decoding module is used to process the demultiplexed video signal, including decoding and scaling processing, etc. The image synthesis module, such as an image synthesizer, is used to superimpose and mix the GUI signal generated by the graphics generator according to user input or generated by itself with the video image after scaling processing to generate an image signal for display. The frame rate conversion module is used to convert the input video frame rate. The display formatting module is used to change the received video output signal after frame rate conversion to a signal that conforms to the display format, such as outputting an RGB data signal.

[0069] In some embodiments, the audio processor is used to receive an external audio signal, and perform decompression and decoding, as well as noise reduction, digital-to-analog conversion, and amplification processing according to a standard codec protocol of the input signal to obtain a sound signal that can be played in a speaker.

[0070] In some embodiments, the user may input a user command in a graphical user interface (GUI) displayed on the display 260, and the user input interface receives the user input command through the graphical user interface (GUI). Alternatively, the user may input a user command by inputting a specific sound or gesture, and the user input interface receives the user input command by recognizing the sound or gesture through a sensor.

[0071] In some embodiments, the "user interface" is a medium interface for interaction and information exchange between an application or operating system and a user, which realizes the conversion between the internal form of information and the form acceptable to the user. The commonly used form of user interface is the graphical user interface (GUI), which refers to a user interface related to computer operation displayed in a graphical manner. It can be an interface element such as an icon, window, control, etc. displayed on the display screen of an electronic device, where the control can include icons, buttons, menus, tabs, text boxes, dialog boxes, status bars, navigation bars, widgets, and other visual interface elements.

[0072] See also Figure 3 In some embodiments, the system is divided into four layers, from top to bottom, namely, the application layer (Applications) layer (referred to as "application layer"), the application framework layer (Application Framework) layer (referred to as "framework layer"), the Android runtime (Android runtime) and system library layer (referred to as "system runtime library layer"), and the kernel layer.

[0073] In some embodiments, at least one application is running in the application layer, and these applications can be window programs, system settings programs, clock programs, etc. provided by the operating system, or applications developed by third-party developers. In specific implementations, the application packages in the application layer are not limited to the above examples.

[0074] The framework layer provides application programming interfaces (APIs) and programming frameworks for applications. The application framework layer includes some predefined functions. The application framework layer is equivalent to a processing center that determines the actions that applications in the application layer take. Through the API interface, applications can access system resources and obtain system services during execution.

[0075] As Figure 3 shown, in the embodiments of the present application, the application framework layer includes Managers, Content Provider, etc. Among them, the Managers include at least one of the following modules: ActivityManager is used to interact with all the activities running in the system; Location Manager is used to provide access to the system location service for system services or applications; Package Manager is used to retrieve various information related to the application packages currently installed on the device; NotificationManager is used to control the display and clearing of notification messages; Window Manager is used to manage icons, windows, toolbars, wallpapers, and desktop widgets on the user interface.

[0076] In some embodiments, the ActivityManager is used to manage the life cycles of various applications and the usual navigation back functions, such as controlling the exit, opening, and back of applications. The Window Manager is used to manage all window programs, such as obtaining the display screen size, determining whether there is a status bar, locking the screen, taking screenshots, and controlling the changes of the display window (such as shrinking the display window, jittering the display, distorting the display, etc.).

[0077] In some embodiments, the system runtime layer provides support for the upper layer, i.e., the framework layer. When the framework layer is used, the Android operating system will run the C / C++ libraries included in the system runtime layer to implement the functions to be achieved by the framework layer.

[0078] In some embodiments, the kernel layer is the layer between the hardware and the software. As Figure 3 shown, the kernel layer includes at least one of the following drivers: audio driver, display driver, Bluetooth driver, camera driver, WIFI driver, USB driver, HDMI driver, sensor driver (such as fingerprint sensor, temperature sensor, pressure sensor, etc.), and power driver, etc.

[0079] In some embodiments, the user can input a user command in the graphical user interface (GUI) displayed on the display, and then the user input interface receives the user input command through the graphical user interface (GUI). Alternatively, the user can input a user command by inputting a specific sound or gesture, and then the user input interface recognizes the sound or gesture through the sensor to receive the user input command. The user interface is an interface that can be used to receive control inputs (such as: physical buttons on the display device body, or others, etc.).

[0080] In some embodiments, a "user interface" is a media interface for interaction and information exchange between an application or an operating system and a user, which realizes the conversion between the internal form of information and the form acceptable to the user. The common manifestation form of a user interface is a graphical user interface (GUI), which refers to a user interface related to computer operations displayed in a graphical manner. It can be an interface element such as an icon, a window, a control, etc. displayed on the display screen of an electronic device, where the control can include visible interface elements such as an icon, a button, a menu, a tab, a text box, a dialog box, a status bar, a navigation bar, a Widget, etc.

[0081] In some embodiments, the display device 200 can obtain an image frame through an image collector; for example, the mobile device 100 captures an image frame through its own camera; the display device 200 can also establish a communication connection with the server 300 and receive the image frame sent by the server 300.

[0082] In some embodiments, the display device 200 displays a video picture through a display. Among them, the display principle of the video is based on the visual persistence effect of the human eye; a video is formed by continuously playing images frame by frame. The rate of continuous frame-by-frame playback is generally determined by the number of frames per second (Frames Per Second, FPS) of the picture.

[0083] FPS is a definition in the field of images, which refers to the number of frames transmitted per second of the picture. Generally speaking, it refers to the number of frames of an animation or a video. FPS is a measurement of the amount of information used to store and display dynamic videos. The more frames per second, the smoother the displayed action will be. Usually, to avoid unsmooth actions, the minimum FPS value is 30.

[0084] In some embodiments, a movie is played at a speed of 24 image frames per second, that is, 24 image frames are continuously projected on the screen within one second. In FPS, "F" is the English word Frame (picture, frame), "P" is the English word Per (per), and "S" is the English word Second (second). In Chinese, it is "how many frames per second". For example, the FPS value of a movie is 24.

[0085] In some embodiments, for any frame of image, when detecting scenes such as gestures, limbs, and faces, it is necessary to detect the image frame through a preset detection algorithm to obtain a human key point image, and then render the image frame based on the human key point image.

[0086] For example, when users are taking photos, recording videos, or live streaming, they add decorations to the human face in the image (such as glasses, hats, hair, etc.); because the human body changes in real time when the image is captured, that is, each frame of the image is different. Therefore, it is necessary to detect each frame of the image to locate the human body, and then make the image rendering result correspond to the image frame to ensure that the image rendering result corresponds to the image frame.

[0087] Figure 4a This is a schematic diagram of an i-th frame image exemplarily shown in some embodiments of the present application; Figure 4b Schematic diagram of an i-th frame of a human body key point image exemplarily shown in some embodiments of the present application; the display device 200 collects the key point image of the human body through its own camera. Figure 4a The i-th frame image shown in FIG. 1 is then detected according to a preset detection algorithm (such as a human posture detection algorithm OpenPose, etc.), and the i-th frame image is obtained as follows Figure 4b Then, the display device 200 renders the i-th frame image based on the i-th frame human key point image. Detecting the i-th frame image according to the preset detection algorithm is equivalent to identifying the human key points of the image frame.

[0088] in, Figure 4a Indicates image frames captured by the display device 200, such as human images, landscape images, etc.; Figure 4b It is a human key point image obtained by the display device 200 based on the collected image frame and calculated according to a preset detection algorithm, and is used to represent information such as the position and posture of the human body in the image frame. Figure 4b It can indicate the positions of the human body's head, hands, elbows, knees, feet, waist, etc.; the posture is standing with legs, arms spread out, facing downward.

[0089] based on Figure 4a and Figure 4b For example, in a live broadcast scene, you need to add a "magic wand" decoration to the human hand in the live broadcast picture; suppose that in the live broadcast picture, there is a Figure 4a The i-th frame image shown in FIG. 1 is detected by detecting the i-th frame image, and the following is obtained: Figure 4b The human body key point image shown in FIG. Figure 4b The human body key point image shown locates the position of the human hand.

[0090] Figure 4c is a schematic diagram of an i-th frame image rendering exemplarily shown in some embodiments of the present application; after locating the position of the human hand, based on the position of the human hand, in the example Figure 4a Add a "magic wand" decoration to the i-th frame image shown in the figure (such as Figure 4c (shown by the dashed line).

[0091] In some embodiments, the rendering methods generally include synchronous rendering and asynchronous rendering; among them, the synchronous rendering method is as follows:

[0092] Based on the above Figure 4a 、 Figure 4b and Figure 4c , Figure 5a is a schematic diagram of a (i + 1)-th frame image exemplarily shown in some embodiments of the present application; Figure 5b is a schematic diagram of a (i + 1)-th frame human key point image exemplarily shown in some embodiments of the present application; among them, the i-th frame image and the (i + 1)-th frame image are adjacent frame images, and the i-th frame image is earlier than the (i + 1)-th frame image. That is to say, Figure 4a the next frame of the image frame shown in Figure 5a is the image frame shown in

[0093] After rendering the i-th frame detection image, obtain the (i + 1)-th frame image shown in Figure 5a , and perform human key point recognition on the (i + 1)-th frame image to obtain the (i + 1)-th frame human key point image shown in Figure 5b , and then render the (i + 1)-th frame image based on the time interval between obtaining adjacent two-frame images, based on the (i + 1)-th frame human key point image; and so on, render each frame image.

[0094] The asynchronous rendering method is as follows:

[0095] Obtain Figure 4a the i-th frame image shown in, and perform human key point recognition on the i-th frame image based on a preset detection algorithm to obtain Figure 4b the i-th frame human key point image shown in; then render the i-th frame image based on the i-th frame human key point image; during the process of rendering the i-th frame image, asynchronously obtain Figure 5a the (i + 1)-th frame image shown in according to the time interval between obtaining adjacent two-frame images, and render the (i + 1)-th frame image based on the latest obtained human key point image according to the time interval between obtaining adjacent two-frame images; and so on, render each frame image.

[0096] In the above synchronous rendering method, if the execution time of the preset detection algorithm is relatively long, resulting in a relatively large time interval between obtaining the (i + 1)-th frame image and obtaining the i-th frame image, even if the number of frames played per second is small, resulting in a relatively large time interval between displaying adjacent two-frame images; that is, the rate of continuously playing image frames is relatively low, which will cause situations such as unsmooth and jerky video pictures.

[0097] In the above asynchronous rendering method, if the execution time of the preset detection algorithm is relatively long, resulting in Figure 4bThe human body key point image of the i-th frame is relatively long, which will result in rendering the i+1-th frame image based on the human body key point image of the i-th frame when rendering the i+1-th frame image; Figure 5c Schematic diagram of rendering of an i+1th frame image in some embodiments of the present application; after locating the human hand position based on the i-th frame human key point image, based on the human hand position, in the example Figure 5a Add a "magic wand" decoration to the i-th frame image shown in the figure (such as Figure 5c (shown by the dashed line in the middle). Figure 5c It can be seen that, because the right hand part of the i+1th frame image has changed relative to the i-th frame image, Figure 5c In the rendering shown, the position of the "magic wand" decoration on the right does not match the position of the right hand part, that is, the positions do not correspond.

[0098] As a result, when rendering the picture, the matching degree between the human key point image and the image frame is low, that is, the accuracy of the human key point image used for rendering the current frame image is low, causing the picture rendering result to deviate from the image frame.

[0099] Based on the above Figure 4a , Figure 4b , Figure 4c , Figure 5a , Figure 5b and Figure 5c , Figure 6 A schematic diagram of a picture rendering is shown as an example; Figure 6 As shown in FIG. 1 , when rendering the i+1th frame image, the right arm of the character in the i+1th frame image is facing upward, while the right arm of the i-th frame human key point image is facing downward; assuming that the image rendering is for the right arm of the character, when rendering the i+1th frame image, the rendering is based on the i-th frame human key point image, resulting in the following Figure 5c The rendering result shown is displayed below the right arm in the i+1th frame image, and is not displayed in the right arm in the i+1th frame image; therefore, when rendering the picture, the accuracy of the human body key point image used in the image frame rendering is low, causing the picture rendering result to deviate from the image frame.

[0100] In summary, in order to increase the matching degree between the human key point image and the image frame, improve the accuracy of the human key point image used in the image frame rendering, and avoid the deviation between the image rendering result and the image frame. In some embodiments, a schematic diagram of an image processing method is provided, such as Figure 7 As shown, the method can be applied to a display device 200 for performing an image processing method. The method includes the following contents:

[0101] The display device 200 calculates the ratio of the time taken to detect one frame of image to the interval time between two adjacent frames of image, and obtains the number of queues.

[0102] In some embodiments, if it is determined that the time taken to detect a frame of image is greater than the time interval between two adjacent frames of images obtained, then according to the integer division algorithm, the ratio result between the time taken to detect a frame of image and the time interval between two adjacent frames of images obtained is calculated; wherein, the time interval between two adjacent frames of images obtained is the ratio of 1 second to the preset FPS value. For example, if the preset FPS value is 100, then the time interval between two adjacent frames of images obtained is Tc = 1000 ms / 100 = 10 ms.

[0103] When performing the integer algorithm, the ratio result generally includes at least the following two parts: the integer part and the remainder part. In some embodiments, if the ratio result includes the integer part and the remainder part, then the ratio result is rounded up to obtain the queue number. Wherein, rounding up means adding 1 to the integer part of the ratio result and ignoring the remainder part of the ratio result.

[0104] For example, if the time taken for the preset detection algorithm to detect a frame of image is Ta = 50 ms, and the time interval between two adjacent frames of images obtained by the display device 200 through its own camera is Tc = 20 ms, then the ratio result is Ta / Tc = 50 / 20 = 2.5. That is, the integer part of the ratio result is 2, and the remainder part is 10. Further, the queue number n = 2 + 1 = 3.

[0105] In some embodiments, if the ratio result only includes the integer part, then the ratio result is used as the queue number. For example, if the time taken for the preset detection algorithm to detect a frame of image is Ta = 40 ms, and the time interval between two adjacent frames of images obtained by the display device 200 through its own camera is Tc = 20 ms, then the ratio result is Ta / Tc = 40 / 20 = 2. Further, the queue number n = 2.

[0106] In some embodiments, if it is determined that the time taken to detect a frame of image is less than or equal to the time interval between two adjacent frames of images obtained, that is, the integer part of the ratio result is 0 and the queue number is 1, then the image frame is directly obtained; which is equivalent to performing human key point recognition on each frame of image according to the synchronous detection method.

[0107] After calculating the queue number, the display device 200 creates a cache queue according to the queue number; taking the above embodiments as an example, when the queue number n = 3, 3 cache queues are created, namely n1, n2, and n3.

[0108] In some embodiments, the display device 200 obtains the image frames frame by frame, that is, there is an acquisition order for obtaining the image frames; for example, the i-th image frame is obtained at the i-th moment, and the (i + 1)-th image frame is obtained at the (i + 1)-th moment, where the i-th moment is earlier than the (i + 1)-th moment.

[0109] Therefore, when storing image frames into the cache queue, the image frames can be sequentially stored into the cache queue based on the acquisition order of the image frames. Taking the above embodiments as an example, the number of cache queues is 3, namely n1, n2, and n3; assuming that the 1st image frame, the 2nd image frame, ……, the i-th image frame are acquired in sequence, after acquiring the 1st image frame, store the 1st image frame into the cache queue n1; after acquiring the 2nd image frame, store the 2nd image frame into the cache queue n2; after acquiring the 3rd image frame, store the 3rd image frame into the cache queue n3; after acquiring the 4th image frame, store the 4th image frame into the cache queue n1; ……; after acquiring the i-th image frame, store the i-th image frame into the cache queue nx; where x is the remainder of i divided by the number of queues.

[0110] For example, if i = 16 and n = 3, then x is the remainder of i divided by the number of queues n, which is 1. That is, after acquiring the 16th image frame, store the 16th image frame into the cache queue n1.

[0111] After sequentially storing the image frames into the cache queue, for any cache queue, the display device 200 detects the human body image of the image frames in the cache queue; and generates a human key point image based on the human body image; where the human key point image represents the human body posture in the human body image. As Figure 4b Or Figure 5b shown, the human key point image corresponds to different human body postures. That is to say, when the display device 200 performs human key point recognition on the image frames of each cache queue, it is executed asynchronously. For example, when the display device 200 performs human key point recognition on the 1st image frame in the cache queue n1, the display device 200 asynchronously performs human key point recognition on the 2nd image frame in the cache queue n2. This improves the detection efficiency of the image frames.

[0112] After generating the human key point image, store the human key point image into the cache queue. Then, the display device 200 will sequentially render the human key point images and image frames in the cache queue according to the order of the cache queue.

[0113] Among them, the order of the cache queue is the storage order of the image frames; for example, the cache queue includes n1, n2, and n3 in total. When the display device 200 acquires the 1st image frame, it stores the 1st image frame into the cache queue n1; when the display device 200 acquires the 2nd image frame, it stores the 2nd image frame into the cache queue n2; when the display device 200 acquires the 3rd image frame, it stores the 3rd image frame into the cache queue n3; when the display device 200 acquires the 4th image frame, it stores the 4th image frame into the cache queue n1; and so on. It can be seen from this that the order of the cache queue is cache queue n1, cache queue n2, cache queue n3.

[0114] Before rendering the picture, traverse the storage status in each cache queue; in some embodiments, if the storage status in each cache queue is stored, then starting from the first cache queue, according to the storage order of the cache queue, render the human key point images and image frames in the cache queue in sequence.

[0115] Taking the above embodiments as an example, the order of the cache queues is n1, n2, n3, and the first cache queue is n1; after determining that n1, n2, and n3 have all stored the image frames and the human key point images corresponding to the image frames, starting from the cache queue n1, in the order of n1, n2, n3, loop to render the image frames and the human key point images of the image frames in the cache queue.

[0116] For example, first render the first image frame and the first human key point image in the cache queue n1; then render the second image frame and the second human key point image in the cache queue n2; then render the third image frame and the third human key point image in the cache queue n3; after the third image frame and the third human key point image are rendered, pull the fourth image frame and the fourth human key point image from the cache queue n1 again, and render the fourth image frame and the fourth human key point image; and so on, render each frame of the image.

[0117] In some embodiments, if the storage status in the first cache queue is stored, then starting from the first cache queue, according to the order of the cache queue, render the human key point images and image frames in the cache queue in sequence.

[0118] Because the time interval for the cache queue to store the image frames is the time interval for the display device 200 to acquire two adjacent frames of images, when performing picture rendering, it is rendered based on the time interval for the display device 200 to acquire two adjacent frames of images. Therefore, when it is determined that the first cache queue stores the human key point images and image frames, according to the order of the cache queue, based on the time interval for acquiring two adjacent frames of images, pull the human key point images and image frames from each cache queue in sequence; and after pulling the human key point images and image frames, render the human key point images and image frames.

[0119] Taking the above embodiments as an example, the acquisition time of the first image frame is Tc1, and the time when the cache queue n1 stores the first image frame and the first human key point image is Ta1; the acquisition time of the second image frame is Tc2, and the time when the cache queue n2 stores the second image frame and the second human key point image is Ta2; where Ta1 = Tc1 + Ta (the time consumed to detect one frame of image), Ta2 = Tc2 + Ta, Tc2 - Tc1 = Tc (the time interval for acquiring two adjacent frames of images); from this, it can be obtained that Ta2 - Ta1 = Tc2 - Tc1 = Tc.

[0120] That is to say, after the rendering of the first image frame is completed, the second image frame and the second human key point image have been stored in the cache queue n2, and then the second image frame can be rendered; and so on, rendering each image frame, which ensures the smoothness of frame-by-frame rendering.

[0121] In some embodiments, after the rendering of any image frame and the human key point image in the cache queue is completed, set a label for the rendered image frame and the human key point image; the label is used to indicate that the image frame and the human key point image have been rendered and will not be rendered repeatedly.

[0122] Taking the above embodiments as an example, after the rendering of the first image frame and the first human key point image in the cache queue n1 is completed, set a label for the first image frame and the first human key point image; when rendering the image frame and the human key point image in the cache queue n1 again, render them in the order of the cached image frames in the cache queue n1; for example, after the rendering of the third image frame and the third human key point image is completed, render the fourth image frame and the fourth human key point image in the cache queue n1.

[0123] That is to say, multiple image frames and the human key point images of the image frames can be stored in any cache queue. Taking the above embodiments as an example, the cache queue n1 stores the first image frame, the first human key point image, the fourth image frame, the fourth human key point image,..., the (3m + 1)th image frame and the (3m + 1)th human key point image; the cache queue n2 stores the second image frame, the second human key point image, the fifth image frame, the fifth human key point image,..., the (3m + 2)th image frame and the (3m + 2)th human key point image; the cache queue n3 stores the third image frame, the third human key point image, the sixth image frame, the sixth human key point image,..., the (3m + 3)th image frame and the (3m + 3)th human key point image; where m is a natural number.

[0124] In some embodiments, after the rendering of the image frame and the human key point image in any cache queue is completed, release the cache of the rendered image frame and the human key point image; that is to say, any cache queue only stores one image frame and the human key point image corresponding to the image frame.

[0125] Taking the above embodiments as examples, after the rendering of the first image frame and the first human key point image in the cache queue n1 is completed, the first image frame and the first human key point image are released in the cache queue n1. After the display device 200 obtains the fourth image frame, the fourth image frame is stored in the cache queue n1, thereby ensuring that only one image frame is stored in the cache queue, so as to avoid the problem of disordered rendering of image frames during screen rendering, increase the matching degree between the human key point image and the image frame, and improve the accuracy of the human key point image used for image frame rendering.

[0126] In some embodiments, the rendering time is less than or equal to the time interval between obtaining two adjacent image frames; wherein, the rendering time represents the duration of rendering any image frame and the human key point image of the image frame.

[0127] To better illustrate the technical solution of the present invention, Figure 8 an exemplary schematic diagram of an image processing method is shown, as Figure 8 shown, the content is as follows:

[0128] The time Ta for detecting one image frame is 130 ms, and the time interval Tc between obtaining two adjacent image frames is 40 ms. As Figure 8 shown, the time interval between obtaining the image frame f1 and obtaining the image frame f2 is 40 ms.

[0129] According to the time Ta for detecting one image frame and the time interval Tc between obtaining two adjacent image frames, the queue number n is calculated. Since the ratio result of Ta and Tc is 3.25, that is, the integer part of the ratio result is 3, and then the queue number n = 4 is obtained. That is to say, the cache queues are as Figure 8 shown, and successively include the cache queue n1, the cache queue n2, the cache queue n3, and the cache queue n4. Among them, the cache queue n1 is the first cache queue, and the image frame f1 is the first image frame.

[0130] As Figure 8 shown, the display device 200 obtains image frames frame by frame, that is, the order of obtaining frame by frame is the image frame f1, the image frame f2, the image frame f3,.... Among them, the acquisition interval between any two adjacent frames is Tc; for example, the time when the image frame f3 is obtained is t3, and the time when the image frame f4 is obtained is t4, and t3 - t4 = Tc.

[0131] After the display device 200 obtains the image frame f1, it stores the image frame f1 in the cache queue n1, and executes a preset detection algorithm to perform human key point recognition on the image frame f1 to obtain the human key point image a1; then it stores the human key point image a1 in the cache queue n1. That is to say, at this time, the cache queue n1 stores the image frame f1 and the human key point image a1 corresponding to the image frame f1.

[0132] Since the time Ta required to detect one frame of image is greater than the time interval Tc between obtaining two adjacent frames of images, and Ta is 3.25 times of Tc. Therefore, before obtaining the human key point image a1, the display device 200 will sequentially obtain the image frames f2, f3, and f4 based on the time interval Tc between obtaining two adjacent frames of images.

[0133] After the display device 200 obtains the image frame f2, it stores the image frame f2 into the cache queue n2, and executes a preset detection algorithm to perform human key point recognition on the image frame f2 to obtain the human key point image a2; then it stores the human key point image a2 into the cache queue n2.

[0134] And so on, after the display device 200 obtains the image frame f3, it stores the image frame f3 into the cache queue n3, and executes a preset detection algorithm to perform human key point recognition on the image frame f3 to obtain the human key point image a3; then it stores the human key point image a3 into the cache queue n3.

[0135] After the display device 200 obtains the image frame f4, it stores the image frame f4 into the cache queue n4, and executes a preset detection algorithm to perform human key point recognition on the image frame f4 to obtain the human key point image a4; then it stores the human key point image a4 into the cache queue n4.

[0136] That is to say, when the display device 200 executes the preset detection algorithm, it is executed asynchronously to improve the efficiency of human key point recognition for the image frames.

[0137] As Figure 8 shown, the time interval between the moments of obtaining the human key point images of any two adjacent frames is Tc; if the moment of obtaining the human key point image a3 is r3 and the moment of obtaining the human key point image a4 is r4, r3 - r4 = Tc.

[0138] After obtaining the image frame f1, the storage state of the cache queue n1 is monitored in real time. If it is monitored that the cache queue n1 stores the human key point image, starting from the cache queue n1, the image frames and the human key point images are cyclically pulled into the rendering queue in the order of n1, n2, n3, n4, and the image frames and the human key point images in the rendering queue are rendered in sequence.

[0139] For example, when it is determined that the human key point image a1 is stored in the cache queue n1, the human key point image a1 and the image frame f1 are pulled from the cache queue n1 and sent to the cache queue. Furthermore, the to-be-rendered image h1 is recorded in the cache queue; wherein, the to-be-rendered image h1 includes the human key point image a1 and the image frame f1. Then, the to-be-rendered image h1 (i.e., the human key point image a1 and the image frame f1) in the cache queue is started to be rendered.

[0140] Since the time interval between the obtaining times of any two adjacent frames of human key point images is Tc, the time interval between storing any two adjacent frames of to-be-rendered images in the rendering queue is also Tc. Therefore, when rendering the to-be-rendered images, the rendering is based on the time interval Tc, thereby avoiding the problems of low rendering smoothness and jitter of the rendering picture caused by the excessive time Ta consumed for detecting one frame of image, and improving the smoothness of the picture rendering.

[0141] Since any to-be-rendered image includes the image frame and the human key point image of this image frame, that is, the image frame and the human key point image of the to-be-rendered image correspond to each other. Figure 9 Exemplarily, a schematic diagram of picture rendering is shown, as Figure 9 shown, the human key point image of this image frame is recognized based on the image frame, and picture rendering is performed according to this image frame and the human key point image of this image frame, thereby avoiding the problems of inaccurate picture rendering result and mismatch with the image frame caused by the excessive time Ta consumed for detecting one frame of image, and improving the accuracy of the picture rendering.

[0142] In some embodiments, after the image frame and the human key point image in any cache queue are rendered, the rendered image frame and human key point image are cached and released. For example, the image frame f1 and the human key point image a1 of the cache queue n1 are sent to the rendering queue, and after the rendering is completed, the image frame f1 and the human key point image a1 in the cache queue n1 are released. That is, no information of any image frame is stored in the cache queue n1 after the cache is released.

[0143] Since the image frame f5 has not been obtained after the image frame f1 and the human key point image a1 are rendered, the cache queue n1 is an empty queue at this time. Thus, it can be seen that only the information of one image frame is stored in any cache queue, thereby ensuring that the image frame and the human key point image correspond to each other, that is, the human key point image is obtained based on this image frame, improving the accuracy rate of the human key point image used for rendering the image frame, and avoiding the deviation between the picture rendering result and the image frame.

[0144] And so on. After storing the image frame f4 in the buffer queue n4, it is equivalent to completing one round of storage, and a new round of storage starts from the image frame f5. That is to say, a new round of storage starts from the (m*n + 1)-th image frame; where n is the number of queues in the buffer queue and m is a natural number.

[0145] In some embodiments, for different image frames, the time consumption of the preset detection algorithm for human key point recognition is different, that is, the time consumption for detecting one frame of image is variable.

[0146] If the time consumption for detecting one frame of image is variable, the historical maximum value of the time consumption for detecting one frame of image can be calculated periodically, and the number of queues can be calculated based on this maximum value, so as to ensure the smoothness and accuracy of the frame rendering, and improve the flexibility and real-time performance of image processing.

[0147] For example, the cycle time is set to 1 minute. Assuming the current time is 3:00, then the maximum value of the time consumption for detecting one frame of image between 2:59 - 3:00 is calculated as 50ms, and "50ms" is used as the time consumption for detecting one frame of image between 3:00 - 3:01, and then the number of queues between 3:00 - 3:01 is calculated.

[0148] As can be seen from the above technical solutions, before rendering the image frame, the number of queues is determined according to the ratio of the time consumption for detecting one frame of image to the time interval for obtaining two adjacent image frames, and a buffer queue is created based on the number of queues; then the obtained image frames are sequentially stored in the buffer queue; among them, the image frames stored in the buffer queue are asynchronously recognized for human key points; the efficiency and real-time performance of image frame detection are improved.

[0149] Because the image frames stored in the buffer queue are asynchronously recognized for human key points, the time interval for obtaining the human key point images of adjacent buffer queues is equivalent to the time interval for obtaining two adjacent image frames. When rendering, based on the order of the buffer queues, the image frames in the buffer queue and the human key point images corresponding to the image frames are sequentially rendered.

[0150] Because the time interval for obtaining the human key point images of adjacent buffer queues is equivalent to the time interval for obtaining two adjacent image frames, the problem of low frame smoothness caused by frame stuttering is avoided during frame rendering, the matching degree between the human key point images and the image frames is increased, the accuracy of the human key point images used for image frame rendering is improved, and the deviation between the frame rendering result and the image frame is avoided.

[0151] For the similar parts among the embodiments provided in this application, reference can be made to each other. The specific embodiments provided above are only several examples under the general concept of this application and do not constitute a limitation on the protection scope of this application. For those skilled in the art, any other embodiments extended based on the solution of this application without creative efforts fall within the protection scope of this application.

Claims

1. A display device, characterized in that, Including: A display configured to display an image frame screen; A camera configured to collect image frames at a preset interval; The preset interval represents the interval time between two adjacent frames of images obtained by the camera; A controller configured to: Calculate the ratio of the time taken to detect one frame of image to the interval time between two adjacent frames of images to obtain the queue number; Create a cache queue according to the queue number; the cache queue is used to store image frames; Store the image frames into the cache queue in sequence according to the order of obtaining the image frames; For any cache queue, perform human key point recognition on the image frames in the cache queue to obtain human key point images; Render the human key point images and image frames in the cache queue in sequence according to the order of the cache queue; the order of the cache queue is the storage order of the image frames; wherein, the rendering time is less than or equal to the interval time between two adjacent frames of images; The rendering time represents the duration of rendering any frame of image and the human key point image of the image frame.

2. The display device according to claim 1, wherein Before the controller creates a cache queue according to the queue number, it is also configured to: If it is determined that the time taken to detect one frame of image is less than or equal to the interval time between two adjacent frames of images, directly obtain the image frame; If it is determined that the time taken to detect one frame of image is greater than the interval time between two adjacent frames of images, calculate the queue number.

3. The display device according to claim 2, wherein The controller calculates the ratio of the time taken to detect one frame of image to the interval time between two adjacent frames of images to obtain the queue number, which is specifically configured as: Perform an integer division operation on the time taken to detect one frame of image and the interval time between two adjacent frames of images to obtain a ratio result; the interval time between two adjacent frames of images is the ratio of 1 second to the preset FPS value; If there is a remainder in the ratio result, round up the ratio result to obtain the queue number.

4. The display device according to claim 1, characterized in that The controller stores the image frames into the cache queue in sequence according to the order of obtaining the image frames, which is specifically configured as: Store the i-th image frame into the x-th cache queue; where x is the remainder of i divided by the queue number.

5. The display device according to claim 1, characterized in that, The controller performs human key point recognition on the image frames in the cache queue to obtain human key point images, which is specifically configured as: Detect the human image in the image frame; Generate a human key point image based on the human image; the human key point image represents the human posture in the human image; After generating the human key point image, store the human key point image into the cache queue.

6. The display device according to claim 1, wherein The controller renders the human key point images and image frames in the cache queue in sequence according to the order of the cache queue, which is specifically configured as: Traverse the storage status in each cache queue; When it is determined that there are human key point images and image frames stored in the first cache queue, starting from the first cache queue, render the human key point images and image frames in the cache queue in sequence according to the order of the cache queue.

7. The display device according to claim 6, wherein The controller renders the human key point images and image frames in the cache queue in sequence according to the order of the cache queue, which is specifically configured as: Pull the human key point images and image frames from the cache queue in sequence based on the time interval between obtaining two adjacent frames of images according to the order of the cache queue; For any cache queue, after pulling the human key point images and image frames in the cache queue, render the human key point images and image frames in the cache queue.

8. The display device according to claim 7, characterized in that, After the controller renders the human key point images and image frames in the cache queue, it is further configured to: Release the cache of the cache queue.

9. An image processing method, characterized in that, The method is applied to a display device; The method includes: Calculate the ratio of the time taken to detect one frame of image to the time interval between obtaining two adjacent frames of images to obtain the number of queues; Create a cache queue according to the number of queues; the cache queue is used to store image frames; Store the image frames into the cache queue in sequence according to the order of obtaining the image frames; For any cache queue, perform human key point recognition on the image frames in the cache queue to obtain human key point images; Render the human key point images and image frames in the cache queue in sequence according to the order of the cache queue; the order of the cache queue is the storage order of the image frames; wherein, the rendering time is less than or equal to the time interval between obtaining two adjacent frames of images; the rendering time represents the duration of rendering any frame of image and the human key point image of the image frame.

Images