Display device and image recognition method

Abstract

In some embodiments of the present application, a display device and an image recognition method are provided. The method can identify the first head center point and the first body center point in the current image frame after acquiring the current image frame; and calculate the difference value of the current image frame based on the first head center point, the first body center point and the historical head center point and the historical body center point; when it is judged that the difference value of the current image frame is less than the difference threshold, the current body image of the current image frame is identified based on the historical body region; wherein the historical body region, the historical head center point and the historical body center point are obtained by identifying the historical image frame according to the target detection model; the difference value of the historical image frame is greater than or equal to the difference threshold; finally, the current body key point is identified from the current body image. In this way, the target detection model is avoided when recognizing any image frame, the efficiency of recognizing the human body image is improved, and the efficiency of image recognition is improved.

Description

Technical Field

[0001] The present application relates to the field of smart home technology, and in particular to a display device and an image recognition method. Background Art

[0002] With the widespread use of terminal devices and the popularization of camera technology, networks, and chips, deep learning algorithms for image recognition are increasingly being deployed in terminal devices. In some scenarios, image recognition involves identifying human bodies in images.

[0003] Currently, human image recognition is generally performed through a top-down recognition method. The top-down recognition method involves detecting the human body region in the image frame using an object detection (OD) model and cropping the human body region to obtain a human image. The cropped human body image is then sent to a pose estimation (PE) model, which detects the human body key points in the image.

[0004] When the picture is displayed, the greater the number of frames per second (FPS), the more images need to be recognized per second. For each frame of the image, the human body area must first be detected by the OD model, and then the human body key points must be detected by the PE model, resulting in low image recognition efficiency and poor real-time performance. Summary of the Invention

[0005] Some embodiments of the present application provide a display device and an image recognition method to solve the problems of low efficiency and poor real-time performance of image recognition.

[0006] In a first aspect, some embodiments of the present application provide a display device, including:

[0007] a display configured to display an image of a human body;

[0008] The controller is configured as:

[0009] Get the current image frame;

[0010] Identify the current image frame according to the posture detection model, and obtain a first head center point and a first body center point in the current image frame;

[0011] Calculating a difference value of the current image frame; the difference value of the current image frame is the sum of the distance between the first head center point and the historical head center point and the distance between the first body center point and the historical body center point;

[0012] If the difference value of the current image frame is less than the difference threshold, identifying the current human body image of the current image frame based on the historical human body region; wherein the historical human body region, the historical head center point, and the historical human body center point are obtained by identifying the historical image frame according to the target detection model; and the difference value of the historical image frame is greater than or equal to the difference threshold;

[0013] If the difference value of the current image frame is greater than or equal to the difference threshold, identifying the current image frame according to the target detection model to obtain a current human body image;

[0014] Identify current human body key points from the current human body image.

[0015] In a second aspect, some embodiments of the present application further provide an image recognition method, which is applied to a display device;

[0016] The method comprises:

[0017] Get the current image frame;

[0018] Identify the current image frame according to the posture detection model, and obtain a first head center point and a first body center point in the current image frame;

[0019] Calculating a difference value of the current image frame; the difference value of the current image frame is the sum of the distance between the first head center point and the historical head center point and the distance between the first body center point and the historical body center point;

[0020] If the difference value of the current image frame is less than the difference threshold, identifying the current human body image of the current image frame based on the historical human body region; wherein the historical human body region, the historical head center point, and the historical human body center point are obtained by identifying the historical image frame according to the target detection model; and the difference value of the historical image frame is greater than or equal to the difference threshold;

[0021] If the difference value of the current image frame is greater than or equal to the difference threshold, identifying the current image frame according to the target detection model to obtain a current human body image;

[0022] Identify current human body key points from the current human body image.

[0023] It can be seen from the above technical solution that after obtaining the current image frame, the display device directly uses the posture detection model (instead of using the target detection model first) to obtain the first head center point and the first body center point in the current image frame; then calculates the difference value of the image frame based on the first head center point, the first body center point, the historical head center point and the historical body center point.

[0024] When it is determined that the difference value of the image frame is less than the difference threshold, the human body image of the current image frame is directly identified based on the historical human body area, avoiding the use of the target detection model when recognizing any image frame, improving the efficiency of recognizing human body images, and thus improving the efficiency of image recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] In order to more clearly illustrate the technical solution of the present application, the following is a brief introduction to the drawings required for use in the embodiments. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0026] Figure 1 This is a schematic diagram of an application scenario of a display device in some embodiments of the present application;

[0027] Figure 2 This is a hardware configuration block diagram of a display device in some embodiments of the present application;

[0028] Figure 3 This is a software configuration block diagram of a display device in some embodiments of the present application;

[0029] Figure 4 This is a schematic diagram of key points of a human body in some embodiments of the present application;

[0030] Figure 5a This is a schematic diagram of an image frame in some embodiments of the present application;

[0031] Figure 5b This is a schematic diagram of a human body region in some embodiments of the present application;

[0032] Figure 5c This is a schematic diagram of key points of a human body in some embodiments of the present application;

[0033] Figure 6 This is a flowchart of an image recognition method in some embodiments of the present application;

[0034] Figure 7 A schematic diagram of an image recognition method in some embodiments of the present application;

[0035] Figure 8 This is a schematic diagram of the process structure of a posture detection model in some embodiments of the present application;

[0036] Figure 9 This is a schematic diagram of a human body image in some embodiments of the present application;

[0037] Figure 10 This is a schematic diagram of the process structure of a target detection model in some embodiments of the present application;

[0038] Figure 11This is a schematic diagram of a downsampling module in some embodiments of the present application;

[0039] Figure 12 A schematic diagram of a historical human body region in some embodiments of the present application;

[0040] Figure 13 This is a schematic diagram of a current human body image before affine transformation in some embodiments of the present application;

[0041] Figure 14 This is a schematic diagram of an angle in some embodiments of the present application;

[0042] Figure 15 This is a flowchart of an image recognition method in some embodiments of the present application. DETAILED DESCRIPTION

[0043] In order to make the purpose and implementation of this application clearer, the exemplary implementation of this application will be clearly and completely described below in conjunction with the drawings in the exemplary embodiments of this application. Obviously, the described exemplary embodiments are only part of the embodiments of this application, not all of the embodiments.

[0044] It should be noted that the brief descriptions of terms in this application are only for the purpose of facilitating the understanding of the embodiments described below, and are not intended to limit the embodiments of this application. Unless otherwise specified, these terms should be understood according to their ordinary and usual meanings.

[0045] In the specification and claims of this application and the accompanying drawings, the terms "first," "second," "third," etc. are used to distinguish similar or similar objects or entities, and are not necessarily intended to limit a particular order or sequence, unless otherwise noted. It should be understood that the terms used in this manner are interchangeable under appropriate circumstances.

[0046] The terms "comprise," "include," and "have," and any variations thereof, are intended to cover but not exclude inclusion; for example, a product or device comprising a list of components is not necessarily limited to all the components expressly listed but may include other components not expressly listed or inherent to such product or device.

[0047] Figure 1 This is a schematic diagram of an application scenario according to some embodiments of the present application. This diagram is intended to illustrate a type of scenario in which there are multiple display devices and a server that can communicate with the display devices. These display devices include but are not limited to devices with data receiving and processing functions, image display functions, and / or sound output functions. Figure 1 The illustrated scenario includes a mobile device 100 , a display device 200 , and a server 300 .

[0048] Based on the Internet of Things technology, communication connections can be established between multiple devices in the above scenario, such as communication between the mobile device 100 and the display device 200, so that the human body image captured by the mobile device 100 is projected onto the display device 200; communication between the server 300 and the display device 200, so that the display device 200 receives the video data sent by the server 300.

[0049] The communication protocols that enable the aforementioned Internet of Everything (IoE) can include local area network (LAN) protocols, wide area network (WAN) protocols, and short-range wireless communication protocols that are not restricted by the network. LAN protocols include, but are not limited to, the HSP communication protocol; WAN protocols include, but are not limited to, the Artificial Intelligence & Internet of Things (AIOT) protocol; and short-range wireless communication protocols include, but are not limited to, Bluetooth transmission protocols and infrared transmission protocols.

[0050] Based on the aforementioned communication protocol types, the display device's communication protocol channels can be divided into LAN protocol channels based on local area networks, WAN protocol channels based on wide area networks, and other protocol channels. Other protocol channels include Bluetooth protocol channels and infrared protocol channels. The display device in the above scenario may support one or more of the aforementioned protocol channels.

[0051] The display device 200 can establish a communication connection with the server 300 to exchange information with the server 300, such as providing various content and interactive information to the display device. The display device can be allowed to communicate via a local area network (LAN), a wireless local area network (WLAN), and other networks. The server 300 can be a cluster or multiple clusters, and can include one or more types of servers.

[0052] It should be noted that Figure 1 The same scenario shown may also include other display devices, including but not limited to touch-sensitive integrated devices, projection devices, AI fitness mirrors, tablet computers, computers, laptop computers, and other external devices; mobile device 100 is also a display device. The number of similar terminal devices is not limited here.

[0053] Figure 2 A hardware configuration block diagram of a display device according to an exemplary embodiment is shown.

[0054] In some embodiments, the display apparatus 200 includes at least one of a tuner 210 , a communicator 220 , a detector 230 , an external device interface 240 , a controller 250 , a display 260 , an audio output interface 270 , a memory, a power supply, and a user interface.

[0055] In some embodiments, the controller includes a processor, a video processor, an audio processor, a graphics processor, RAM, ROM, and first to nth interfaces for input / output.

[0056] In some embodiments, the display 260 includes a display screen component for presenting images, and a driving component for driving image display, a component for receiving image signals output from a controller, and a component for displaying video content, image content, and a menu control interface and a user control UI interface.

[0057] In some embodiments, the display 260 may be a liquid crystal display, an OLED display, a projection display, or a projection device and a projection screen.

[0058] In some embodiments, the communicator 220 is a component used to communicate with external devices or servers using various communication protocols. For example, the communicator may include at least one of a Wi-Fi module, a Bluetooth module, a wired Ethernet module, or other network communication protocol chip or a near-field communication protocol chip, as well as an infrared receiver. The display device can use the communicator 220 to send and receive control signals and data signals with the external control device 100 or the server 400.

[0059] In some embodiments, the external device interface 240 may include, but is not limited to, any one or more of the following: a high-definition multimedia interface (HDMI), an analog or digital high-definition component input interface (component), a composite video input interface (CVBS), a USB input interface (USB), an RGB port, etc. It may also be a composite input / output interface formed by multiple of the above interfaces.

[0060] In some embodiments, the tuner-demodulator 210 receives broadcast television signals via wired or wireless reception, and demodulates audio and video signals, such as EPG data signals, from a plurality of wireless or wired broadcast television signals.

[0061] In some embodiments, the controller 250 and the tuner 210 may be located in different separate devices, that is, the tuner 210 may also be located in an external device of the main device where the controller 250 is located, such as an external set-top box.

[0062] In some embodiments, controller 250 controls the operation of the display device and responds to user operations using various software control programs stored in memory. Controller 250 controls the overall operation of the display device. For example, in response to receiving a user command to select a UI object for display on display 260, controller 250 may perform operations related to the object selected by the user command.

[0063] In some embodiments, the object may be any selectable object, such as a hyperlink, an icon, or other operable control. Operations associated with the selected object include: displaying a page, document, or image connected to the hyperlink, or executing a program corresponding to the icon.

[0064] In some embodiments, the controller includes at least one of a central processing unit (CPU), a video processor, an audio processor, a graphics processing unit (GPU), RAM (Random Access Memory, RAM), ROM (Read-Only Memory, ROM), a first interface to an nth interface for input / output, a communication bus (Bus), etc.

[0065] The CPU is used to execute operating system and application instructions stored in memory, as well as various interactive instructions received from external input, to execute various applications, data, and content, ultimately displaying and playing various audio and video content. The CPU can include multiple processors, such as a main processor and one or more sub-processors.

[0066] In some embodiments, a graphics processor is used to generate various graphical objects, such as icons, operation menus, and graphics displayed in response to user input commands. The graphics processor includes an arithmetic unit, which performs operations based on various user interaction commands and displays various objects according to display attributes; and a renderer, which renders the various objects generated by the arithmetic unit. These rendered objects are then displayed on the display.

[0067] In some embodiments, a video processor is used to receive an external video signal and perform video processing such as decompression, decoding, scaling, noise reduction, frame rate conversion, resolution conversion, and image synthesis according to the standard codec protocol of the input signal to obtain a signal that can be directly displayed or played on a display device.

[0068] In some embodiments, the video processor includes a demultiplexing module, a video decoding module, an image synthesis module, a frame rate conversion module, a display formatting module, and the like. The demultiplexing module is used to demultiplex the input audio and video data stream. The video decoding module is used to process the demultiplexed video signal, including decoding and scaling processing. The image synthesis module, such as an image synthesizer, is used to superimpose and mix the GUI signal generated by the graphics generator according to user input or itself with the scaled video image to generate an image signal for display. The frame rate conversion module is used to convert the input video frame rate. The display formatting module is used to receive the frame rate converted video output signal and change the signal to conform to the display format signal, such as outputting an RGB data signal.

[0069] In some embodiments, the audio processor is used to receive an external audio signal, perform decompression and decoding according to the standard codec protocol of the input signal, as well as noise reduction, digital-to-analog conversion, and amplification processing to obtain a sound signal that can be played in a speaker.

[0070] In some embodiments, the user may input a user command through a graphical user interface (GUI) displayed on the display 260, and the user input interface receives the user input command through the graphical user interface (GUI). Alternatively, the user may input a user command through a specific sound or gesture, and the user input interface may recognize the sound or gesture through a sensor to receive the user input command.

[0071] In some embodiments, a "user interface" is a medium interface for interaction and information exchange between an application or operating system and a user, which realizes the conversion between the internal form of information and the form acceptable to the user. A common form of user interface is a graphical user interface (GUI), which refers to a user interface related to computer operations that is displayed in a graphical manner. It can be an interface element such as an icon, window, or control displayed on the display screen of an electronic device, where a control can include visual interface elements such as icons, buttons, menus, tabs, text boxes, dialog boxes, status bars, navigation bars, and widgets.

[0072] See also Figure 3 In some embodiments, the system is divided into four layers, from top to bottom: the application layer (referred to as the "application layer"), the application framework layer (referred to as the "framework layer"), the Android runtime and system library layer (referred to as the "system runtime library layer"), and the kernel layer.

[0073] In some embodiments, at least one application runs in the application layer. These applications can be window programs, system settings programs, clock programs, etc. that come with the operating system, or applications developed by third-party developers. In specific implementations, the application packages in the application layer are not limited to the above examples.

[0074] The framework layer provides applications with an application programming interface (API) and programming framework. The application framework layer includes predefined functions. The application framework layer acts as a processing center, determining the actions taken by applications in the application layer. Through the API, applications can access system resources and services during execution.

[0075] like Figure 3 As shown, in the embodiment of the present application, the application framework layer includes managers, content providers, view systems, etc., wherein the manager includes at least one of the following modules: an activity manager for interacting with all activities running in the system; a location manager for providing system services or applications with access to system location services; a package manager for retrieving various information related to the application packages currently installed on the device; a notification manager for controlling the display and clearing of notification messages; and a window manager for managing icons, windows, toolbars, wallpapers, and desktop widgets on the user interface.

[0076] In some embodiments, the activity manager is used to manage the lifecycle of each application and common navigation back functions, such as controlling the exit, opening, and backing of an application. The window manager is used to manage all window programs, such as obtaining the display screen size, determining whether there is a status bar, locking the screen, taking screenshots, and controlling display window changes (such as shrinking the display window, shaking the display, distorting the display, etc.).

[0077] In some embodiments, the system runtime layer provides support for the upper layer, namely the framework layer, including databases (such as WEBKIT, OpenGL ES database, etc.) and virtual machines; when the framework layer is used, the Android operating system will run the C / C++ library contained in the system runtime layer to implement the functions to be implemented by the framework layer.

[0078] In some embodiments, the kernel layer is a layer between hardware and software. Figure 3 As shown, the kernel layer includes at least one of the following drivers: audio driver, display driver, Bluetooth driver, camera driver, WIFI driver, USB driver, HDMI driver, sensor driver (such as fingerprint sensor, temperature sensor, pressure sensor, etc.), and power driver, etc.

[0079] In some embodiments, a user may enter a user command through a graphical user interface (GUI) displayed on a display, and the user input interface receives the user input command through the graphical user interface (GUI). Alternatively, a user may enter a user command through a specific sound or gesture, and the user input interface may recognize the sound or gesture through a sensor to receive the user input command. A user interface is an interface that can be used to receive control input (such as a physical button on a display device, or other interface).

[0080] In some embodiments, a "user interface" is a medium interface for interaction and information exchange between an application or operating system and a user, which realizes the conversion between the internal form of information and the form acceptable to the user. A common form of user interface is a graphical user interface (GUI), which refers to a user interface related to computer operations that is displayed in a graphical manner. It can be an interface element such as an icon, window, or control displayed on the display screen of an electronic device, where a control can include visual interface elements such as icons, buttons, menus, tabs, text boxes, dialog boxes, status bars, navigation bars, and widgets.

[0081] In some embodiments, the display device 200 can obtain image frames through an image collector; for example, the mobile device 100 collects image frames through its own camera; the display device 200 can also establish a communication connection with the server 300 to receive image frames sent by the server 300.

[0082] In some embodiments, for any image frame, the image frame is detected by a preset human posture recognition algorithm (such as the openpose algorithm) to obtain key points of the human body in the image; and the posture of the human body is represented according to the key points of the human body. Figure 4 An exemplary diagram of a key point of a human body is shown, as shown in FIG. Figure 4 As shown in the figure, the key points of the human body are the nodes of the human limbs, such as wrists, elbows, knees, hips, ankles, head, neck, etc.; they can represent the current posture and movement of the human body ( Figure 4 The action in the figure is that the person stands with arms outstretched; one arm is tilted downward and the other arm is tilted upward).

[0083] For example, in an artificial intelligence (AI) fitness scenario, the display device 200 captures human images frame by frame; then, for any human image, the human body key points in the human body image are identified through a human posture recognition algorithm; finally, the human body key points are compared with the comparison key points set at the current moment to obtain a comparison result. Feedback suggestions are provided to the user based on the comparison results. For example, if the comparison key points set at the current moment indicate that the human arm is horizontally open, but the human body key points in the human body image at the current moment are not in a horizontal state, a suggestion to "adjust the arm upwards" or "adjust downwards" will be fed back to the user.

[0084] In some embodiments, the recognition of human body images is generally performed through a top-down recognition method; Figure 5a An image frame schematic diagram is shown as an example. Figure 5b A schematic diagram of a human body region is shown as an example. Figure 5c A schematic diagram of key points of a human body is shown as an example.

[0085] The top-down recognition method can first Figure 5a The original image frame shown is input to the object detection (OD) model to detect the human body area in the human body image, and obtain Figure 5b The human body area shown; Figure 5b As shown, the “dashed box” corresponding to the human body represents the human body area, which is used to mark the range where the human body is located.

[0086] Then, according to the human body area, the human body image is cropped and the obtained human body image is input into the pose estimation (PE) model, and the PE model recognizes the key points of the human body in the human body image; Figure 5c As shown, the nodes at both ends of any straight line represent the key points of the human body (such as the nodes at both ends of a straight line represent the wrist key point and elbow key point of the human arm). The human body posture can be represented by the lines between the key points of the human body.

[0087] In some embodiments, the display device 200 can capture the user's body image through an image capture device installed on itself (such as a camera); the display device 200 can also establish a communication connection with a mobile device (such as a mobile phone), and the mobile device 100 can capture the user's body image, and then display the user's body image on the display device 200 in a mirroring manner.

[0088] Figure 6 An exemplary flow chart of an image recognition method is shown as follows: Figure 6 As shown, the content is as follows:

[0089] Step 610: Capture image frames;

[0090] Image frames are acquired through the Open Graphics Library (OpenGL), a cross-language, cross-platform application programming interface for rendering 2D and 3D vector graphics.

[0091] Step 620: OD model identification;

[0092] The image frame is scaled so that the scaled image frame meets the recognition condition of the OD model; for example, the recognition condition of the OD model is to recognize an image frame with a preset resolution (such as 224×224); that is, the resolution of the image frame needs to be scaled to 224×224.

[0093] The scaled image frame is then input into the OD model, so that the OD model performs image recognition on the scaled image frame to obtain a recognition result.

[0094] Step 630: Determine whether there is a human body; if so, execute step 640; otherwise, return to step 610 to capture the next frame of image.

[0095] Step 640: cropping the human body image;

[0096] If a human body is detected in the recognition result, the human body image is cropped from the image frame based on the human body region in the image frame.

[0097] Step 650: Input the PE model and identify key points of the human body;

[0098] The cropped human body image is input into the PE model so that the PE model can identify the key points of the human body in the human body image.

[0099] In the above method, for any frame image, it is necessary to first identify the human body image through the OD model, and then identify the human body key points in the human body image through the PE model, resulting in low efficiency and poor real-time performance of image recognition.

[0100] Therefore, in order to improve the efficiency and real-time performance of image recognition, in some embodiments, Figure 7 An exemplary diagram of an image recognition method is provided, such as Figure 7 As shown, the method can be applied to a display device 200 for performing an image recognition method. The method includes the following contents:

[0101] Step 710: Acquire the current image frame.

[0102] The display device 200 obtains the current image frame; wherein, the display device 200 can obtain the image frame through its own image acquisition device (such as a camera); and can also receive image frames sent by a server or other mobile devices (such as a mobile phone).

[0103] Step 720: The posture detection model identifies the current image frame.

[0104] In some embodiments, the display device 200 captures images frame by frame and identifies the first head center point and the first body center point of each frame based on the posture detection model. For example, the i-th image frame (i.e., the current image frame) is input into the posture detection model, and the posture detection model calculates the first head center point and the first body center point of the current image frame.

[0105] Step 730: Calculate the difference value of the current image frame.

[0106] After predicting the first head center point and the first body center point in the previous image frame, the display device 200 calculates the difference value of the current image frame based on the distance between the first head center point and the historical head center point and the distance between the first body center point and the historical body center point.

[0107] Among them, the historical head center point and the historical body center point are obtained by identifying the historical image frames based on the target detection model.

[0108] Step 740, determine whether the current image frame is less than the difference threshold; if so, execute step 750, otherwise execute step 760; wherein the difference threshold can be a value preset based on experience, such as 0.05, 0.1, etc., which is not limited here.

[0109] Step 750: Crop the current human body image according to the historical human body regions.

[0110] According to the historical human body region, the current human body image is cropped in the current image frame; wherein the historical human body region is based on the historical image frame and is obtained by identifying the historical image frame according to the target detection model; the difference value of the historical image frame is greater than or equal to the difference threshold.

[0111] Step 760: Crop the current human body image according to the target detection model.

[0112] The current image frame is input into the target detection model, and the target detection model predicts the current human body area of the current image frame, and then the current human body image is cropped in the current image frame according to the current human body area.

[0113] Step 770: Identify the current human body key points from the human body image.

[0114] The angle between the connecting line between the center point of the head and the center point of the body in the body image and the vertical direction is determined; and whether the body image needs to be affine transformed is determined based on the angle. In some embodiments, the current body key point in the body image is determined based on the affine transformation.

[0115] In step 720, Figure 8 The following is an exemplary diagram showing the flow structure of a posture detection model: Figure 8 As shown in the figure, the process of the posture detection model includes standard convolution, inverse residual convolution network, convolution, depthwise separable convolution and inverse residual convolution network; standard convolution can be two-dimensional convolution, three-dimensional convolution, etc., which is not specifically limited here.

[0116] The current image frame is input into each convolution layer of the posture detection model, and each convolution layer is trained to obtain the first head center point and the first body center point of the image frame; as well as each body key point (or each body skeleton point) in the current image frame.

[0117] like Figure 8 As shown, the "first convolution module" includes 2D convolution "conv2D", depth-wise separable convolution "DepthwiseConv2D", and activation function "Relu6"; the "second convolution module" includes 2D convolution "conv2D", depth-wise separable convolution "DepthwiseConv2D", activation function "Relu6", and adding feature information "Add". The "third convolution module" includes 2D convolution "conv2D", depth-wise separable convolution "DepthwiseConv2D", Op operator "resizeBilinear", and adding feature information "Add".

[0118] Based on the above Figure 8 For example, after the display device 200 obtains the i-th image frame, the i-th image frame is input into the posture detection model; the posture detection model first identifies the feature vector of the i-th image frame by continuously passing through two first convolution modules; and then according to Figure 8 In the order of convolution layers shown in , the first convolution module, the second convolution module, and the third convolution module downsample the feature vector, that is, reduce the dimension of the feature vector (for example, reduce the dimension of a vector with a dimension of 224×224 to a dimension of 16×16); the depthwise separable convolution and the 2D convolution (that is, two-dimensional convolution) upsample the feature vector, that is, increase the dimension of the feature vector (for example, increase the dimension of a vector with a dimension of 16×16 to a dimension of 56×56), and thus the features of the i-th image frame can be obtained.

[0119] The features of the current image frame include key points of the human body in the current image frame; the key points of the human body include the center point of the head and the center point of the human body.

[0120] Figure 9 A schematic diagram of a human body image is shown as an example. Figure 9 As shown in the figure, after inputting the human body image into the posture detection model, the following can be predicted: Figure 9The "A" point (i.e. the first head center point) and the "B" point (i.e. the first body center point) are shown in FIG.

[0121] In step 730, after predicting the first head center point and the first body center point in the previous image frame, the display device 200 calculates the difference value of the current image frame based on the first head center point, the first body center point and the historical head center point, the historical body center point.

[0122] The historical head center and body center are obtained by using the object detection model to identify the historical image frames. In other words, the object detection model can predict the historical image frames and obtain the historical head center and body center.

[0123] Figure 10 An exemplary process structure diagram of a target detection model is shown as follows: Figure 10 As shown in Figure 3, the process of the target detection model includes an inverse residual convolutional network, depth-separable convolution, and standard convolution.

[0124] Among them, the activation function is used to prevent overfitting when calculating image frame features; the "pool2D layer" is used to extract information and make the features two-dimensional data; the average pooling layer is an average pooling layer for two-dimensional data input.

[0125] Based on the above Figure 10 For example, the display device 200 obtains the i-1th image frame, and after determining that the difference value of the i-1th image frame is greater than or equal to the difference threshold, the i-1th image frame is input into the target detection model; the target detection model first identifies the feature vector of the i-1th image frame by continuously passing through the 2D convolution layer; and then according to Figure 10 The convolutional layer sequence shown in , the downsampling module downsamples the feature vector; the convolutional layers such as depthwise separable convolution, 2D convolution and average pooling layer upsample the feature vector, and then the output matrix of the i-1th image frame can be obtained.

[0126] Then, the "box position" and "point position" are output according to the non-maximum suppression algorithm (NMS); the "box position" is the human body area in the i-1th image frame; the "point position" is the center point of the head and the center point of the human body in the i-1th image frame.

[0127] Figure 11 A schematic diagram of a downsampling module is shown as an example. Figure 11 As shown, the downsampling module includes the input layer "Input", 2d convolution "conv2D", activation function "Prelu" and added feature "Concat"; it can be used as Figure 10In the object detection model shown in Figure 1, the downsampling module reduces the dimensionality of the image frame's feature vector to facilitate computation. For example, a 500×200 resolution image frame fed into the input layer of the object detection model produces a 128×128 feature vector. The downsampling module reduces this 128×128 feature vector to a 56×56 feature vector.

[0128] In some embodiments, the historical image frame is an image frame before the current image frame; for example, the current image frame is the i-th image frame, and the historical image frame is the i-1-th image frame. The i-th image frame is the image frame at the i-th time, and the i-1-th image frame is the image frame at the i-1-th time; the i-1-th time is adjacent to the i-th time, and the i-1-th time is earlier than the i-th time.

[0129] In some embodiments, the difference value of the current image frame is calculated according to the following formula (1);

[0130]

[0131] Where e is the difference value of the current image frame; s1 is the first distance; and s2 is the second distance. The difference value of the current image frame represents the deviation between the spatial position of the human body in the current image frame and the spatial position of the human body in the previous image frame. For example, a larger difference value indicates a greater deviation between the spatial position of the human body in the previous image frame and the previous image frame.

[0132] In some embodiments, the first distance is the spatial distance between the first head center point and the historical head center point; the larger the first distance, the larger the spatial distance between the first head center point and the historical head center point; conversely, the smaller the first distance, the smaller the spatial distance between the first head center point and the historical head center point.

[0133] In some embodiments, the second distance is the spatial distance between the first body center point and the historical body center point; the larger the second distance, the larger the spatial distance between the first body center point and the historical body center point; conversely, the smaller the second distance, the smaller the spatial distance between the first head center point and the historical head center point.

[0134] In some embodiments, the spatial distance between any two points can be calculated based on the spatial coordinates of the two points; for example, based on the spatial coordinates of the first head center point (x1, y1, z1) and the spatial coordinates of the historical head center point (x`1, y`1, z`1), the spatial distance between the first head center point and the historical head center point is calculated.

[0135] In some embodiments, the spatial distance between any two points can be calculated according to a preset distance algorithm (such as the Euclidean distance algorithm and the Mahalanobis distance algorithm, etc.); for example, the Euclidean distance between the first human body center point and the historical human body center point is calculated according to the Euclidean distance algorithm; the Euclidean distance is the spatial distance.

[0136] Based on the above embodiment, for example, for the current image frame (such as the i-th image frame), the display device 200 predicts at the current moment that the first head center point of the i-th image frame is A1 and the first body center point is B1 based on the posture detection model; the historical image frame is the i-1-th image frame, and the display device 200 predicts at the i-1-th moment that the historical head center point of the i-1-th image frame is A`1 and the historical body center point is B`1 based on the target detection model. Calculate the first distance s1 = (A1-A`1) and the second distance s2 = (B1-B`1).

[0137] In some embodiments, the historical image frame is an image frame before the current moment, and the difference value of the image frame is greater than or equal to the difference threshold; for example, the difference value of the i-1th image frame is less than the difference threshold, and the difference value of the i-2th image frame is greater than or equal to the difference threshold, then the i-2th image frame is used as the historical image frame of the i-th image frame (current image frame), and the difference value of the i-th image frame is calculated based on the human body center point and the head center point of the i-2th image frame.

[0138] In some embodiments, the historical image frame is an image frame whose difference value is greater than or equal to the difference threshold and is adjacent to the current image frame; for example, the difference value of the i-1th image frame is less than the difference threshold, the difference value of the i-2th image frame is greater than or equal to the difference threshold, and the difference value of the i-3th image frame is greater than or equal to the difference threshold, then the i-2th image frame serves as the historical image frame of the i-th image frame.

[0139] In some embodiments, the historical image frame first obtains the first historical head center point, the first historical body center point and the historical body area of the historical image frame according to the target detection model; then, based on the historical body area, the second historical head center point, the second historical body center point and the historical body key points of the historical image frame are obtained according to the posture detection model.

[0140] Among them, the first historical head center point and the first historical body center point are used as the historical head center point and the historical body center point corresponding to the current image frame; because the first historical head center point and the first historical body center point are obtained according to the target detection model training, if the difference value of the current image frame is less than the difference threshold, it means that the spatial distance deviation between the first head center point in the current image frame and the historical head center point is small, the spatial distance deviation between the first body center point in the current image frame and the historical body center point is small, and the human body figure in the current image frame is in the historical body area.

[0141] That is to say, when the difference value of the current image frame is less than the difference threshold, the current human body image of the current image frame can be identified based on the historical human body area, thereby improving the recognition accuracy of the current human body image, and there is no need to train the current image frame through the target detection model to first obtain the current human body area and then obtain the current human body image; that is, it is equivalent to using the historical human body area as the current human body area; thereby improving the efficiency of identifying the current human body image and improving the efficiency of image recognition.

[0142] In some embodiments, the historical body area is generated with the historical body center point as the center and a preset multiple of the distance between the historical body center point and the historical head center point as the radius.

[0143] Figure 12 A schematic diagram of a historical human body region is shown as an example. Figure 12 As shown, with point B' (the historical center of the human body) as the center, assuming that the distance between A' and B' is R', and 2R' as the radius (that is, the preset multiple is 2 times), the following is generated: Figure 12 The circular area boundary shown is used to obtain the historical human body area.

[0144] It should be noted that, in some embodiments, the area boundary corresponding to the historical human body area can also be generated with the historical human body center point as the center and a preset multiple (1.5 times, 1.2 times, etc.) of the distance between the historical human body center point and the historical head center point as the radius. The preset multiples are not limited here.

[0145] In step 750, the display device 200 uses the historical human body region as the current human body region and then identifies the current human body region in the current image frame based on the current human body region. This avoids the step of using an object detection model to train the current image frame to obtain the current human body region, improving the efficiency of identifying the current human body image and image recognition.

[0146] In step 770, after the human body image is obtained, an affine transformation is performed on the current human body image according to the first head center point and the first body center point.

[0147] In some embodiments, when performing affine transformation on the current human body image, the first head center point and the first human body center point are connected to obtain a connecting line; Figure 13 An exemplary diagram of a current human body image before affine transformation is shown as follows: Figure 13 As shown, in some embodiments, the acquired current human body image is tilted relative to the vertical direction, where the vertical direction is a direction perpendicular to the bottom surface.

[0148] For example, when the image acquisition device is acquiring images, the image acquisition device is not parallel to the ground, resulting in the acquired image frame being tilted, and therefore the acquired current human body image is tilted relative to the vertical direction.

[0149] For example, when the image acquisition device is acquiring images, the human body's posture is tilted relative to the ground, so the current human body image acquired is tilted relative to the vertical direction.

[0150] That is, the connection line between A (the first head center point) and B (the first body center point) in the current human body image has a certain angle with the vertical direction.

[0151] Figure 14 A schematic diagram of an angle is shown as an example, Figure 14 As shown, the connecting line is the line between point A and point B, and the vertical direction is the direction perpendicular to the bottom surface. Calculate the angle (e.g., 30°, 15°, etc.) between the connecting line between A and B and the vertical direction, and determine whether the angle is greater than an angle threshold; the angle threshold can be a value preset based on experience, such as 10°, 5°, etc.

[0152] When it is determined that the angle between the connecting line between A and B and the vertical direction is greater than the angle threshold, it is necessary to perform an affine transformation on the current human body image so that the angle between the connecting line between A and B and the vertical direction in the current human body image after the affine transformation is less than the angle threshold; Figure 14 An exemplary diagram of a current human body image after affine transformation is shown, as shown in FIG. Figure 14 As shown, the connecting line between A and B coincides with the vertical direction, that is, the angle between the connecting line between A and B and the vertical direction is 0.

[0153] In some embodiments, the affine transformation is as shown in the following formula (2):

[0154]

[0155] Among them, x` and y` are the coordinate values of the human body image after affine transformation; x and y are the coordinate values of the human body image before affine transformation; Tx and Ty are the image translation amounts during affine transformation; R00, R01, R10 and R11 are the image rotation amounts and image scaling amounts during affine transformation.

[0156] In some embodiments, an affine transformation is performed on the current human body image with the first human body center point as the center; for example, the current human body image is rotated counterclockwise or clockwise with the first human body center point as the center to obtain the current human body image after affine transformation.

[0157] In some embodiments, an affine transformation is performed on the current human body image with the first head center point as the center; for example, the current human body image is rotated counterclockwise or clockwise with the first head center point as the center; and then the rotated current human body image is translated to obtain the current human body image after affine transformation.

[0158] After the current human body image is affine transformed, the current human body key points in the current human body image after the affine transformation are calculated.

[0159] In some embodiments, after the current image frame is input into the posture detection model, the human body key points are obtained; based on the affine transformation of the current human body image, the human body key points are affine transformed to obtain the human body key points after affine transformation; for example, after the i-th image frame is input into the posture detection model, the human body key points i1, i2,...,in are obtained, and after the human body key points are affine transformed, i`1, i`2,...,i`n are obtained.

[0160] In some embodiments, the affine-transformed current human image is input into a posture detection model to obtain human key points, which include but are not limited to the following key points: 1. Nose, 2. Neck, 3. Right Shoulder, 4. Right Elbow, 5. Right Wrist, 7. Left Shoulder, 7. Left Elbow, 8. Left Wrist, 9. Right Hip, 10. Right Knee, 11. Right Ankle, 12. Left Hip, 13. Left Knee, 14. Left Ankle, 15. Left Eye, 16. Right Eye, 17. Left Ear, 18. Right Ear.

[0161] After obtaining the current body key points, the current body key points are compared with the comparison key points of the current comparison image frame to obtain a comparison result. The current comparison image frame is preset. For example, in an AI fitness scenario, the comparison image frame is a preset fitness image frame that represents the required movements during fitness. Similarly, the comparison key points are preset based on the comparison image frame.

[0162] The comparison result indicates the difference between the current human body key point and the comparison key point. Therefore, the display device 200 can render a comparison screen based on the comparison result and display the comparison screen to show the user the comparison result (such as the action to be adjusted, angle, etc.).

[0163] In some embodiments, when it is determined that the difference value of the current image frame is greater than or equal to the difference threshold, the current image frame is input into the target detection model to obtain the current human body area, the second head center point and the second human body center point; wherein the second head center point and the second human body center point are used to calculate the difference value of the next image frame; the current human body area is used to identify the human body image of the next image frame; the current image frame and the next image frame are adjacent image frames; the target detection model is used to identify the human body area, head center point and human body center point in the image frame.

[0164] For example, the current image frame is the i-th image frame. When the calculated difference value of the i-th image frame is greater than or equal to the difference threshold, it means that the spatial distance deviation between the first head center point in the current image frame and the historical head center point is large, and the spatial distance deviation between the first human body center point in the current image frame and the historical human body center point is large. The human body figure in the current image frame may be outside the historical human body area rather than within the historical human body area.

[0165] In other words, to ensure accurate human image recognition, if the difference value of the i-th image frame is greater than or equal to the difference threshold, the historical human region cannot be used as the current human region. Therefore, it is necessary to first input the i-th image frame into the object detection model to obtain the human region, the second head center point, and the second human center point of the i-th image frame.

[0166] Then, the human body area of the i-th image frame is cropped out to obtain the current human body area in the i-th image frame to obtain the current human body image, and the current human body area is affine transformed; the current human body image after affine transformation is then input into the posture detection model to obtain the current human body key points in the current human body image.

[0167] When recognizing the i+1th image frame, the i+1th image frame is input into the posture detection model to obtain the head center point and body center point of the i+1th image frame; the difference value of the i+1th image frame is calculated based on the head center point and body center point of the i+1th image frame and the second head center point and second body center point of the i-th image frame. The method for calculating the difference value has been described above and will not be repeated here.

[0168] If the difference value of the i+1th image frame is less than the difference threshold, the human body region of the i-th image frame is used as the human body region of the i+1th image frame, and the human body image of the i+1th image frame is cropped based on the human body region of the i-th image frame. This avoids the need to use the object detection model to obtain the human body image of the i+1th image frame. In other words, the human body image of the i+1th image frame can be obtained without the object detection model, thereby improving image recognition efficiency.

[0169] In order to better explain the above technical solution, Figure 15 An exemplary flow chart of an image recognition method is shown as follows: Figure 15 As shown, the content is as follows:

[0170] Step 1510: Capture image frames;

[0171] The display device 200 collects image frames frame by frame using its own image collection device (such as a camera); or the display device 200 receives image frames sent by a server, a mobile terminal, etc. frame by frame.

[0172] Step 1520: PE model detection;

[0173] The display device 200 inputs the i-th image frame (current image frame) into the PE model (posture detection model) to obtain the head center point (first head center point), body center point (first body center point) and body key points of the i-th image frame.

[0174] Step 1530 , determine whether the difference value of the current image frame is less than the difference threshold; if so, execute step 1540 ; otherwise, execute step 1560 .

[0175] Assuming that the historical image frame is the i-1th image frame, the difference value of the i-th image frame is calculated based on the head center point and the body center point of the i-th image frame and the head center point and the body center point of the i-1th image frame.

[0176] Among them, the difference value of the i-1th image frame is greater than or equal to the difference threshold; because the difference value of the i-1th image frame is judged to be greater than or equal to the difference threshold, the i-1th image frame is input into the OD model (target detection model) to obtain the human body area, head center point and human body center point of the i-1th image frame.

[0177] Step 1540: Identify the current human body image and determine the current human body key points.

[0178] The human body region of the i-1th image frame is used as the human body region of the i-th image frame, and the i-th human body image (ie, the current human body image) is cropped in the i-th image frame according to the human body region of the i-1th image frame.

[0179] Step 1550: Identify the current human body key points from the human body image.

[0180] The display device 200 connects the center point of the head and the center point of the body of the i-th image frame to obtain a connecting line (e.g., referred to as the i-th connecting line); and calculates the angle between the i-th connecting line and the vertical direction (e.g., referred to as the i-th angle).

[0181] When it is determined that the i-th angle is greater than the angle threshold, the i-th human body image is affine transformed, and based on the i-th human body image after affine transformation, the human body key points in the i-th human body image (ie, the current human body key points) are calculated.

[0182] In some embodiments, the key points of the human body may be key points of the human skeleton, such as wrists, ankles, hips, shoulders, head, knees, facial features, and the like.

[0183] Step 1560: OD model identification.

[0184] The i-th image frame is input into the OD model to obtain the human body region, the head center point (ie, the second head center point), and the human body center point (ie, the second human body center point) of the i-th image frame.

[0185] Among them, the second head center point and the second body center point are used to calculate the difference value of the i+1th image frame; when the difference value of the human body area of the i+1th image frame is less than the difference threshold, it is used as the human body area of the i+1th image frame to crop out the human body image of the i+1th image frame.

[0186] Step 1570 , determine whether there is a human body; if so, execute step 1580 , otherwise execute step 1510 .

[0187] After obtaining the human body region of the i-th image frame, it is identified whether there is a human body in the human body region of the i-th image frame; if there is no human body in the human body region, step 1510 is executed, that is, the i+1-th image frame is identified.

[0188] Step 1580: Crop the human body image and perform affine transformation.

[0189] Based on the human body region in the i-th image frame, the i-th human body image is cropped from the i-th image frame. Then, a connecting line is obtained based on the head center point and the body center point in the i-th human body image; and the angle between the connecting line and the vertical direction is calculated based on the connecting line.

[0190] When it is determined that the angle is greater than an angle threshold (such as ±10°), the i-th human body image is subjected to affine transformation.

[0191] Step 1590: Identify key points of the human body.

[0192] In some embodiments, the i-th human body image after affine transformation is input into the posture detection model again to obtain key points of each human body in the i-th human body image after affine transformation.

[0193] In some embodiments, the affine transformation is performed on the human body key points in step 1520 using the affine parameters used in the affine transformation of the i-th human body image, thereby obtaining the human body key points after the affine transformation.

[0194] It can be seen from the above technical solution that in the process of performing image recognition frame by frame, the display device 200 first inputs the image frame into the posture detection model (rather than the target detection model) for any image frame to obtain the first head center point and the first body center point of the image frame; then calculates the difference value of the image frame based on the first head center point, the first body center point, the historical head center point and the historical body center point.

[0195] If the difference value of the image frame is determined to be less than the difference threshold, it indicates that the spatial distance deviation between the first head center point in the current image frame and the historical head center point is small, and the spatial distance deviation between the first body center point in the current image frame and the historical body center point is small. Because the historical head center points and historical body center points are trained based on the target detection model, it can be determined that the human figure in the current image frame is within the historical body area, improving image recognition accuracy. In other words, the human image can be obtained from the historical body area, thereby avoiding the need to use the target detection model when recognizing the current image frame.

[0196] Based on the above description, when it is determined that the difference value of the image frame is less than the difference threshold, the target detection model will no longer be used to identify the image frame. This can improve the efficiency of identifying human images while ensuring the accuracy of image recognition, thereby improving the efficiency of image recognition.

[0197] Similar parts between the embodiments provided in this application can be referenced to each other. The specific implementation methods provided above are only a few examples under the overall concept of this application and do not constitute a limitation on the scope of protection of this application. For those skilled in the art, any other implementation methods expanded based on the scheme of this application without expending creative work shall fall within the scope of protection of this application.

Claims

1. A display device, characterized in that: include: monitor; The controller is configured as: Get the current image frame; Identify the current image frame according to the posture detection model, and obtain a first head center point and a first body center point in the current image frame; Calculating a difference value of the current image frame; the difference value of the current image frame is calculated by the distance between the first head center point and the historical head center point and the distance between the first body center point and the historical body center point; If the difference value of the current image frame is less than the difference threshold, identifying the current human body image of the current image frame based on the historical human body region; wherein the historical human body region, the historical head center point, and the historical human body center point are obtained by identifying the historical image frame according to the target detection model; and the difference value of the historical image frame is greater than or equal to the difference threshold; If the difference value of the current image frame is greater than or equal to the difference threshold, identifying the current image frame according to the target detection model to obtain a current human body image; Identify current human body key points from the current human body image.

2. The display device according to claim 1, wherein The controller is further configured to: In the step of calculating the difference value of the current image frame, a first distance is calculated; the first distance is the spatial distance between the first head center point and the historical head center point; Calculating a second distance; the second distance is the spatial distance between the first human body center point and the historical human body center point; The difference value of the current image frame is calculated according to the first distance and the second distance; the difference value of the current image frame represents the spatial deviation between the human body in the current image frame and the human body in the historical image frame.

3. The display device according to claim 2, wherein The controller is further configured to: Calculate the difference value of the current image frame according to the following formula (1); Wherein, e is the difference value of the current image frame; s1 is the first distance; and s2 is the second distance.

4. The display device according to claim 1, wherein The controller is further configured to: In the step of identifying the current human body image of the current image frame based on the historical human body region, the historical human body region is used as the current human body region; the historical human body region is generated with the historical human body center point as the center and a preset multiple of the distance between the historical human body center point and the historical head center point as the radius; According to the current human body region, a current human body image is identified in the current image frame.

5. The display device according to claim 1, wherein The controller is further configured to: In the step of identifying the current human body key points from the current human body image, connecting the first head center point and the first human body center point to obtain a connecting line; Calculating the angle between the connecting line and the vertical direction; If the angle is greater than the angle threshold, performing an affine transformation on the current human body image with the first human body center point as the center, and identifying the current human body key points from the current human body image after the affine transformation; If the angle is less than or equal to the angle threshold, the current human body key points are directly identified from the current human body image.

6. The display device according to claim 5, wherein: The controller is further configured to: After obtaining the current human body key point, the current human body key point is compared with the comparison key point of the comparison image frame to obtain a comparison result; the comparison image frame is a preset image frame at the moment of obtaining the current image frame; Rendering a comparison picture according to the comparison result; The display is controlled to display the comparison picture.

7. The display device according to claim 1, wherein The controller is further configured to: In the step of identifying the current image frame according to the target detection model to obtain the current human body image, the current image frame is input into the target detection model to obtain the current human body region, the second head center point and the second human body center point; The second head center point and the second body center point are used to calculate a difference value of a next image frame; The current human body region is used to identify the human body image of the next image frame; the current image frame and the next image frame are adjacent image frames; The target detection model is used to identify the human body area, the head center point and the human body center point in the image frame; The current image frame is cropped according to the current human body region to obtain a current human body image.

8. An image recognition method, characterized in that: The method is applied to a display device; The method comprises: Get the current image frame; Identify the current image frame according to the posture detection model, and obtain a first head center point and a first body center point in the current image frame; Calculating a difference value of the current image frame; the difference value of the current image frame is calculated by the distance between the first head center point and the historical head center point and the distance between the first body center point and the historical body center point; If the difference value of the current image frame is less than the difference threshold, identifying the current human body image of the current image frame based on the historical human body region; wherein the historical human body region, the historical head center point, and the historical human body center point are obtained by identifying the historical image frame according to the target detection model; and the difference value of the historical image frame is greater than or equal to the difference threshold; If the difference value of the current image frame is greater than or equal to the difference threshold, identifying the current image frame according to the target detection model to obtain a current human body image; Identify current human body key points from the current human body image.

9. The method according to claim 8, characterized in that Calculating the difference value of the current image frame includes: Calculating a first distance; the first distance is the spatial distance between the first head center point and the historical head center point; Calculating a second distance; the second distance is the spatial distance between the first human body center point and the historical human body center point; The difference value of the current image frame is calculated according to the first distance and the second distance; the difference value of the current image frame represents the spatial deviation between the human body in the current image frame and the human body in the historical image frame.

10. The method according to claim 8, characterized in that Identifying the current image frame according to the target detection model to obtain a current human body image includes: Inputting the current image frame into the target detection model to obtain a current human body region, a second head center point, and a second human body center point; the second head center point and the second human body center point are used to calculate a difference value of a next image frame; the current human body region is used to identify a human body image of the next image frame; the current image frame and the next image frame are adjacent image frames; the target detection model is used to identify the human body region, the head center point, and the human body center point in the image frame; The current image frame is cropped according to the current human body region to obtain a current human body image.

Images