Display device and human body detection method

Abstract

The embodiment of the invention provides a display device and a human body detection method, and the method comprises the steps: setting an effective detection range for a detection device, and carrying out the skeleton point detection of a human body target in a video image collected by the detection device, so as to generate a human body coordinate. And judging whether the human body target is in an effective detection range or not through the generated human body coordinates. And when the human body target is not in the effective detection range, verifying the current state of the human body target to obtain the accurate current state of the human body target. According to the method, the accuracy and stability of the human body target detection process can be improved, and the problem of poor accuracy and stability of human body detection is solved.

Description

Technical Field

[0001] This application relates to the field of display device technology, and in particular to a display device and a human body detection method. Background Technology

[0002] Intelligent sensing systems can include display devices and detection devices. They can improve the user experience and interaction efficiency of display devices. In intelligent sensing scenarios, the detection device detects human targets and sends the detection results to the display device. The display device then performs interactive judgments based on the detection results to execute appropriate control functions. For example, if the display device detects that the user is not within the effective viewing area, it can pause media playback and resume playback when the user returns to the effective viewing area.

[0003] During the detection process, detection equipment is affected by environmental factors. In actual deployment environments, the detection area of ​​cameras and other detection devices may be affected by factors such as the similarity of clothing color, texture, and material to the human body, or by dynamic changes in light, shadows, and obstructions. These factors can influence the calculation results of human detection algorithms, leading to the identification of non-human targets as human targets. Furthermore, when a human body is at the edge of the camera's field of view, the reduced proportion of the human target in the detected image, the loss of feature information, and the limitations of the algorithm model itself can cause errors in the detection of human targets by the intelligent sensing system. In some scenarios, human targets may not be identified at all, resulting in poor accuracy and stability of human detection. Summary of the Invention

[0004] Some embodiments of this application provide a display device and a human body detection method to solve the problems of poor accuracy and stability in human body detection.

[0005] In a first aspect, some embodiments of this application provide a display device, including: a display, a device interface, and a controller. The display is configured to display a user interface; the device interface is configured to connect to a detection device; and the controller is configured to perform the following steps:

[0006] In response to a control command for detecting a human target, the effective detection range of the detection device and the video images acquired by the detection device are obtained;

[0007] The human target in the video image is detected by a skeletal point detection algorithm to generate human coordinates based on the skeletal points;

[0008] A current state is generated based on the human body coordinates and the effective detection range; the current state includes a first state and a second state; the first state indicates that the human target is within the effective detection range; the second state indicates that the human target is not within the effective detection range.

[0009] When the current state is the second state, a first image set of the video images within a first time period is obtained, and human target detection is performed on the first image in the first image set, and the current state is updated according to the number of image frames containing human targets in the first image set; wherein, if the number of image frames is greater than a first preset frame number threshold, the current state is modified to the first state; if the number of image frames is less than the first preset frame number threshold, the current state is maintained as the second state.

[0010] Secondly, some embodiments of this application also provide a human body detection method, applied to the display device described in the first aspect, wherein the display device is connected to the detection device via a device interface, and the human body detection method includes:

[0011] In response to a control command for detecting a human target, the effective detection range of the detection device and the video images acquired by the detection device are obtained;

[0012] The human target in the video image is detected by a skeletal point detection algorithm to generate human coordinates based on the skeletal points;

[0013] A current state is generated based on the human body coordinates and the effective detection range; the current state includes a first state and a second state; the first state indicates that the human target is within the effective detection range; the second state indicates that the human target is not within the effective detection range.

[0014] When the current state is the second state, a first image set of the video images within a first time period is obtained, and human target detection is performed on the first image in the first image set, and the current state is updated according to the number of image frames containing human targets in the first image set; wherein, if the number of image frames is greater than a first preset frame number threshold, the current state is modified to the first state; if the number of image frames is less than the first preset frame number threshold, the current state is maintained as the second state.

[0015] As can be seen from the above technical solutions, some embodiments of this application provide a display device and a human body detection method. The method responds to a control command for detecting a human target, obtains the effective detection range of the detection device, and performs skeletal point detection on the human target in the video image acquired by the detection device using a skeletal point detection algorithm. Human coordinates are generated based on the skeletal points, and the generated human coordinates are used to determine whether the human target is within the effective detection range, thereby obtaining the current state of the human target. When the human target is not within the effective detection range, the current state of the human target is verified to obtain an accurate current state. This method can improve the accuracy and stability of the human target detection process, thus solving the problem of poor accuracy and stability in human detection. Attached Figure Description

[0016] To more clearly illustrate the technical solution of this application, the drawings used in the embodiments will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 This is a schematic diagram illustrating the usage scenarios of the display device in some embodiments of this application;

[0018] Figure 2 Hardware configuration block diagrams of display devices provided in some embodiments of this application;

[0019] Figure 3 This is a schematic diagram of the software configuration in a display device provided in some embodiments of this application;

[0020] Figure 4 Flowcharts of human body detection methods provided in some embodiments of this application;

[0021] Figure 5 This is a schematic diagram illustrating the effective detection range of the detection equipment provided in some embodiments of this application;

[0022] Figure 6 A two-dimensional rectangular coordinate system created based on the effective detection range is provided for some embodiments of this application;

[0023] Figure 7 A flowchart illustrating a method for generating human body coordinates provided in some embodiments of this application;

[0024] Figure 8 A schematic diagram of a virtual character drawn based on human skeletal points, provided for some embodiments of this application;

[0025] Figure 9 A flowchart illustrating a method for real-time updating of the current state provided in some embodiments of this application;

[0026] Figure 10 A flowchart illustrating a method for verifying the current state provided in some embodiments of this application;

[0027] Figure 11 This application provides flowcharts of methods for obtaining human target results in some embodiments;

[0028] Figure 12 The flowcharts for human body detection provided in some embodiments of this application are shown below. Detailed Implementation

[0029] The embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described below do not represent all embodiments consistent with this application. They are merely examples of systems and methods consistent with some aspects of this application as detailed in the claims.

[0030] It should be noted that the brief descriptions of terms in this application are only for the convenience of understanding the embodiments described below, and are not intended to limit the embodiments of this application. Unless otherwise stated, these terms should be understood in their ordinary and common meaning.

[0031] The terms "first," "second," "third," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar or related objects or entities, and do not necessarily imply a specific order or sequence, unless otherwise specified. It should be understood that such terms are interchangeable where appropriate.

[0032] The terms “comprising” and “having”, and any variations thereof, are intended to cover but not exclude inclusion, for example, a product or device that includes a range of components is not necessarily limited to all of the components that are clearly listed, but may include other components that are not clearly listed or that are inherent to such product or device.

[0033] The term "module" refers to any known or subsequently developed hardware, software, firmware, artificial intelligence, fuzzy logic, or combination of hardware and / or software code that is capable of performing the functions associated with that element.

[0034] In this embodiment, display device 200 generally refers to a device with screen display and data processing capabilities. For example, display device 200 includes, but is not limited to, smart TVs, mobile terminals, computers, monitors, advertising screens, wearable devices, virtual reality devices, augmented reality devices, etc.

[0035] Figure 1 This is a schematic diagram illustrating an operational scenario between a display device and a control device provided in some embodiments of this application. For example... Figure 1As shown, a user can operate the display device 200 via touch operation, a mobile terminal 300, and a control device 100. The control device 100 receives user input commands and converts them into control commands that the display device 200 can recognize and respond to. For example, the control device 100 can be a remote control, a stylus, a gamepad, etc.

[0036] In some embodiments, the control device 100 may be a remote control, and the communication between the remote control and the display device includes at least one of infrared protocol communication, Bluetooth protocol communication, and other short-range communication methods, controlling the display device 200 wirelessly or via a wired connection. Users can control the display device 200 by inputting user commands through buttons on the remote control, voice input, control panel input, etc.

[0037] The mobile terminal 300 can function as a control device for human-computer interaction between the user and the display device 200. It can also function as a communication device for establishing a communication connection with the display device 200 and exchanging data. In some embodiments, the mobile terminal 300 can have software applications installed on it and communicate with the display device 200 via network communication protocols to achieve one-to-one control and data communication. Furthermore, it can transmit audio and video content displayed on the mobile terminal 300 to the display device 200 for synchronized display.

[0038] In some embodiments, the mobile terminal 300 or other electronic devices may also simulate the functions of the control device 100 by running an application that controls the display device 200.

[0039] like Figure 1 The diagram also shows that the display device 200 communicates with the server 400 via various communication methods. This allows the display device 200 to communicate via a local area network (LAN), a wireless local area network (WLAN), and other networks.

[0040] Display device 200 can provide broadcast television reception function, and can also be equipped with intelligent network television function that provides computer support, including but not limited to network television, smart television, Internet Protocol television (IPTV), etc.

[0041] Figure 2 This is a hardware configuration block diagram of a display device 200 provided in some embodiments of this application.

[0042] In some embodiments, the display device 200 may include at least one of a tuner 210, a communication device 220, a detector 230, a device interface 240, a controller 250, a display 260, an audio output device 270, a memory, a power supply, and a user input interface.

[0043] In some embodiments, detector 230 is used to acquire signals from the external environment or to interact with the outside world. For example, detector 230 includes a light receiver, a sensor for acquiring ambient light intensity; or, detector 230 includes an image acquisition device, such as a camera, which can be used to acquire external environmental scenes, user attributes, or user interaction gestures; or, detector 230 includes a sound acquisition device, such as a microphone, for receiving external sounds.

[0044] In some embodiments, the display 260 includes display function components for presenting images and driving components for driving image display. The display 260 is used to receive and display image signals output from the controller 250. For example, the display 260 can be used to display video content, image content, menu control interface components, and user control UI interfaces, etc.

[0045] In some embodiments, the communication device 220 is a component used to communicate with external devices or the server 400 according to various communication protocol types. The display device 200 may have multiple communication devices 220 depending on the supported communication methods. For example, when the display device 200 supports wireless network communication, it may have a communication device 220 with WiFi functionality. When the display device 200 supports Bluetooth connectivity, it needs to have a communication device 220 with Bluetooth functionality.

[0046] The communication device 220 enables the display device 200 to communicate with external devices or the server 400 via wireless or wired connections. Wired connections utilize data cables, interfaces, or other components to connect the display device 200 to external devices. Wireless connections utilize wireless signals or wireless networks. The display device 200 can directly establish a connection with external devices or indirectly through gateways, routers, or other connection devices.

[0047] In some embodiments, the device interface 240 can be an HDMI (High-Definition Multimedia Interface), a VGA (Video Graphics Array) interface, a DVI (Digital Visual Interface) interface, etc. An HDMI interface is a digital audio and video interface that can transmit high-definition video and audio signals. A VGA interface is an analog signal interface that supports higher resolutions. A DVI interface is a digital video interface whose main characteristic is the transmission of digital signals, enabling the output of high-definition video.

[0048] In some embodiments, the controller 250 may include at least one of a central processing unit, a video processor, an audio processor, a graphics processor, and a power processor, and a first to an nth interface for input / output. The controller 250 controls the operation of the display device and responds to user operations through various software control programs stored in memory. The controller 250 controls the overall operation of the display device 200.

[0049] In some embodiments, the controller 250 and the tuner 210 may be located in different separate devices, that is, the tuner 210 may also be located in an external device of the main device where the controller 250 is located, such as an external set-top box.

[0050] In some embodiments, a user can input user commands through a graphical user interface (GUI) displayed on a display 260, and the user input interface receives user input commands through the graphical user interface (GUI).

[0051] In some embodiments, the audio output device 270 can be a built-in speaker of the display device 200 or an external audio output device connected to the display device 200. For the external audio output device connected to the display device 200, the display device 200 may also be provided with an external audio output terminal, through which the audio output device can be connected to the display device 200 to output sound from the display device 200.

[0052] In some embodiments, the user input interface 280 can be used to receive instructions from user input.

[0053] In some embodiments, the display device 200 can establish a communication connection with the detection device 500. The detection device 500 is used to acquire video images. In some embodiments, the detection device 500 can be an external camera. The detection device 500 can capture video streams within its effective detection range in real time, convert the video streams into digital signals, and represent them in the form of frames, each frame being an independent video image. During the detection process, the detection device 500 continuously acquires new video images for processing by the controller 250. The detection device 500 can also be a built-in camera installed inside the display device 200, such as a camera integrated with the display 260.

[0054] To enable user interaction, in some embodiments, the display device 200 may run an operating system. The operating system is a computer program used to manage and control the hardware and software resources of the display device 200. The operating system can control the display device to provide a user interface; for example, the operating system can directly control the display device to provide a user interface, or it can provide a user interface by running an application. The operating system also allows users to interact with the display device 200.

[0055] It should be noted that the operating system can be a native operating system based on a specific operating platform, a third-party operating system that is deeply customized based on a specific operating platform, or an independent operating system specifically developed for the display device 200.

[0056] An operating system can be divided into different modules or levels based on the functions it implements, for example... Figure 3 As shown, in some embodiments, the operating system is divided into four layers, from top to bottom: the Applications layer (referred to as the "Application Layer"), the Application Framework layer (referred to as the "Framework Layer"), the System Library layer, and the Kernel layer.

[0057] In some embodiments, the application layer provides services and interfaces for applications, enabling the display device 200 to run applications and interact with the user based on the applications. The application layer may contain at least one application, which may be a built-in Windows program, system settings program, or clock program of the operating system; or it may be an application developed by a third-party developer. In specific implementations, the application packages in the application layer are not limited to the examples above.

[0058] The framework layer provides application programming interfaces (APIs) and a programming framework for applications. The application framework layer includes predefined functions. It acts as a central processing unit, determining the actions taken by applications within the application layer. Through the API, applications can access system resources and obtain system services during execution.

[0059] like Figure 3As shown, the application framework layer in this embodiment includes a view system, managers, and content providers. The view system designs and implements the application's interface and interactions, and includes lists, grids, text boxes, and buttons. The managers include at least one of the following modules: an activity manager for interacting with all running activities in the system; a location manager for providing system services or applications with access to system location services; a package manager for retrieving various information related to application packages currently installed on the device; a notification manager for controlling the display and clearing of notification messages; and a window manager for managing icons, windows, toolbars, wallpapers, and desktop widgets on the user interface.

[0060] In some embodiments, the Activity Manager manages the lifecycle of individual applications and common navigation and back functions, such as controlling application exit, opening, and back actions. The Window Manager manages all window programs, such as obtaining the screen size, determining if a status bar is present, locking the screen, capturing the screen, and controlling changes to the display window, such as shrinking the display window, shaking the display, or distorting the display.

[0061] In some embodiments, the system runtime library layer can provide support for the framework layer. When the framework layer is used, the operating system runs the instruction library contained in the system runtime library layer, such as the C / C++ instruction library, to implement the functions to be performed by the framework layer.

[0062] In some embodiments, the kernel layer is a functional layer situated between the hardware and software of the display device 200. The kernel layer can implement functions such as hardware abstraction, multitasking, and memory management. For example, ... Figure 3 As shown, hardware drivers can be configured in the kernel layer. The drivers included in the kernel layer can be at least one of the following: audio driver, display driver, Bluetooth driver, camera driver, WIFI driver, USB driver, HDMI driver, sensor driver (such as fingerprint sensor, temperature sensor, pressure sensor, etc.), and power driver, etc.

[0063] It should be noted that the above examples are merely a simple division of operating system functions and do not limit the specific form of the operating system of the display device 200 in this application embodiment. Depending on the function of the display device, the type of operating system, and other factors, the number of levels and the specific level type of the operating system may be expressed in other forms.

[0064] In some embodiments, the display device 200 can perform interactive control based on the detection results of human targets. The detection device 500 is mounted on the display 260 and detects human targets within its detection range. The controller 250 controls the display 260 to continue playing the current video or pause playback based on the received detection results. If a human target is detected, the controller 250 controls the display 260 to display the video playback; if no human target is detected, the controller 250 sends a pause control command to cause the display 260 to pause video playback; if the human target reappears within the detection range of the detection device 500, the controller 250 sends a playback control command again to cause the display 260 to continue playing the video playback, thereby achieving interactive control. Since human target detection is required during interactive control, the accuracy and stability of the detection results directly affect the effectiveness of the interactive control.

[0065] To improve the accuracy and stability of human target detection results, some embodiments of this application provide a human detection method. This method can run on a display device 200, enabling the display device 200 to intelligently play or pause the video playback based on whether the user is within the effective detection range of the detection device 500. For this purpose, the display device 200 may include a display 260, a device interface 240, and a controller 250. The display 260 is configured to display a user interface. The device interface 240 is configured to connect to the detection device 500; in some embodiments, the detection device 500 is positioned centrally on the display 260.

[0066] See Figure 4 The controller 250 is configured to perform the steps corresponding to the human body detection method, including the following:

[0067] S100: In response to control commands for detecting human targets, acquire the effective detection range of the detection device and the video images acquired by the detection device.

[0068] During operation, the display device 200 can receive control commands input by the user. Different control commands can control the display device 200 to perform different control actions. Some control commands can be used to control the display device 200 to detect human targets. In some embodiments, control commands for detecting human targets can be input through the control device 100. For example, after the user presses the "OK" button on the control device 100 (such as a remote control), the display device 200 can be controlled to play the media data selected by the focus marker and display the video playback screen on the monitor 260, while simultaneously issuing control commands for detecting human targets.

[0069] Because the proportion of human targets at the edge of the detection device 500 in the detected image decreases, resulting in the loss of acquired image feature information, the controller 250 in the display device 200 can, in response to the control command for detecting human targets, obtain the effective detection range of the detection device 500 after receiving the control command. Within the effective detection range, the detection device 500 can provide accurate and stable image feature information. See also Figure 5 , Figure 5 This is a schematic diagram of the effective detection range of the testing equipment. The effective detection range of the testing equipment 500 is a fan-shaped area, which includes a first boundary and a second boundary with the testing equipment 500 as the origin.

[0070] Taking a television as an example, when a user watches a video on the television, whether standing, sitting, or lying down, the data detected in the vertical direction will not affect the detection results of the detection device 500 regarding the user's position. Therefore, a two-dimensional Cartesian coordinate system with the detection device 500 as the origin can be constructed based on the first and second boundaries. That is, in some embodiments, the plane of the two-dimensional Cartesian coordinate system is perpendicular to the vertical direction. See also... Figure 6 , Figure 6 In some embodiments, the first and second boundaries of the two-dimensional Cartesian coordinate system created based on the effective detection range are symmetric about the y-axis.

[0071] The first boundary can be set to BorderA, and the two points on the first boundary are point1(0, 0) and point2(x). A y A The second boundary can be set to BorderB, and the two points on the second boundary are point3(0, 0) and point4(x). B y B ), where point2(x A y A ) and point4(x B y BLet y be two points on the edge of the 500 detection device. Then the first boundary BorderA can be represented as y A =k A x A +b A The second boundary, BorderB, can be represented as y B =k B x B +b B .according to Figure 6 It is known that in this application b A and b B All are equal to 0. Points on the first boundary (BorderA) and the second boundary (BorderB) can be calculated using the expressions for these boundaries, for example... Figure 6 Points A(-960, 1080) and B(960, 1080) are given.

[0072] In some embodiments, in order to calculate the boundary equation y = kx + b, the display device 200 may first execute "public static Line calculateline(Point point1, point point2)" to define a public static calculation line, then execute "double slope = (point2.y - point1.y) / (point2.x - point1.x)" to calculate the slope of the boundary equation, then execute "double intercept = point1.y - slope * point1.x" to calculate the intercept of the boundary equation, and finally return the equation of the line, i.e., "return new Line(slope, intercept)".

[0073] While acquiring the effective detection range of the detection device 500, the display device 200 also acquires video images captured by the detection device 500. The video images are images from the real-time video stream captured by the detection device 500. Since the user may or may not be within the effective detection range when the detection device 500 acquires the real-time video stream, the video images include images containing human targets and images not containing human targets. The real-time video stream acquired by the detection device 500 includes video images within the effective detection range and video images outside the effective detection range. The edges outside the first and second boundaries are considered outside the effective detection range of the detection device 500, and feature information at these edges may be missing in the acquired real-time video stream.

[0074] In some embodiments, after the controller 250 obtains the effective detection range of the detection device 500 and the video images captured by the detection device 500, it can also obtain the confidence level of the detected human target before performing skeletal point detection on the human target in the video image using the skeletal point detection algorithm. It should be noted that when using the confidence level to detect human targets, a higher confidence level will reduce the recognition rate of human targets while increasing the accuracy of human target recognition.

[0075] The confidence level for detecting human targets includes a first confidence level and a second confidence level. The first confidence level is the default confidence level for performing human target skeletal point detection on a single frame of video image, and is a constant value set in the display device 200. The second confidence level is a value set based on the first confidence level. To improve the stability of the detection results, the display device 200 can optimize the confidence level for detecting human targets by lowering the confidence level to improve the recognition rate of human targets. Specifically, the second confidence level can be set to a value lower than the first confidence level. For example, if the first confidence level is 0.7, the second confidence level can be set to 0.4, reducing the confidence level from 0.7 to 0.4, thereby improving the recognition rate of human targets. The drawback of lowering the confidence level is that it also increases the false recognition rate of human targets. Therefore, recognizing a human target can be used as the first condition for detecting a user, and based on the second confidence level, skeletal point detection can be performed on human targets in the video image using a skeletal point detection algorithm to further identify human targets in the video image.

[0076] S200: Performs skeletal point detection on human targets in video images using a skeletal point detection algorithm to generate human coordinates based on the skeletal points.

[0077] See Figure 7 In some embodiments, the controller 250 performs skeletal point detection on the human target in the video image using a skeletal point detection algorithm to generate human coordinates based on the skeletal points. That is, the display device 200 can first acquire the real-time video stream detected by the detection device (S710), and then perform frame segmentation processing on the video stream to obtain multiple frames of video images (S720). After performing frame segmentation processing on the video stream, the video stream is decomposed into consecutive frames, each of which is the aforementioned video image. In some embodiments, to improve the running efficiency and detection accuracy of the skeletal point detection algorithm, each frame of video image can be preprocessed, including scaling, cropping, and normalization, to obtain multiple preprocessed frames of images.

[0078] After acquiring the video image, the display device 200 can perform human target detection on the video image using a human bounding box detection model to obtain a target image (S730). In some embodiments, human target detection can be performed on the preprocessed video image using the YOLOv5 human bounding box detection model to obtain a target image containing human targets. When performing human target detection, the identified human targets are selected by rectangular bounding boxes, meaning that the location of the human target in the acquired target image is simultaneously displayed with a human bounding box. Video images that do not contain human targets are invalid data and are discarded after detection.

[0079] In some embodiments, after acquiring a target image containing a human object, feature extraction is performed on the target image using a skeletal point detection model to obtain skeletal point coordinates (S740). The skeletal point detection model is a neural network model trained using sample skeletal point images, which include sample images and their skeletal point labels. The skeletal point detection algorithm is the algorithm within the skeletal point detection model. By using the skeletal point detection model to extract features from the target image containing the human object, the contours of the human object and human skeletal points in the image can be identified. See also... Figure 8 , Figure 8 This is a schematic diagram of a virtual character drawn based on human skeletal points. The obtained skeletal points include: nose (0), right shoulder (1), left shoulder (2), right elbow (3), left elbow (4), right wrist (5), left wrist (6), right hip (7), left hip (8), right knee (9), left knee (10), right ankle (11), left ankle (12), and neck (13). In some embodiments, after identifying the skeletal points, the positions of the human skeletal points can be predicted using regression models or heatmap prediction methods. A regression model is a predictive modeling technique that studies the relationship between dependent and independent variables through regression analysis, and can predict the position of the skeletal points relative to the human body based on their characteristics. After predicting the positions of the skeletal points, the coordinates of the human skeletal points can be obtained using a skeletal point detection algorithm.

[0080] In some embodiments, in order to define the coordinates of a skeleton point, the display device 200 may first execute "public static class skeletonPoint" to define a public static class skeleton point, then execute "float x; float y; boolean is_detect; public SkeletonPoint(float x, float y, boolean is_detect)" to determine whether the skeleton point has been detected, and finally execute "this.x = x; this.y = y; this.is_detect = is_detect" to assign values ​​to the coordinates of the skeleton point.

[0081] In some embodiments, bone points can be connected based on the predicted bone point locations to link the bone points into the human skeletal structure. Figure 8 The system detects a virtual character and displays the results of the skeletal point detection on the monitor 260 of the display device 200, allowing users to intuitively observe the results. Post-processing, such as denoising and smoothing, can also be applied to the predicted skeletal point positions to improve the stability and accuracy of skeletal point detection.

[0082] Based on the acquired skeletal point coordinates, key skeletal point coordinates can be extracted (S750). In some embodiments, the extracted key skeletal point coordinates include nose coordinates, right shoulder coordinates, left shoulder coordinates, and neck coordinates. When a user watches a video displayed on the monitor 260, for example, while seated, they may stretch or extend their legs, causing changes in the position of their elbows, wrists, or knees. This results in inaccurate human body coordinates obtained from skeletal points such as the right wrist, left wrist, right knee, and left knee. However, when the head, shoulders, and neck change position due to movement, the displacement changes are smaller, meaning the human body coordinates obtained from nose 0, right shoulder 1, left shoulder 2, and neck 13 are more stable. Therefore, the controller extracts nose coordinates, right shoulder coordinates, left shoulder coordinates, and neck coordinates as key skeletal point coordinates.

[0083] After obtaining the coordinates of key skeletal points, human body coordinates can be generated based on these coordinates (S760). When the controller 250 generates human body coordinates based on the key skeletal point coordinates, it needs to traverse the key skeletal point coordinates through a skeletal point detection model to detect whether the coordinates of the nose, right shoulder, left shoulder, and neck are all present. In some embodiments, if the obtained key skeletal point coordinates include a first feature coordinate and a second feature coordinate, then the first center point coordinates of the first and second feature coordinates are calculated, and human body coordinates are generated based on the first center point coordinates. The first and second feature coordinates are used to represent two mutually symmetrical joints in the human body target; the first feature coordinate can be set as the left shoulder coordinate, and the second feature coordinate as the right shoulder coordinate. For example, if the left and right shoulder coordinates are obtained simultaneously while the user is watching a video, the center point coordinates of the left and right shoulder coordinates are output as human body coordinates.

[0084] In some embodiments, if the obtained key skeletal point coordinates include one of the first feature coordinates and the second feature coordinates, calculate the second center coordinates of the remaining skeletal point coordinates, and generate the human body coordinates according to the second center coordinates, where the remaining skeletal point coordinates are the key skeletal point coordinates in the video image except the first feature coordinates or the second feature coordinates. Exemplarily, during the process of a user watching a video frame, the user may cover the positions of the key skeletal points. For example, cover the positions of the left shoulder and the right shoulder with a quilt or a blanket, or the user watches the video frame in a side-lying position. At this time, since the right shoulder 1 skeletal point and the left shoulder 2 skeletal point cannot be detected, or only one of the right shoulder 1 skeletal point and the left shoulder 2 skeletal point can be detected, it is impossible to obtain the left shoulder coordinates and the right shoulder coordinates simultaneously. At this time, calculate the remaining key skeletal point coordinates, that is, the center coordinates of the nose coordinate and the neck coordinate to obtain the human body coordinates. Since the user may watch the video frame in a tilted head position, therefore, by using the second center coordinates of the remaining key skeletal point coordinates as the human body coordinates, the accuracy of the human body coordinate result can be improved.

[0085] S300: Generate the current state according to the human body coordinates and the effective detection range.

[0086] In some embodiments, the current state includes a first state and a second state. The first state is used to represent that the human body target is within the effective detection range, and the second state is used to represent that the human body target is not within the effective detection range. According to the content in step S100 above, it can be known that the detection ability of the detection device 500 is reduced at the edges near the first boundary and the second boundary outside the effective detection range. Therefore, the second state also represents that the human body target is at the edge near the first boundary or the second boundary.

[0087] The obtained human body coordinates include the human body abscissa and the human body ordinate. The human body coordinates (x, y) can be set, and the human body abscissa x and the human body ordinate y are extracted from the human body coordinates (x, y). Referring again to Figure 6 , Figure 6 the dot in represents the human body coordinate point, that is, the human body target. The same human body ordinate y can correspond to different human body abscissas x. Then, according to the human body ordinate y, the first abscissa of the first boundary and the second abscissa of the second boundary can be obtained. The first abscissa can be set as x1, and the second abscissa can be set as x2. If the human body abscissa x is between the first abscissa x1 and the second abscissa x2, that is, x1 < x < x2, then mark the current state of the human body target as the first state, that is, the human body target is within the effective detection range. If the human body abscissa x is not between the first abscissa x1 and the second abscissa x2, that is, x ≤ x1 or x ≥ x2, then mark the current state as the second state, that is, the human body target is not within the effective detection range.

[0088] For example, when a user is near the first or second boundary, if the user is lying down while watching the video, and the user's shoulders and head are within the effective detection range of the detection device 500, but the legs and ankles are not, then the user's right knee 9, left knee 10, right ankle 11, and left ankle 12 skeletal points will not be detected. However, key skeletal points such as the user's nose 0, right shoulder 1, left shoulder 2, or neck 13 can be detected. After obtaining the human body coordinates based on the coordinates of the key skeletal points, the position of the human body's horizontal coordinate is determined based on the human body's vertical coordinate. If the human body's horizontal coordinate is between the first and second boundaries, then the current state of the human target is marked as the first state, that is, the human target is within the effective detection range. At this time, even if the user's legs and ankles are not within the effective detection range, the video on the display 260 will continue to play.

[0089] See Figure 9 , Figure 9 The flowchart for the method of real-time updating the current state describes a process where, in some embodiments, after acquiring the current state of the human target, the user may move within or leave the effective detection range, necessitating real-time updates. The process includes initializing human coordinates and boundary equations (S910), as well as key skeletal point coordinates (S920). When the skeletal point detection model runs, the algorithm returns to the step of acquiring skeletal point coordinates and generates human coordinates using the key skeletal point coordinates. Human coordinates can be acquired periodically according to a second duration (S931), and it is determined whether the human coordinates are within the effective detection range. The current state of the human target is then updated in real-time based on these coordinates to either the first state (S941) or the second state (S942). If detection is paused, the skeletal point detection algorithm does not return to the step of acquiring skeletal point coordinates, and the acquired current state is maintained (S932). In some embodiments, to improve the real-time performance of the updated current state, the second duration can be set to 500ms.

[0090] In some embodiments, the current state further includes a third state and a fourth state. The third state characterizes a state where the human target is within the effective detection range and the distance between the human target and the first or second boundary gradually decreases. The fourth state characterizes a state where the human target transitions from the first state to the second state. During the process of updating the current state, assuming the current state is the first state, the target image can be acquired periodically according to the second time interval, the coordinates of skeletal points in the target image can be detected, and human coordinates can be generated based on the coordinates of key skeletal points in the skeletal point coordinates. The human horizontal coordinate is then extracted from the human coordinates. When the difference between the human horizontal coordinate and the first or second horizontal coordinate is less than a coordinate threshold, the current state is marked as the third state, that is, the human target is within the effective detection range and close to the edge of the first or second boundary.

[0091] For example, if a user moves towards the first or second boundary while watching a video within the effective detection range and reaches the edge of either boundary, the current state is marked as the third state. At this time, skeletal point feature extraction is performed on the target image to detect skeletal points. When the human target is on the first or second boundary, the image feature information acquired by the detection device 500 may be missing, making the detection of skeletal points unstable. For example, there may be a brief period of missing skeletal point features. Therefore, if no skeletal point features are extracted within the second time period, it indicates that the user is moving out of the boundary, and the current state is marked as the fourth state, i.e., the state where the human target transitions from within the effective detection range to outside the effective detection range.

[0092] In some embodiments, when a user is watching a video within the effective detection range, if skeletal points are covered, the detection device 500 cannot detect the skeletal features of the human target and therefore cannot obtain the human coordinates. In this case, the current state of the human target should be determined using historical states. Historical states are states generated in the previous detection cycle. When the controller 250 obtains the current state as the first state, it performs skeletal feature extraction on the human target in the video image and obtains the historical state. If no skeletal features are extracted in the video image and the historical state is not the fourth state, the current state is maintained as the first state, meaning that the user is still within the effective detection range of the detection device 500.

[0093] When the acquired current state is the first state, the user is within the effective detection range of the detection device 500, and the controller 250 controls the display 260 to continuously play the video, without needing to activate the intelligent pause or intelligent energy-saving function. At this time, it is only necessary to update the current state of the human target in real time according to the steps S310-S342 described above. However, when the acquired current state of the human target is the second state, because the detection result of the detection device 500 is inaccurate, it is necessary to further detect whether a human target exists within the effective detection range to verify the accuracy of the acquired current state.

[0094] S400: When the current state is the second state, obtain the first image set of video images within the first duration, perform human target detection on the first image in the first image set, and update the current state according to the number of image frames in the first image set that contain human targets.

[0095] Specifically, if the number of image frames is greater than the first preset frame count threshold, the current state is changed to the first state; if the number of image frames is less than the first preset frame count threshold, the current state is kept as the second state.

[0096] The first image is a video image used for human target detection, and the first image set is the set of first images to be detected within a first time period. See also Figure 10 In the second state, after acquiring the first image set of the video image (S1010), human target detection is performed on the first image of the F frames in the first image set (S1020) to obtain the number of image frames f containing human targets in the first image set.

[0097] The number of image frames f containing human targets is compared with a first preset frame threshold to obtain the results of human targets within the effective detection range, thus verifying the accuracy of the current state. The first preset frame threshold can be set according to the total number of frames F of the first image in which human target detection is performed. In some embodiments, the first preset frame threshold can be set to 0.5F. If the number of image frames containing human targets in the first image set is greater than or equal to the first preset frame threshold, i.e., f ≥ 0.5F, then the result of human targets within the effective detection range is marked as having human targets; if the number of image frames containing human targets in the first image set is less than the first preset frame threshold, i.e., f < 0.5F, then the result of human targets within the effective detection range is marked as having no human targets. When the result of human targets within the effective detection range is having human targets, it means that the user is within the effective detection range, and the previously obtained current state was an incorrect state. At this time, the current state needs to be corrected and changed to the first state (S1031). When the result of human targets within the effective detection range is having no human targets, it means that the user is not within the effective detection range, and the obtained current state is correct. The current state is maintained as the second state (S1032).

[0098] In some embodiments, when performing human target detection on the first image in the first image set, the result of human target within the effective detection range can also be obtained by detecting the number of skeletal points in the human target. That is, a first image containing a human target is obtained, skeletal point detection is performed on the first image containing the human target to obtain the number of skeletal points, and if the number of detected skeletal points is greater than the number threshold, the result of human target within the effective detection range is output as "there is a human target", and the current state is output as the first state.

[0099] In some embodiments, after the current state is modified to the first state, the detection of whether there is a human target within the effective detection range can continue to be performed by the time window detection method. The time window detection method obtains the human target result by detecting the number of image frames containing human targets at different times, and then obtains the current state of the human target. There is no need to obtain the current state of the human target through the skeleton point detection model and skeleton point detection algorithm, which can reduce the operating pressure of the display device 200.

[0100] See Figure 11In some embodiments, after modifying the current state to the first state or maintaining the current state to the second state, the controller 250 can continue to detect human targets within the effective detection range using a time window detection method. When the current state is modified to the first state, a second image set of video images within a third duration is obtained (S1111), and human target detection is performed on the second images in the second image set (S1121). If the number of image frames containing human targets is less than a second preset frame number threshold, it can be determined that there are no human targets, and the current state is modified to the second state (S1131). When there are no human targets within the effective detection range, the judgment logic for the state from no human targets to having human targets within the effective detection range is triggered, that is, a third image set of video images within a first duration is obtained, and human target detection is performed on the third images in the third image set; if the number of image frames containing human targets in the third image is greater than or equal to the first preset frame number threshold, the current state is modified to the first state again (S1112-S1132). It should also be noted that the first image set, the second image set, and the third image set are sets of video images acquired in different time periods.

[0101] While the current state remains in the second state, a third image set of video images within the first duration is acquired (S1112), and human target detection is performed on the third image in the third image set (S1122). If the number of image frames containing a human target is greater than or equal to a first preset frame count threshold, the result of the human target can be determined as having a human target, and the current state is modified to the first state (S1132). When there is a human target within the effective detection range, the judgment logic for the state from having a human target to not having a human target within the effective detection range is triggered, that is, the second image set of video images within the third duration is acquired; human target detection is performed on the second image in the second image set; if the number of image frames containing a human target in the second image is less than a second preset frame count threshold, the current state is modified to the second state (S1111-S1131). Since the current state can characterize whether there is a human target within the effective detection range, the controller 250 can control the display 260 to pause or resume playback according to the acquired current state. It can be understood that this process is a cyclic judgment process.

[0102] In some embodiments, the third duration should be greater than the first duration, and the second preset frame count threshold should be less than the first preset frame count threshold. The first duration can be 1 second, the third duration can be 10 seconds, and the second preset frame count threshold can be 0.04F. For example, the detection device is set to acquire 5 frames of video images per second. When the current state is the first state, 50 frames of video images are acquired within 10 seconds. Human target detection is performed on these 50 frames of video images. If the number of frames containing a human target is 1 (less than the second preset frame count threshold), the current state is output as the second state, triggering a judgment on the state from no human target to having a human target within the effective detection range, and the detection of human targets within the effective detection range continues. When the current state is the second state, 5 frames of video images are acquired within 1 second, and human target detection is performed on these 5 frames of video images. If the number of frames containing a human target is 5 (greater than the first preset frame count threshold), the current state is output as the first state, triggering a judgment on the state from having a human target to not having a human target within the effective detection range, and the detection of human targets within the effective detection range continues. It should also be noted that the values ​​of the first duration, the third duration, the first preset frame rate threshold, and the second preset frame rate threshold mentioned above are only examples and can be set to other values. For example, the second preset frame rate threshold can also be 0.1fF.

[0103] In some embodiments, APPEAR represents the presence of a human target and DISAPPEAR represents the absence of a human target. When generating the current state of a human target based on the human target results, the display device 200 can first execute "private void checkUserAppear()" to detect a human target, and then execute "boolean stateChanged = mPoseDetectedTotal > 0 && (mPoseDetectedOK / mPoseDetectedTotal >= PLAYER_DETECT_RATE)" to check whether the total number of human target detections is greater than 0 and whether the successful detection rate has reached a preset threshold. After that, "resetPoseDetectStatistic()" is executed to reset the position detection. If a human target appears, "mBodystate = BodyState, APPEAR" is executed, and the current state of the human target is modified to have a human target. Then, the judgment logic from having a human target to not having a human target is called "callCheckDisappearMessage()".

[0104] See Figure 12According to the above embodiment, the display device 200 uses the reduced confidence level as a benchmark to perform human target detection through a human body bounding box detection model, generates human body bounding box and target image, performs skeletal point detection through a skeletal point detection model to generate skeletal point coordinates, generates human body coordinates based on skeletal point coordinates, and generates the current state of the human target based on the human body coordinates. Finally, the detection results of the current state are used to perform cyclic detection of human target results through a time window detection method, so as to pause and resume the video screen played on the display 260 according to the human target results.

[0105] Based on the aforementioned display device 200, some embodiments of this application also provide a human body detection method, applied to the display device 200 provided in the above embodiments. The display device 200 is connected to the detection device 500 via a device interface 240. The human body detection method includes:

[0106] In response to a control command for detecting a human target, the effective detection range of the detection device and the video images acquired by the detection device are obtained;

[0107] The human target in the video image is detected by a skeletal point detection algorithm to generate human coordinates based on the skeletal points;

[0108] A current state is generated based on the human body coordinates and the effective detection range; the current state includes a first state and a second state; the first state indicates that the human target is within the effective detection range; the second state indicates that the human target is not within the effective detection range.

[0109] When the current state is the second state, a first image set of the video images within a first time period is obtained, and human target detection is performed on the first image in the first image set, and the current state is updated according to the number of image frames containing human targets in the first image set; wherein, if the number of image frames is greater than a first preset frame number threshold, the current state is modified to the first state; if the number of image frames is less than the first preset frame number threshold, the current state is maintained as the second state.

[0110] As can be seen from the above technical solutions, some embodiments of this application provide a display device and a human body detection method. The method responds to a control command for detecting a human target, obtains the effective detection range of the detection device, and performs skeletal point detection on the human target in the video image acquired by the detection device using a skeletal point detection algorithm. Human coordinates are generated based on the skeletal points, and the generated human coordinates are used to determine whether the human target is within the effective detection range, thereby obtaining the current state of the human target. When the human target is not within the effective detection range, the current state of the human target is verified to obtain an accurate current state. This method can improve the accuracy and stability of the human target detection process, thus solving the problem of poor accuracy and stability in human detection.

[0111] The same or similar parts among the various embodiments in this specification can be referred to mutually, and will not be repeated here.

[0112] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application.

[0113] For ease of explanation, the above description has been provided in conjunction with specific embodiments. However, the above exemplary discussion is not intended to be exhaustive or to limit the embodiments to the specific forms disclosed above. Various modifications and variations can be obtained based on the above teachings. The selection and description of the above embodiments are for the purpose of better explaining the principles and practical applications, thereby enabling those skilled in the art to better utilize the described embodiments and various different variations of embodiments suitable for specific use considerations.

Claims

1. A display device, characterized in that, include: The monitor is configured to display the user interface; The device interface is configured to connect to testing equipment; The controller is configured as follows: In response to a control command for detecting a human target, the effective detection range of the detection device and the video images acquired by the detection device are obtained; The human target in the video image is detected by a skeletal point detection algorithm to generate human coordinates based on the skeletal points; The current state is generated based on the human body coordinates and the effective detection range; The current state includes a first state and a second state; The first state is used to characterize the human target within the effective detection range; The second state is used to characterize that the human target is not within the effective detection range; When the current state is the second state, a first image set of the video images within a first time period is obtained, and human target detection is performed on the first image in the first image set, and the current state is updated according to the number of image frames containing human targets in the first image set; wherein, if the number of image frames is greater than a first preset frame number threshold, the current state is modified to the first state; if the number of image frames is less than the first preset frame number threshold, the current state is maintained as the second state.

2. The display device according to claim 1, characterized in that, Before performing skeletal point detection on the human target in the video image using the skeletal point detection algorithm, the controller is also configured to: Obtain the confidence level for detecting human targets; the confidence level includes a first confidence level and a second confidence level, wherein the first confidence level is the default confidence level for performing human target skeleton point detection on a single frame video image; the second confidence level is less than the first confidence level; Based on the second confidence level, skeletal point detection is performed on human targets in video images using a skeletal point detection algorithm.

3. The display device according to claim 1, characterized in that, The controller performs skeletal point detection on the human target in the video image using a skeletal point detection algorithm, and is specifically configured as follows: Acquire the real-time video stream detected by the detection device; Perform frame segmentation processing on the video stream to obtain multiple frames of the video image; Human target detection is performed on the video image using a human bounding box detection model to obtain a target image, wherein the target image is an image containing the human target. Feature extraction is performed on the target image using a skeleton point detection model to obtain skeleton point coordinates. The skeleton point detection model is a neural network model trained using sample skeleton point images, which include a sample image and the skeleton point labels of the sample image. Extract the coordinates of key bone points from the bone point coordinates; The human body coordinates are generated based on the coordinates of the key skeletal points.

4. The display device according to claim 3, characterized in that, The controller is configured to generate the human body coordinates based on the coordinates of the key skeletal points, specifically as follows: Traverse the coordinates of the key bone points; If the key skeletal point coordinates include a first feature coordinate and a second feature coordinate, then the first center point coordinates of the first feature coordinate and the second feature coordinate are calculated, and the human body coordinates are generated based on the first center point coordinates; the first feature coordinate and the second feature coordinate are respectively used to characterize two mutually symmetrical joints in the human body target; If the key skeletal point coordinates include one of the first feature coordinates and the second feature coordinates, then the second center point coordinates of the remaining skeletal point coordinates are calculated, and the human body coordinates are generated based on the second center point coordinates. The remaining skeletal point coordinates are the key skeletal point coordinates in the video image other than the first feature coordinates or the second feature coordinates.

5. The display device according to claim 3, characterized in that, The effective detection range includes a first boundary and a second boundary with the detection device as the origin; the human body coordinates include the horizontal and vertical coordinates of the human body; the controller generates a current state based on the human body coordinates and the effective detection range, specifically configured as follows: Extract the horizontal and vertical coordinates of the human body from the human body coordinates; The first horizontal coordinate of the first boundary and the second horizontal coordinate of the second boundary are obtained based on the vertical coordinate of the human body. If the horizontal coordinate of the human body is between the first horizontal coordinate and the second horizontal coordinate, then the current state is marked as the first state; If the horizontal coordinate of the human body is not between the first horizontal coordinate and the second horizontal coordinate, then the current state is marked as the second state.

6. The display device according to claim 5, characterized in that, The current state also includes a third state and a fourth state. The third state is used to characterize the state in which the human target is within the effective detection range and the distance between the human target and the first boundary or the second boundary gradually decreases. The fourth state is used to characterize the state in which the human target changes from the first state to the second state. After generating the current state based on the human body coordinates and the effective detection range, the controller is further configured to: When the current state is the first state, the target image is acquired periodically according to the second duration; Detect the human body coordinates in the target image, and extract the human body's horizontal coordinates from the human body coordinates; When the difference between the human body's horizontal coordinate and the first or second horizontal coordinate is less than a coordinate threshold, the current state is marked as the third state. Skeletal point feature extraction is performed on the target image to detect skeletal points in the target image; If no skeletal point features are extracted within the second time period, the current state is marked as the fourth state.

7. The display device according to claim 6, characterized in that, After generating the current state based on the human body coordinates and the effective detection range, the controller is further configured to: When the current state is the first state, skeletal point detection is performed on the human target in the video image; Obtain historical states, which are states generated within the previous detection period; If the skeletal point is not detected in the video image and the historical state is not the fourth state, then the current state is maintained as the first state.

8. The display device according to claim 1, characterized in that, After the controller executes the modification of the current state to the first state, it is also configured to: Obtain a second set of images from the video within a third duration, wherein the third duration is longer than the first duration; Perform human target detection on the second image in the second image set; If the number of frames in the second image containing the human target is less than a second preset frame threshold, then the current state is changed to the second state; Obtain a third image set of the video images within a first time period, wherein the first image set, the second image set, and the third image set are sets of video images obtained within different time periods; Perform human target detection on the third image in the third image set; If the number of image frames containing the human target in the third image is greater than or equal to the first preset frame number threshold, then the current state is modified to the first state.

9. The display device according to claim 8, characterized in that, After the controller executes the action of maintaining the current state as the second state, it is further configured to: Obtain a third set of images from the video images within the first time period; Perform human target detection on the third image in the third image set; If the number of image frames containing the human target is greater than or equal to a first preset frame count threshold, then the current state is modified to the first state; Obtain a second set of images from the video images within the third time period; Perform human target detection on the second image in the second image set; If the number of image frames containing the human target in the second image is less than a second preset frame count threshold, then the current state is changed to the second state.

10. A method for human body detection, characterized in that, Applied to the display device according to any one of claims 1-9, wherein the display device is connected to the detection device via a device interface, the human body detection method includes: In response to a control command for detecting a human target, the effective detection range of the detection device and the video images acquired by the detection device are obtained; The human target in the video image is detected by a skeletal point detection algorithm to generate human coordinates based on the skeletal points; A current state is generated based on the human body coordinates and the effective detection range; the current state includes a first state and a second state; the first state indicates that the human target is within the effective detection range; the second state indicates that the human target is not within the effective detection range. When the current state is the second state, a first image set of the video images within a first time period is obtained, and human target detection is performed on the first image in the first image set, and the current state is updated according to the number of image frames containing human targets in the first image set; wherein, if the number of image frames is greater than a first preset frame number threshold, the current state is modified to the first state; if the number of image frames is less than the first preset frame number threshold, the current state is maintained as the second state.

Images