A method for determining the positions of key points on a display device and the human body.

Abstract

This application provides a method for determining the positions of key points on a display device and the human body. This method eliminates traditional dedicated hardware such as sensors / radar during user posture data acquisition and processing, collecting data solely through a camera and directly acquiring the video stream using WebRTC technology natively supported by the browser, without requiring additional equipment. The human posture recognition model is integrated into the user device using TensorFlow.js, enabling on-device inference and avoiding data uploads to the cloud, thus eliminating transmission latency at its source. Model inference is completed on the browser side, accelerated by a web graphics library to improve processing efficiency and ensure real-time performance. The approximate positions of key points are located using the maximum probability of a heatmap, and then corrected using offsets to improve the positioning accuracy of human key points, further optimizing transmission efficiency and solving the problem of data transmission latency and its impact on transmission efficiency in current user posture data acquisition and processing.

Description

Technical Field

[0001] This application relates to the field of display device technology, and in particular to a display device and a method for determining the location of key points on the human body. Background Technology

[0002] To enhance user experience, smart TVs and other display devices are increasingly offering rich interactive functions. For example, they can use sensors or radar systems to capture user posture information and enable diverse interactive applications based on these posture detection results. These include convenient TV operation and control, immersive entertainment experiences, precise fitness and exercise analysis, and preliminary medical and health checks.

[0003] Currently, the technical approach to user posture detection mainly consists of two stages: data acquisition and data processing. In the data acquisition stage, existing technologies generally employ sensors or radar systems as the primary means of data acquisition. These devices can capture the user's motion state and convert it into electrical signals. Subsequently, this raw posture data is uploaded to a cloud server for processing. The cloud server, leveraging high-performance computing resources and pre-trained deep learning models, performs real-time analysis of the uploaded data and feeds the analysis results back to the terminal to guide corresponding interactive operations.

[0004] However, the aforementioned existing technical solutions have the following problems. On the one hand, reliance on cloud processing inevitably leads to latency during data transmission, which affects the real-time nature of interaction and user experience, especially in scenarios requiring rapid response, such as fitness guidance or game interaction. On the other hand, using sensors or radar for attitude data acquisition not only increases hardware costs but also faces bottlenecks in data transmission efficiency between the sensing device and the host, limiting the overall flexibility and scalability of the system. In summary, current user attitude data acquisition and processing suffers from data transmission latency, which also affects transmission efficiency. Summary of the Invention

[0005] Some embodiments of this application provide a method for determining the positions of key points on a display device and a human body, in order to solve the problem of data transmission delay and reduced transmission efficiency in the current process of user posture data acquisition and processing.

[0006] In a first aspect, some embodiments of this application provide a display device, including: The display is configured to show the user interface; The camera is configured to capture video stream data of the user's posture. The controller is configured as follows: In response to a command to detect the user's posture, the camera is invoked via the browser's real-time web communication function to acquire the video stream data; The current video frame in the video stream data is input into the human pose recognition model based on TensorFlow.js integrated in the browser; Based on TensorFlow.js, the human pose recognition model is run in the browser using a web graphics library to obtain a heatmap and offset containing multiple human key points; the human pose recognition model is used to detect and locate the human key points; the heatmap is used to characterize the probability distribution of the human key points; the offset is used to correct the position of the human key points in the heatmap. The first positions corresponding to the multiple key points of the human body are determined based on the heat map; The first position of each human body key point is adjusted according to the offset corresponding to each human body key point to obtain the second position of each human body key point.

[0007] The above technical solution has the following advantages or beneficial effects: In the process of user posture data acquisition and processing, the display device abandons traditional dedicated hardware such as sensors / radar, collecting data only through a camera and directly acquiring the video stream using WebRTC technology natively supported by the browser, without the need for additional devices. The human posture recognition model is integrated into the user device using TensorFlow.js, enabling on-device inference and avoiding data uploads to the cloud, thus eliminating transmission latency at the source. Model inference is completed on the browser side, accelerated by a web graphics library, improving processing efficiency, ensuring real-time performance, and solving the problem of data transmission latency and its impact on transmission efficiency in current user posture data acquisition and processing processes.

[0008] In some embodiments, the controller is specifically configured to, before inputting the current video frame from the video stream data into the human pose recognition model integrated in the browser using TensorFlow.js: Obtain the image data corresponding to the current video frame; Scale the image data to a preset size; Convert the scaled image data from RGBA four-channel format to RGB three-channel format; The pixel values ​​of the image data in the RGB channels are normalized from integer format to floating-point format to obtain a normalized floating-point array.

[0009] The above technical solution has the following advantages or beneficial effects: By scaling and converting channel formats, it ensures that the preprocessed image data meets the input requirements of the subsequent human pose recognition model, improving the model's recognition accuracy and stability. By removing the alpha channel and normalizing pixel values ​​to floating-point format, the amount of data can be reduced, lowering the computational complexity and memory consumption during model inference, and improving processing efficiency. Pixel value normalization makes the numerical range of the input data more concentrated and stable, helping the model maintain numerical stability during training and inference.

[0010] In some embodiments, the controller, based on TensorFlow.js, runs the human pose recognition model in the browser via a web graphics library to obtain a heatmap and offset containing multiple human key points, specifically configured as follows: The normalized floating-point array is reshaped into a three-dimensional tensor according to the three dimensions of height, width, and number of channels; The three-dimensional tensor is input into the human pose recognition model; Using the inference capabilities of TensorFlow.js, the human pose recognition model is run in the browser via a web graphics library to perform inference calculations on the three-dimensional tensor, resulting in a heatmap and offset containing multiple human key points.

[0011] The above technical solution has the following advantages or beneficial effects: By defining the height, width, and number of channels of the three-dimensional tensor, it ensures that the input data is fully compatible with the model input layer structure, avoiding model inference errors or performance degradation caused by dimensional mismatch, and improving system stability. Heatmaps provide candidate regions for key points, and offsets further correct the coordinates. Compared with traditional methods that rely solely on heatmap peak positioning, this reduces positioning errors and improves the accuracy of human body key point positioning. Simultaneously, calculations are performed locally on the TV using JavaScript, eliminating the need to upload data to the cloud, thereby reducing transmission latency and protecting user privacy.

[0012] In some embodiments, the controller, in response to a user pose detection command, invokes the camera via the browser's real-time web communication function to acquire the video stream data, specifically configured as follows: In response to the command to detect the user's posture, a request to acquire a video stream is initiated to the camera through the browser's real-time web communication function; Receive the video stream returned by the camera in accordance with the acquisition request.

[0013] The above technical solution has the following advantages or beneficial effects: By directly initiating a video stream acquisition request to the camera through the browser's built-in interface, data transmission delays in intermediate stages can be avoided, ensuring real-time acquisition of the video stream. Utilizing the browser's built-in interface for video stream acquisition eliminates the need for additional installation or configuration of complex software or hardware, reducing system resource consumption. This interface supports cross-platform use, improving the flexibility and efficiency of resource utilization. Since video stream data is acquired and processed directly on the browser side, the number and range of data transmissions over the network are reduced, thereby lowering the risk of data leakage and further protecting the security and privacy of user data.

[0014] In some embodiments, the controller is further configured to: input the current video frame from the video stream data into the TensorFlow.js-based human pose recognition model integrated in the browser before: The video stream data is bound to the video playback tag in the browser's built-in Hypertext Markup Language to play the video stream; Draw the current video frame corresponding to the playing video stream data onto the canvas; Extract the pixel data of the current video frame.

[0015] The above technical solution has the following advantages or beneficial effects: through the browser's built-in... <video>and <canvas>By combining elements with JavaScript programming, efficient video frame capture and rendering can be achieved without relying on additional video processing libraries or software, simplifying the technical implementation process and improving development efficiency. Since this method is entirely based on browser built-in elements and JavaScript, it has excellent cross-platform compatibility, running stably on different operating systems and devices without requiring additional adaptation for specific platforms, thus broadening application scenarios and usage scope. Combining real-time video stream playback and instant video frame capture ensures the real-time nature of the extraction process, meeting the needs of scenarios with high requirements for real-time pose detection.

[0016] In some embodiments, the controller determines the first positions corresponding to the plurality of key human body points based on the heat map, specifically configured as follows: Find the location of the maximum probability for each key point of the human body in the heat map; The position of the maximum probability is taken as the first position corresponding to each of the key points of the human body.

[0017] The above technical solution has the following advantages or beneficial effects: By directly searching for the maximum probability value in the heatmap, the candidate location of each key point can be quickly determined. The global search strategy ensures that the absolute peak location in the heatmap is found, avoiding positioning deviations caused by local extrema. The first location serves as a reference point for offset correction, providing a data foundation for the subsequent accurate positioning of human body key points.

[0018] In some embodiments, the controller adjusts the first position of each human keypoint according to the offset corresponding to each human keypoint to obtain the second position of each human keypoint, specifically configured as follows: Extract the first direction offset and the second direction offset corresponding to each of the first positions from the offset; Add the first coordinate of each first position to the first direction offset to obtain the first coordinate of each second position; Add the second coordinate of each first position to the second direction offset to obtain the second coordinate of each second position.

[0019] The above technical solution has the following advantages or beneficial effects: By correcting the coarse position of the heatmap using offsets, the positional error caused by model downsampling or feature map resolution limitations can be effectively reduced, improving the accuracy of key point coordinate localization. Simultaneously, the offsets are generated by the model through deconvolution operations in the decoder stage, fusing global semantic information from low-resolution feature maps with local detail information from high-resolution feature maps. By extracting and applying the offsets, the model's multi-scale modeling capabilities can be fully utilized, avoiding the accuracy decrease caused by feature loss when directly regressing coordinates.

[0020] In some embodiments, after the controller adjusts the first position of each human keypoint according to the offset corresponding to each human keypoint to obtain the second position of each human keypoint, it is further configured to: Obtain the probability value corresponding to each key point of the human body in the heat map; The probability value is compared with the probability threshold; Delete human key points whose probability value is lower than the probability threshold, and retain human key points whose probability value is higher than or equal to the probability threshold.

[0021] The above technical solution has the following advantages or beneficial effects: threshold filtering can eliminate low-confidence key points (such as erroneous points caused by occlusion, blurring, or false detection), ensuring that only high-confidence key points participate in subsequent interaction logic. This process distinguishes real key points from noise by quantifying confidence, avoiding misoperations (such as incorrect gesture recognition), thereby improving the accuracy of key point localization and user experience.

[0022] In some embodiments, after the controller runs the human pose recognition model in the browser using a web graphics library based on TensorFlow.js to obtain a heatmap and offsets containing multiple human key points, it is further configured to: Detect all key human body points included in the heat map; Clustering was performed on all the aforementioned key human body points to obtain multiple clusters; The number of clusters corresponding to the clusters after clustering is counted to obtain the number of users contained in the heatmap. When the number of users corresponds to one user, the first positions corresponding to multiple human body key points are determined according to the heat map; and the first positions of each human body key point are adjusted according to the offset corresponding to each human body key point to obtain the second position of each human body key point. When the number of users corresponds to multiple users, for each user's corresponding human body key point, the steps of determining the first position corresponding to the multiple human body key points according to the heat map, and adjusting the first position of each human body key point according to the offset corresponding to each human body key point, are performed to obtain the second position of each human body key point.

[0023] The above technical solution has the following advantages or beneficial effects: By distinguishing the key points of different users through clustering algorithms, the problem of key point confusion when multiple users overlap can be avoided, and the clustering process does not require a pre-set number of users, and can dynamically adapt to different user scenarios. At the same time, after different users are distinguished, the key point localization task can be processed separately for each user, and the coordinates of key points in different clusters can be calculated in parallel, reducing the processing pressure of a single thread.

[0024] Secondly, some embodiments of this application provide a method for determining the location of key points on the human body, which can be applied to the display device of the first aspect. The display device includes a display, a camera, and a controller. The method includes: In response to a command to detect the user's posture, the camera is invoked via the browser's real-time web communication function to acquire the video stream data; The current video frame in the video stream data is input into the human pose recognition model based on TensorFlow.js integrated in the browser; Based on TensorFlow.js, the human pose recognition model is run in the browser using a web graphics library to obtain a heatmap and offset containing multiple human key points; the human pose recognition model is used to detect and locate the human key points; the heatmap is used to characterize the probability distribution of the human key points; the offset is used to correct the position of the human key points in the heatmap. The first positions corresponding to the multiple key points of the human body are determined based on the heat map; The first position of each human body key point is adjusted according to the offset corresponding to each human body key point to obtain the second position of each human body key point.

[0025] The above technical solution has the following advantages or beneficial effects: In the process of user posture data acquisition and processing, the method abandons traditional dedicated hardware such as sensors / radar, collecting data only through a camera and directly acquiring the video stream using WebRTC technology natively supported by the browser, without the need for additional equipment. It integrates the human posture recognition model into the user device using TensorFlow.js, enabling on-device inference and avoiding data uploads to the cloud, thus eliminating transmission latency at the source. Model inference is completed on the browser side, accelerated by a web graphics library, improving processing efficiency, ensuring real-time performance, and solving the problem of data transmission latency and its impact on transmission efficiency in current user posture data acquisition and processing processes. Attached Figure Description

[0026] To more clearly illustrate the technical solutions in some embodiments of this application or in the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0027] Figure 1 This is a schematic diagram illustrating an operational scenario between a display device and a control device provided in some embodiments of this application; Figure 2 This is a schematic diagram of the hardware configuration of a display device provided in some embodiments of this application; Figure 3 This is a schematic diagram of the software configuration of a display device provided in some embodiments of this application; Figure 4 A schematic flowchart illustrating a method for determining the location of key points on the human body using a display device, provided in some embodiments of this application. Figure 5 This application provides a flowchart illustrating the process of outputting heatmaps and offsets using a human pose recognition model, as shown in some embodiments. Figure 6 A schematic diagram illustrating the process of a display device determining a second position based on a first position according to some embodiments of this application; Figure 7 This application provides schematic diagrams illustrating the effect of a display device outputting the second position of key human body points in some embodiments. Figure 8 A schematic diagram illustrating the process of screening key points of the human body using a display device provided in some embodiments of this application; Figure 9 A schematic diagram illustrating the process of performing multi-person posture detection using a display device provided in some embodiments of this application; Figure 10 This is a schematic diagram illustrating the posture recognition process of a display device for single-person and multi-person scenarios, provided in some embodiments of this application. Detailed Implementation

[0028] The embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described below do not represent all embodiments consistent with this application. They are merely examples of systems and methods consistent with some aspects of this application as detailed in the claims.

[0029] It should be noted that the brief descriptions of terms in this application are only for the convenience of understanding the embodiments described below, and are not intended to limit the embodiments of this application. Unless otherwise stated, these terms should be understood in their ordinary and common meaning.

[0030] In this application, the terms "first," "second," "third," etc., used in the specification, claims, and accompanying drawings are used to distinguish similar or related objects or entities, and do not necessarily imply a specific order or sequence, unless otherwise specified. It should be understood that such terms are interchangeable where appropriate.

[0031] The terms "comprising" and "having," and any variations thereof, are intended to cover but not exclude inclusion, for example, a product or device that includes a range of components is not necessarily limited to all of the components that are clearly listed, but may include other components that are not clearly listed or that are inherent to such product or device.

[0032] The term "module" refers to any known or subsequently developed hardware, software, firmware, artificial intelligence, fuzzy logic, or combination of hardware and / or software code that is capable of performing the functions associated with that element.

[0033] In this embodiment, the display device 200 generally refers to a device with screen display and data processing capabilities. For example, the display device 200 includes, but is not limited to, smart TVs, mobile terminals, computers, monitors, advertising screens, wearable devices, virtual reality devices, augmented reality devices, etc.

[0034] Figure 1 This is a schematic diagram illustrating an operational scenario between a display device and a control device provided in some embodiments of this application. For example... Figure 1 As shown, a user can operate the display device 200 via touch operation, a mobile terminal 300, and a control device 100. The control device 100 receives user input commands and converts them into control commands that the display device 200 can recognize and respond to. For example, the control device 100 can be a remote control, a stylus, a gamepad, etc.

[0035] The mobile terminal 300 can function as a control device for human-computer interaction between the user and the display device 200. It can also function as a communication device for establishing a communication connection with the display device 200 and exchanging data. In some embodiments, the mobile terminal 300 can have software applications installed on it and communicate with the display device 200 via network communication protocols to achieve one-to-one control and data communication. Furthermore, it can transmit audio and video content displayed on the mobile terminal 300 to the display device 200 for synchronized display.

[0036] In some embodiments, the mobile terminal 300 or other electronic devices may also simulate the functions of the control device 100 by running an application that controls the display device 200.

[0037] like Figure 1 The diagram also shows that the display device 200 communicates with the server 400 via various communication methods. This allows the display device 200 to communicate via a local area network (LAN), a wireless local area network (WLAN), and other networks.

[0038] Display device 200 can provide broadcast television reception function, and can also be equipped with intelligent network television function that provides computer support function, including but not limited to network television, smart television, Internet Protocol television (IPTV), etc.

[0039] Figure 2 Provided for some embodiments of this application Figure 1 Hardware configuration block diagram of display device 200.

[0040] In some embodiments, the display device 200 may include at least one of a tuner 210, a communication device 220, a detector 230, a device interface 240, a controller 250, a display 260, an audio output device 270, a memory, a power supply, and a user input interface 280.

[0041] In some embodiments, detector 230 is used to acquire signals from the external environment or to interact with the outside world. For example, detector 230 includes a light receiver, a sensor for acquiring ambient light intensity; or, detector 230 includes an image acquisition device, such as a camera, which can be used to acquire external environmental scenes, user attributes, or user interaction gestures; or, detector 230 includes a sound acquisition device, such as a microphone, for receiving external sounds.

[0042] In some embodiments, the display 260 includes display function components for presenting images and driving components for driving image display. The display 260 is used to receive and display image signals output from the controller 250. For example, the display 260 can be used to display video content, image content, menu control interface components, and user control UI interfaces, etc.

[0043] In some embodiments, the communication device 220 is a component used to communicate with external devices or the server 400 according to various communication protocol types. The display device 200 may have multiple communication devices 220 depending on the supported communication methods. For example, when the display device 200 supports wireless network communication, it may have a communication device 220 with WiFi functionality. When the display device 200 supports Bluetooth connectivity, it needs to have a communication device 220 with Bluetooth functionality.

[0044] The communication device 220 enables the display device 200 to communicate with external devices or the server 400 via wireless or wired connections. Wired connections utilize data cables, interfaces, or other components to connect the display device 200 to external devices. Wireless connections utilize wireless signals or wireless networks. The display device 200 can directly establish a connection with external devices or indirectly through gateways, routers, or other connection devices.

[0045] In some embodiments, the controller 250 may include at least one of a central processing unit, a video processor, an audio processor, a graphics processor, and a power processor, and a first to an nth interface for input / output. The controller 250 controls the operation of the display device and responds to user operations through various software control programs stored in memory. The controller 250 controls the overall operation of the display device 200.

[0046] In some embodiments, the controller 250 and the tuner 210 may be located in different separate devices, that is, the tuner 210 may also be located in an external device of the main device where the controller 250 is located, such as an external set-top box.

[0047] In some embodiments, a user can input user commands through a graphical user interface (GUI) displayed on a display 260, and the user input interface 280 receives the user input commands through the graphical user interface (GUI).

[0048] In some embodiments, the audio output device 270 can be a built-in speaker of the display device 200 or an external audio output device connected to the display device 200. For the external audio output device connected to the display device 200, the display device 200 may also be provided with an external audio output terminal, through which the audio output device can be connected to the display device 200 to output sound from the display device 200.

[0049] In some embodiments, the user input interface 280 can be used to receive instructions from user input.

[0050] To enable user interaction, in some embodiments, the display device 200 may run an operating system. The operating system is a computer program used to manage and control the hardware and software resources of the display device 200. The operating system can control the display device to provide a user interface; for example, the operating system can directly control the display device to provide a user interface, or it can provide a user interface by running an application. The operating system also allows users to interact with the display device 200.

[0051] It should be noted that the operating system can be a native operating system based on a specific operating platform, a third-party operating system that is deeply customized based on a specific operating platform, or an independent operating system specifically developed for display devices.

[0052] An operating system can be divided into different modules or levels based on the functions it implements, for example... Figure 3 As shown, in some embodiments, the system is divided into four layers, from top to bottom: the Applications layer (referred to as the "Application Layer"), the Application Framework layer (referred to as the "Framework Layer"), the System Library layer, and the Kernel layer.

[0053] In some embodiments, the application layer provides services and interfaces for applications, enabling the display device 200 to run applications and interact with the user based on the applications. The application layer may contain at least one application, which may be a built-in Windows program, system settings program, or clock program of the operating system; or it may be an application developed by a third-party developer. In specific implementations, the application packages in the application layer are not limited to the examples above.

[0054] The framework layer provides application programming interfaces (APIs) and a programming framework for applications. The application framework layer includes predefined functions. It acts as a central processing unit, determining the actions taken by applications within the application layer. Through the API, applications can access system resources and obtain system services during execution.

[0055] like Figure 3 As shown, the application framework layer in this embodiment includes a view system, managers, and content providers. The view system designs and implements the application's interface and interactions, and includes lists, grids, text boxes, and buttons. The managers include at least one of the following modules: an activity manager for interacting with all running activities in the system; a location manager for providing system services or applications with access to system location services; a package manager for retrieving various information related to application packages currently installed on the device; a notification manager for controlling the display and clearing of notification messages; and a window manager for managing icons, windows, toolbars, wallpapers, and desktop widgets on the user interface.

[0056] In some embodiments, the Activity Manager manages the lifecycle of individual applications and common navigation and back functions, such as controlling application exit, opening, and back actions. The Window Manager manages all window programs, such as obtaining the screen size, determining if a status bar is present, locking the screen, capturing the screen, and controlling changes to the display window, such as shrinking the display window, shaking the display, or distorting the display.

[0057] In some embodiments, the system runtime library layer can provide support for the framework layer. When the framework layer is used, the operating system runs the instruction library contained in the system runtime library layer, such as the C / C++ instruction library, to implement the functions to be performed by the framework layer.

[0058] In some embodiments, the kernel layer is a functional layer situated between the hardware and software of the display device 200. The kernel layer can implement functions such as hardware abstraction, multitasking, and memory management. For example, ... Figure 3 As shown, hardware drivers can be configured in the kernel layer. The kernel layer can contain at least one of the following drivers: audio driver, display driver, Bluetooth driver, camera driver, WIFI driver, USB driver, HDMI driver, sensor driver (such as fingerprint sensor, temperature sensor, pressure sensor, etc.), and power driver, etc.

[0059] It should be noted that the above examples are merely a simple division of operating system functions and do not limit the specific form of the operating system of the display device 200 in this application embodiment. Depending on the function of the display device, the type of operating system, and other factors, the number of levels and the specific level type of the operating system may be expressed in other forms.

[0060] To enhance user experience, smart TVs and other display devices are increasingly offering rich interactive functions. For example, they can use sensors or radar systems to capture user posture information and enable diverse interactive applications based on these posture detection results. These include convenient TV operation and control, immersive entertainment experiences, precise fitness and exercise analysis, and preliminary medical and health checks.

[0061] Currently, the technical approach to user posture detection mainly consists of two stages: data acquisition and data processing. In the data acquisition stage, existing technologies generally employ sensors or radar systems as the primary means of data acquisition. These devices can capture the user's motion state and convert it into electrical signals. Subsequently, this raw posture data is uploaded to a cloud server for processing. The cloud server, leveraging high-performance computing resources and pre-trained deep learning models, performs real-time analysis of the uploaded data and feeds the analysis results back to the terminal to guide corresponding interactive operations.

[0062] However, the aforementioned existing technical solutions have the following problems. On the one hand, reliance on cloud processing inevitably leads to latency during data transmission. This latency affects the real-time nature of interaction and user experience, especially in scenarios requiring rapid response, such as fitness guidance or game interaction. On the other hand, uploading data to the cloud for processing also raises concerns about user data security, with personal privacy information facing the risk of leakage during transmission and storage. Furthermore, using sensors or radar for attitude data acquisition not only increases hardware costs but also faces bottlenecks in data transmission efficiency between sensing devices and the host, limiting the overall flexibility and scalability of the system. In summary, current user attitude data acquisition and processing suffers from data transmission latency and affects transmission efficiency.

[0063] To address the data transmission latency and efficiency issues inherent in current user posture data acquisition and processing, this application provides a display device 200 in certain embodiments. The display device 200 includes a monitor 260, a camera, and a controller 250. The monitor 260 is configured to display a user interface, the camera is configured to acquire video stream data of the user's posture, and the controller 250, by running an application, enables the display device 200 to execute a method for determining the location of key human body points. These key human body points refer to critical parts of the user's body, such as the tip of the nose, shoulders, and elbows. In the process of acquiring and processing user posture data, the display device 200 eliminates traditional dedicated hardware such as sensors / radar, acquiring data solely through the camera and directly obtaining the video stream using WebRTC technology natively supported by the browser, without requiring additional equipment. The human posture recognition model is integrated into the user device using TensorFlow.js, enabling edge-side inference and avoiding data uploads to the cloud, thus eliminating transmission latency at its source. Model inference is performed on the browser side, using Web Graphics Library (WebGL) for acceleration, improving processing efficiency, ensuring real-time performance, and solving the problem of data transmission delay and impact on transmission efficiency in the current user pose data acquisition and processing process.

[0064] TensorFlow.js is a JavaScript-based machine learning framework that allows machine learning models to run directly in a browser or Node.js environment without relying on server-side processing. Its core functionalities include model loading, edge inference, training capabilities, and hardware acceleration. Specifically, model loading refers to supporting the loading of pre-trained models (such as human pose recognition models) from local files, remote URLs, or memory; edge inference refers to completing model inference computation locally on the device (such as in a browser), avoiding data uploads to the cloud and improving privacy and real-time performance; training capabilities support model fine-tuning or incremental training on the browser side; and hardware acceleration leverages the parallel computing capabilities of GPUs through WebGL to significantly improve inference speed, especially suitable for image processing tasks (such as pose estimation).

[0065] Compared to existing technologies, this application directly integrates the human pose recognition model implemented in TensorFlow.js into the browser environment, leveraging TensorFlow.js's edge inference capabilities to complete model inference calculations locally on the device, thus achieving local inference. For example, the human pose recognition model implemented in TensorFlow.js is directly integrated into the browser environment, utilizing its edge inference capabilities to complete the following process: During the deployment of TensorFlow.js, the browser's WebRTC interface can be used to obtain video stream data from the camera, and the JavaScript API provided by TensorFlow.js can be used to load and run the human pose recognition model. For example, the video stream can be obtained through the `navigator.mediaDevices.getUserMedia()` method, and then HTML5 can be used... <video>Tags and <canvas>The element combines JavaScript to capture and process video frames, and finally inputs these processed video frames into the TensorFlow.js model for inference.

[0066] During the human pose recognition model loading stage, pre-trained human pose recognition models (such as PoseNet and MoveNet) can be loaded from local or remote locations using TensorFlow.js APIs (such as tf.loadLayersModel()). Supported model formats may include TensorFlow SavedModel, Keras HDF5, or TensorFlowLite. During the video stream data acquisition phase, the browser's WebRTC interface (such as navigator.mediaDevices.getUserMedia()) can be used to access the camera and obtain real-time video stream data. Alternatively, HTML5 can be used... <video>Tag playback video stream, and combined with <canvas>Elements capture video frames (e.g., 15-30 frames per second); In the video frame preprocessing and model inference stages, preprocessing involves scaling, normalizing, and other operations on the captured video frames to convert them into the model input format (such as a 224×224 RGB image). During the inference computation phase, the preprocessed image is input into the TensorFlow.js model, and the GPU inference accelerated by WebGL outputs heatmaps and offsets of human key points. Then, the heatmaps and offsets are parsed to determine the precise coordinates of the human key points (such as the tip of the nose, joints, etc.).

[0067] In some embodiments, TensorFlow.js supports model inference using CPU or GPU in the browser. For most modern browsers, WebGL is key to GPU acceleration. TensorFlow.js maps model computation tasks to the GPU via WebGL, leveraging its parallel processing capabilities to accelerate matrix operations (such as convolution operations), thus speeding up inference. This on-device processing avoids uploading video stream data to the cloud for processing, reducing data transmission latency, improving processing efficiency, and enhancing user data security. Simultaneously, browser-side WebGL acceleration improves processing efficiency and ensures real-time performance. By deploying TensorFlow.js locally in the browser and using WebGL for GPU-accelerated computation, human pose recognition is transformed from the cloud to the local device, thereby solving the problem of data transmission latency and its impact on efficiency in current user pose data acquisition and processing.

[0068] To facilitate understanding of the technical solutions in some embodiments of this application, the steps are described in detail below with reference to some specific embodiments and accompanying drawings. Figure 4 This application provides a schematic flowchart of a method for determining the location of key human body points using a display device, as shown in some embodiments. Figure 4 As shown, it may include the following steps: Step S1: In response to the user's posture detection command, the camera is invoked through the browser's real-time web communication function to obtain video stream data; In some embodiments, when the system receives a user's pose detection command, it can use the browser's built-in WebRTC (Web Real-Time Communication) technology to access the user's device's camera to capture real-time video stream data. Utilizing WebRTC technology enables low-latency video stream data capture, ensuring real-time pose detection. Because WebRTC is a technology natively supported by browsers, it has excellent cross-platform compatibility. Users do not need to install additional software or plugins; they can perform pose detection directly through their browser.

[0069] Upon receiving a user posture detection command, the TV can immediately trigger the camera to enter working mode and send a video stream acquisition request. Specifically, in response to the user posture detection command, the TV can initiate a video stream acquisition request to the camera via the browser's real-time web communication function, and then receive the video stream data returned by the camera based on the acquisition request.

[0070] For example, the system can access the user device's camera module via WebRTC technology in a browser (such as HTML5). WebRTC technology can be applied to real-time communication scenarios such as live streaming, and it offers low data transmission latency. Specifically, when the system receives a command to detect the user's posture, it can initiate a video stream acquisition request to the connected camera through a browser's built-in preset interface (such as the `navigator.mediaDevices.getUserMedia({ video: true})` method). This request aims to acquire real-time video data from the camera for subsequent human posture detection and analysis. After receiving the acquisition request, the camera will return the corresponding video stream data based on its current working status and permission settings. The system then receives and processes this video stream data to achieve real-time monitoring and recognition of the user's posture. In this way, directly initiating a video stream acquisition request to the camera through the browser's preset interface avoids data transmission delays in intermediate stages, ensuring real-time acquisition of the video stream. Utilizing the browser's built-in interface for video stream acquisition eliminates the need for additional installation or configuration of complex software or hardware devices, reducing system resource consumption. This interface supports cross-platform use, improving the flexibility and efficiency of resource utilization. Since video stream data is acquired and processed directly on the browser, the number of times and the scope of data transmission over the network can be reduced, thereby reducing the risk of data leakage and further protecting the security and privacy of user data.

[0071] In some embodiments, the television can extract the current video frame from the video stream and then perform image preprocessing operations on that frame sequentially. These operations may include, but are not limited to, scaling, color space conversion (such as converting to grayscale or normalized RGB), pixel value standardization, and data type conversion. By performing normalized preprocessing on the video frame, interference caused by differences in the acquisition environment and inconsistent device parameters can be reduced, ensuring that the input data maintains consistency in numerical distribution and spatial scale. This improves the robustness and generalization ability of the subsequent human pose recognition model and ensures the stability and accuracy of the model's inference process.

[0072] In some embodiments, the current video frame in a video stream can be extracted as follows: First, the video stream data is bound to a video playback tag in the browser's built-in Hypertext Markup Language to play the video stream. Then, the current video frame corresponding to the played video stream data is drawn onto the canvas, and finally, the pixel data of the current video frame is extracted.

[0073] For example, when extracting the current video frame from a video stream, one can first utilize the browser's built-in Hypertext Markup Language (HTML)... <video>The video playback tag binds the acquired video stream data to this tag. This operation enables the video stream to be played in real time on the browser page. Subsequently, JavaScript programming captures the current video frame from the playing video stream and draws it onto a predefined... <canvas>On the canvas element. Finally, the `getImageData()` method of the canvas is used to extract the pixel data of the current video frame. This data contains detailed information such as the color and brightness of each pixel in the video frame, providing a foundation for subsequent image preprocessing and pose recognition. Thus, through the browser's built-in... <video>and <canvas>By combining elements with JavaScript programming, efficient video frame capture and rendering can be achieved without relying on additional video processing libraries or software, simplifying the technical implementation process and improving development efficiency. Since this method is entirely based on browser built-in elements and JavaScript, it has excellent cross-platform compatibility, running stably on different operating systems and devices without requiring additional adaptation for specific platforms, thus broadening application scenarios and usage scope. Combining real-time video stream playback and instant video frame capture ensures the real-time nature of the extraction process, meeting the needs of scenarios with high requirements for real-time pose detection.

[0074] After the current video frame is extracted, image preprocessing can be performed on the current video frame to obtain a normalized floating-point array corresponding to the current video frame. Specifically, the image data corresponding to the current video frame is obtained, then the image data is scaled to a preset size, then the scaled image data is converted from the four-channel RGBA format to the three-channel RGB format, and finally the pixel values ​​of the RGB channel image data are normalized from integer format to floating-point format to obtain a normalized floating-point array.

[0075] For example, for the current video frame extracted from the video stream, a series of image preprocessing operations need to be performed to obtain a normalized floating-point array adapted to the input of the subsequent human pose recognition model. Firstly, this can be achieved through the browser's built-in... <canvas>The `getImageData()` method is used to retrieve the raw image data corresponding to the current video frame. This data typically contains four RGBA channels (red, green, blue, and alpha), with each channel's pixel value represented as an integer from 0 to 255. The retrieved raw image data is then scaled to a preset size (e.g., 257x257 pixels) to fit the input requirements of the subsequent human pose recognition model. Scaling can be achieved using algorithms such as bilinear interpolation or nearest-neighbor interpolation to ensure the image retains relatively clear features during scaling. Next, the scaled image data is converted from the RGBA four-channel format to the RGB three-channel format. This step is achieved by removing the alpha channel, retaining only the red, green, and blue color channels to reduce data volume and simplify subsequent processing. The pixel values ​​of the RGB channel image data are then normalized from the integer format of 0-255 to the floating-point format of 0-1. Normalization is achieved by dividing each pixel value by 255, ensuring the pixel values ​​are distributed within the [0,1] range, improving the model's numerical stability and convergence speed. In this way, by scaling and converting the channel format, the preprocessed image data is ensured to meet the input requirements of the subsequent human pose recognition model, improving the model's recognition accuracy and stability. By removing the alpha channel and normalizing the pixel values ​​to floating-point format, the amount of data can be reduced, lowering the computational complexity and memory consumption during model inference and improving processing efficiency. Pixel value normalization makes the numerical range of the input data more concentrated and stable, helping the model maintain numerical stability during training and inference. After step S1 is completed, step S2 can be executed.

[0076] Step S2: Input the current video frame from the video stream data into the human pose recognition model based on TensorFlow.js integrated in the browser; In some embodiments, the current video frame can be extracted from continuous video stream data through a cyclic or timed sampling mechanism. Subsequently, the video frame can be preprocessed using the HTML5 Canvas element, including scaling, normalization, and channel adjustment, to adapt to the input requirements of the TensorFlow.js-based human pose recognition model. The preprocessed image data is converted to floating-point format and passed as an input tensor to the TensorFlow.js model integrated in the browser. This preprocessing step ensures the video frame data matches the model's input requirements, improving the accuracy of model inference. Simultaneously, integrating the model directly into the browser avoids the latency and security risks associated with uploading data to the cloud. After step S2 is completed, step S3 can be executed.

[0077] Step S3: Based on TensorFlow.js, run the human pose recognition model in the browser using a web graphics library to obtain a heatmap and offset containing multiple human key points; the human pose recognition model is used to detect and locate human key points; the heatmap is used to represent the probability distribution of human key points; the offset is used to correct the position of human key points in the heatmap. In some embodiments, a pre-trained human pose recognition model (such as PoseNet) can be loaded and run in a browser using the TensorFlow.js library. This model extracts features and locates keypoints in the input image using a convolutional neural network (CNN), outputting a heatmap and offsets containing multiple human keypoints. The heatmap represents the probability distribution of each keypoint in the image, while the offsets are used to correct the keypoint positions in the heatmap to improve localization accuracy. Specifically, the model output is parsed using the TensorFlow.js tensor operation interface to obtain the heatmap and offset data for the keypoints. The combination of the heatmap and offsets enables precise localization of human keypoints.

[0078] In some embodiments, the human pose recognition model can employ a pose estimation network model based on a Convolutional Neural Network (CNN) (Pose Network model, hereinafter referred to as the PoseNet model). The core idea of ​​the PoseNet model is to extract image features using a deep convolutional neural network and generate a heatmap of human key points through multi-layer convolution and deconvolution operations. Its architecture combines the feature extraction capabilities of traditional CNNs with a lightweight network structure, enabling real-time inference on the browser side.

[0079] In some embodiments, the TV terminal reshapes a normalized floating-point array into a three-dimensional tensor structure according to the input requirements of the human pose recognition model. The human pose recognition model is used to detect and locate human key points. Subsequently, this three-dimensional tensor is input as input data into the pre-trained human pose recognition model. This model is based on a convolutional neural network architecture and can predict the spatial distribution information of multiple human key points in parallel during a single forward inference process, presenting it in two sets of outputs. The first is a heatmap tensor, where each channel corresponds to a probability distribution map of a preset human key point, and the value reflects the confidence that a corresponding key point exists at that location. The second is an offset tensor, used to correct the position of human key points in the heatmap, thereby improving the precision of key point localization. In this way, the prediction mechanism combining heatmap and offset can significantly improve the accuracy of key point localization while maintaining high detection efficiency. The construction of the three-dimensional tensor ensures that the model input and network structure are strictly matched, avoiding inference errors caused by dimensional mismatch, thus enabling the pose detection results to have both global perception and detail resolution capabilities, meeting the needs of TV interactive applications for accurate human pose analysis.

[0080] In some embodiments, the input to the PoseNet model is an RGB image (e.g., 224x224x3), and the output is a heatmap, where each keypoint corresponds to a channel representing the probability distribution of that point's location in the image. In PoseNet, the role of CNNs is mainly reflected in the following aspects: First, feature extraction, such as extracting high-level semantic features of the image through multiple convolutional layers; second, scale invariance processing, such as using multi-scale feature fusion mechanisms to enhance the model's adaptability to different body sizes and distances; third, keypoint localization, such as restoring spatial resolution by upsampling feature maps through deconvolutional layers, ultimately generating the heatmap.

[0081] Table 1 shows a list of key points on the human body. There are 17 key points, including the tip of the nose, left eye, right eye, left ear, and right ear. These can also be set according to actual needs. This application does not impose any specific limitations on them.

[0082]

[0083] Figure 5 The following is a schematic diagram of the process of outputting heatmaps and offsets through a human pose recognition model according to some embodiments of this application. The display device 200 first reshapes the normalized floating-point array into a three-dimensional tensor according to the three dimensions of height, width and number of channels. Then, the three-dimensional tensor is input into the human pose recognition model. Then, using the inference function of TensorFlow.js, the human pose recognition model is run in the browser through the Web graphics library to perform inference calculation on the three-dimensional tensor to obtain a heatmap and offset containing multiple human key points.

[0084] For example, when reshaping a normalized floating-point array into a 3D tensor according to the dimensions of height (H) × width (W) × number of channels (C), if the preprocessed image size is 257 × 257 pixels and uses an RGB three-channel format, the reshaped tensor will be 257 × 257 × 3, ensuring that the data dimensions perfectly match the model's input layer. Then, the reshaped 3D tensor is input into a TensorFlow.js-based human pose recognition model, such as the PoseNet model (the following explanation uses the PoseNet model, but this is not a limitation). The PoseNet model can perform inference computations on the input 3D tensor, and the inference mechanism can be implemented based on the TensorFlow.js tensor manipulation interface.

[0085] Display device 200 can use TensorFlow.js to load pre-trained human pose recognition models (such as the PoseNet model) and complete tensor construction and forward inference within this framework to obtain heatmaps and offset outputs. TensorFlow.js, as a JavaScript library, allows deep learning models to be run directly in a browser. Utilizing its inference capabilities, combined with the GPU acceleration of Web graphics libraries (such as WebGL), it enables efficient inference computation on the input 3D tensors. For example, the PoseNet model extracts features and performs spatial transformations on the input tensors through multiple layers of convolution, pooling, and upsampling operations. Specifically, in the feature encoding stage, spatial resolution can be gradually reduced and the number of channels increased through convolutional layers to extract multi-level features from low-level edges to high-level semantics; in the feature decoding stage, spatial resolution can be restored through deconvolution or transposed convolution, and multi-scale features can be fused using skip connections to ultimately generate a feature map with the same size as the input image. Ultimately, the model output contains two parts of structured data. One part is a heatmap, where 17 human keypoints correspond to 17 channels. Each channel's heatmap is generated using a Gaussian kernel, with pixel values ​​representing the probability that the location belongs to the corresponding keypoint, and peak positions corresponding to the coarse coordinates of the keypoint. The other part is an offset, where each keypoint type corresponds to two channels, representing the offset of the keypoint in the first direction (horizontal, x) and the second direction (vertical, y). The offset is used to correct the peak positions of the heatmap, improving the accuracy of keypoint localization. By defining the height, width, and number of channels of the three-dimensional tensor, the input data is ensured to be fully compatible with the model's input layer structure, avoiding model inference errors or performance degradation due to dimensional mismatch, and improving system stability. The heatmap provides candidate regions for keypoints, and the offset further corrects the coordinates. Compared to traditional methods that rely solely on heatmap peaks for localization, this reduces localization errors and improves the accuracy of human keypoint localization. Furthermore, the calculation is performed locally on the TV using JavaScript, eliminating the need to upload data to the cloud, thus reducing transmission latency and protecting user privacy. After step S3 is completed, step S4 can be executed.

[0086] Step S4: Determine the first position corresponding to each of the multiple key points on the human body based on the heat map; In some embodiments, the first position corresponding to multiple human body key points can be determined as follows: First, find the position with the maximum probability corresponding to each human body key point in the heat map, and then use the position with the maximum probability as the first position corresponding to each human body key point.

[0087] For example, the heatmap output by the model is a three-dimensional tensor, including height, width, and the number of keypoint types, such as 17 channels corresponding to 17 human body keypoints. The heatmap of each channel is generated using a Gaussian distribution, and the pixel value represents the probability density of the location belonging to the corresponding keypoint. For example, in the heatmap of the i-th channel, the higher the pixel value, the greater the probability that the location belongs to the i-th type of keypoint (such as the left shoulder). For each keypoint type (i.e., each heatmap channel), the display device 200 can locate the pixel position with the highest probability value in that channel through a global search (such as traversing all pixels) or an optimization algorithm (such as peak detection), and take the position of the highest probability value corresponding to each keypoint type as the first position of that keypoint (i.e., the coarse positioning result). In this way, by directly searching for the highest probability value in the heatmap, the candidate position of each keypoint can be quickly determined. The global search strategy ensures that the absolute peak position in the heatmap is found, avoiding positioning deviations caused by local extrema. The first position serves as the reference point for offset correction, providing a data foundation for the subsequent accurate positioning of human body keypoints. After step S4 is completed, step S5 can be executed.

[0088] Step S5: Adjust the first position of each human body key point according to the offset corresponding to each human body key point to obtain the second position of each human body key point.

[0089] Figure 6 The flowchart illustrating the process of a display device determining a second position based on a first position according to a first position in some embodiments of this application is shown below. After determining the first position corresponding to a key point on the human body, the second position can be obtained in the following manner: First, extract the first direction offset and the second direction offset corresponding to each first position from the offset. Then, add the first coordinate of each first position to the first direction offset to obtain the first coordinate of each second position. Finally, add the second coordinate of each first position to the second direction offset to obtain the second coordinate of each second position.

[0090] For example, the approximate location of each key point (i.e., the coordinates corresponding to the maximum probability in the heatmap channel) is obtained from the heatmap, designated as the first location. From the offset tensor, a first-direction offset (e.g., horizontal offset) and a second-direction offset (e.g., vertical offset) corresponding to the first location are extracted. Then, the horizontal coordinate of the first location is added to the horizontal offset to obtain the precise horizontal coordinate of the second location (the first coordinate of the second location). Similarly, the vertical coordinate of the first location is added to the vertical offset to obtain the precise vertical coordinate of the second location (the second coordinate of the second location). This process, by combining the probability distribution of the heatmap with the fine-tuning function of the offset, achieves precise positioning of the key point coordinates, ultimately outputting as shown below. Figure 7 The diagram illustrates the accurate locations of key human body points (0 represents the nose, 1 the left eye, 2 the right eye, 3 the left ear, 4 the right ear, 5 the left shoulder, 6 the right shoulder, 7 the left elbow, 8 the right elbow, 9 the left wrist, 10 the right wrist, 11 the left hip, 12 the right hip, 13 the left knee, 14 the right knee, 15 the left ankle, and 16 the right ankle; these are illustrative examples and do not constitute a limitation on key human body points). By correcting the coarse positions of the heatmap using offsets, positional errors caused by model downsampling or feature map resolution limitations can be effectively reduced, improving the accuracy of key point coordinate localization. Simultaneously, the offsets are generated by the model through deconvolution operations in the decoder stage, fusing global semantic information from low-resolution feature maps with local detail information from high-resolution feature maps. By extracting and applying the offsets, the model's multi-scale modeling capabilities can be fully utilized, avoiding the accuracy loss caused by feature loss when directly regressing coordinates.

[0091] It should be noted that this application abandons traditional dedicated hardware such as sensors / radar in the process of user posture data acquisition and processing. Data is collected solely through a camera, and the video stream is directly obtained using the browser's native WebRTC technology, without the need for additional equipment. The human posture recognition model is integrated into the user device using TensorFlow.js, enabling on-device inference and avoiding data uploads to the cloud, thus eliminating transmission latency at the source. This is achieved through the browser's built-in HTML5... <video>Tags and <canvas>The combination of elements enables efficient capture and rendering of video frames without the need for additional video processing libraries. Model inference is performed on the browser side, utilizing WebGL acceleration to improve processing efficiency and ensure real-time performance. Simultaneously, the PoseNet model introduces a multi-scale feature extraction mechanism, fusing low-level details and high-level semantic information through skip connections, enhancing its adaptability to different body shapes and poses and reducing the need for data retransmission due to insufficient model accuracy. By using the maximum probability of heatmaps to roughly locate key points, and then using offsets for correction, the positioning accuracy of human key points is improved, reducing repeated detection and data correction caused by positioning errors, further optimizing transmission efficiency, and solving the problem of data transmission latency and its impact on transmission efficiency in current user pose data acquisition and processing.

[0092] Figure 8 This is a schematic diagram of the process of a display device filtering human key points according to some embodiments of this application. In some embodiments, after the display device 200 obtains the second position of each human key point, it can obtain the probability value corresponding to each human key point in the heat map, and then compare the probability value with the probability threshold, delete human key points with probability values ​​lower than the probability threshold, and retain human key points with probability values ​​higher than or equal to the probability threshold.

[0093] For example, after the model outputs a heatmap, for each keypoint's corresponding heatmap channel (e.g., 17 channels for 17 keypoints), its probability value (i.e., the pixel value at that location in the heatmap) is extracted. This value represents the model's confidence that the location belongs to the corresponding keypoint, ranging from [0,1]. Simultaneously, a probability threshold is set, and the probability value of each keypoint is compared with a preset probability threshold (e.g., 0.3). If the probability value is ≥ the threshold, the keypoint is retained; if the probability value is < the threshold, the keypoint is deleted. The final output only contains keypoints with high confidence (≥ the threshold), forming a filtered set of keypoints for subsequent interactive applications. The heatmap probability value reflects the model's confidence in the keypoint location. Thus, threshold filtering eliminates low-confidence keypoints (such as erroneous points caused by occlusion, blurring, or false detection), ensuring that only high-confidence keypoints participate in subsequent interactive logic. This process distinguishes between real keypoints and noise by quantifying confidence, avoiding misoperations (such as incorrect gesture recognition), thereby improving the accuracy of keypoint localization and user experience.

[0094] The above describes a scenario for single-person posture detection. In real-world applications, there may be situations where multiple people need to be detected. Based on this, the display device can perform the following processing. Figure 9 This is a schematic diagram illustrating the process of a display device performing multi-person posture detection according to some embodiments of this application. In some embodiments, after obtaining a heatmap containing multiple human key points and offsets, the display device 200 can detect all human key points included in the heatmap, and perform clustering processing on all human key points to obtain multiple clusters. Then, it counts the number of clusters corresponding to the clusters after clustering processing to obtain the number of users included in the heatmap. When the number of users corresponds to one user, the first position corresponding to each of the multiple human key points is determined according to the heatmap. And, the first position of each human key point is adjusted according to the offset corresponding to each human key point to obtain the second position of each human key point. When the number of users corresponds to multiple users, for each human key point corresponding to a user, the steps of determining the first position corresponding to each of the multiple human key points according to the heatmap and adjusting the first position of each human key point according to the offset corresponding to each human key point to obtain the second position of each human key point are executed respectively.

[0095] For example, in the pose recognition process in a multi-person scenario, the display device 200 first outputs a heatmap using a human pose recognition model (such as PoseNet) to detect all possible human key points in the image (such as 17 key points including the tip of the nose, shoulder, and elbow). Then, cluster analysis (such as using DBSCAN or K-means algorithms) is performed on all detected key points, dividing them into multiple clusters based on their spatial distribution in the image (such as positional similarity and distance thresholds). Each cluster represents a potential user individual. The number of clusters obtained after clustering is counted, with each cluster corresponding to one user, thus determining the number of users in the image (e.g., 2 clusters represent 2 users). For all key points of a single user, the location with the highest probability value is found in the heatmap channel, serving as the initial coordinates (first position) of the key point. The first position is fine-tuned by combining the offset output by the model; the offset represents the offset value of the key point within a local area of ​​the heatmap, and the final coordinates (second position) are obtained through weighted calculation. In a multi-user scenario, the localization process is performed independently for the key points of each cluster (i.e., each user). First, the initial positions of all key points for the user are determined in the heatmap, and then the second positions are obtained through offset correction. Web Workers or multithreading techniques are used to process the key points of different users in parallel, avoiding main thread blocking. This way, key points of different users can be distinguished through clustering algorithms, avoiding key point confusion when multiple users overlap. Furthermore, clustering processing does not require a pre-set number of users and can dynamically adapt to different user scenarios. Simultaneously, after differentiating users, key point localization tasks can be processed separately for each user, and the coordinates of key points in different clusters can be calculated in parallel, reducing the processing pressure on a single thread.

[0096] Figure 10 This document illustrates the pose recognition process of a display device for single-person and multi-person scenes according to some embodiments of this application. In some embodiments, the display device 200 generates a heatmap representing the probability distribution of human keypoint locations using a model. Human keypoints are detected from the generated heatmap, typically by identifying locations with higher probability values ​​as candidate keypoints. Based on the number, distribution, or other characteristics of the detected keypoints, it is determined whether the current image represents a multi-person scene. If it is a multi-person scene, instance segmentation is first performed to distinguish different human instances. For each segmented instance, its keypoints are clustered to ensure that each keypoint belongs to the correct human instance, ultimately outputting the pose information of multiple people. If it is a single-person scene, the detected keypoints are processed directly to output the pose information of a single person.

[0097] Based on the display device 200 described above, some embodiments of this application also provide a method for determining the location of key points on the human body, which can be applied to the display device 200 in the above embodiments. In some embodiments, the method may include the following: In response to a command to detect the user's posture, the camera is invoked via the browser's real-time web communication function to acquire the video stream data; The current video frame in the video stream data is input into the human pose recognition model based on TensorFlow.js integrated in the browser; Based on TensorFlow.js, the human pose recognition model is run in the browser using a web graphics library to obtain a heatmap and offset containing multiple human key points; the human pose recognition model is used to detect and locate the human key points; the heatmap is used to characterize the probability distribution of the human key points; the offset is used to correct the position of the human key points in the heatmap. The first positions corresponding to the multiple key points of the human body are determined based on the heat map; The first position of each human body key point is adjusted according to the offset corresponding to each human body key point to obtain the second position of each human body key point.

[0098] As can be seen from the above technical solutions, the aforementioned method eliminates traditional dedicated hardware such as sensors / radar in the process of user posture data acquisition and processing. It only uses a camera to collect data and directly acquires the video stream using WebRTC technology natively supported by the browser, without requiring additional equipment. TensorFlow.js is used to integrate the human posture recognition model into the user device, enabling on-device inference and avoiding data uploads to the cloud, thus eliminating transmission latency at its source. Model inference is completed on the browser side, using WebGL acceleration to improve processing efficiency and ensure real-time performance, solving the problem of data transmission latency and its impact on transmission efficiency in current user posture data acquisition and processing processes.

[0099] The same or similar parts among the various embodiments in this specification can be referred to mutually, and will not be repeated here.

[0100] Those skilled in the art will clearly understand that the techniques in the embodiments of the present invention can be implemented using software plus necessary general-purpose hardware platforms. Based on this understanding, the technical solutions in the embodiments of the present invention, or the parts that contribute to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods of various embodiments or certain parts of the embodiments of the present invention.

[0101] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application.

[0102] For ease of explanation, the above description has been provided in conjunction with specific embodiments. However, the above exemplary discussion is not intended to be exhaustive or to limit the embodiments to the specific forms disclosed above. Various modifications and variations can be obtained based on the above teachings. The selection and description of the above embodiments are for the purpose of better explaining the principles and practical applications, thereby enabling those skilled in the art to better utilize the described embodiments and various different variations of embodiments suitable for specific use considerations.< / canvas> < / video> < / canvas> < / canvas> < / video> < / canvas> < / video> < / canvas> < / video> < / canvas> < / video> < / canvas> < / video>

Claims

1. A display device, characterized in that, include: The display is configured to show the user interface; The camera is configured to capture video stream data of the user's posture. The controller is configured as follows: In response to a command to detect the user's posture, the camera is invoked via the browser's real-time web communication function to obtain the video stream data; The current video frame in the video stream data is input into the human pose recognition model based on TensorFlow.js integrated in the browser; Based on the TensorFlow.js, the human pose recognition model is run in the browser through a web graphics library to obtain a heatmap and offset containing multiple human key points. The human posture recognition model is used to detect and locate key points of the human body; The heatmap is used to characterize the probability distribution of the key points of the human body; the offset is used to correct the position of the key points of the human body in the heatmap; The first positions corresponding to the multiple key points of the human body are determined based on the heat map; The first position of each human body key point is adjusted according to the offset corresponding to each human body key point to obtain the second position of each human body key point.

2. The display device according to claim 1, characterized in that, Before the controller inputs the current video frame from the video stream data into the human pose recognition model based on TensorFlow.js integrated in the browser, it is specifically configured as follows: Obtain the image data corresponding to the current video frame; Scale the image data to a preset size; Convert the scaled image data from RGBA four-channel format to RGB three-channel format; The pixel values ​​of the image data in the RGB channels are normalized from integer format to floating-point format to obtain a normalized floating-point array.

3. The display device according to claim 2, characterized in that, The controller, based on TensorFlow.js, runs the human pose recognition model in the browser via a web graphics library to obtain a heatmap and offset containing multiple human key points, specifically configured as follows: The normalized floating-point array is reshaped into a three-dimensional tensor according to the three dimensions of height, width, and number of channels; The three-dimensional tensor is input into the human pose recognition model; Using the inference capabilities of TensorFlow.js, the human pose recognition model is run in the browser via a web graphics library to perform inference calculations on the three-dimensional tensor, resulting in a heatmap and offset containing multiple human key points.

4. The display device according to claim 1, characterized in that, Before the controller inputs the current video frame from the video stream data into the human pose recognition model integrated in the browser based on TensorFlow.js, it is also configured to: The video stream data is bound to the video playback tag in the browser's built-in Hypertext Markup Language to play the video stream data; Draw the current video frame corresponding to the playing video stream data onto the canvas; Extract the pixel data of the current video frame.

5. The display device according to claim 1, characterized in that, The controller determines the first positions corresponding to the multiple key points of the human body based on the heat map, and is specifically configured as follows: Find the location of the maximum probability for each key point of the human body in the heat map; The position of the maximum probability is taken as the first position corresponding to each of the key points of the human body.

6. The display device according to claim 1, characterized in that, The controller adjusts the first position of each human key point according to the offset corresponding to each human key point to obtain the second position of each human key point, specifically configured as follows: Extract the first direction offset and the second direction offset corresponding to each of the first positions from the offset; Add the first coordinate of each first position to the first direction offset to obtain the first coordinate of each second position; Add the second coordinate of each first position to the second direction offset to obtain the second coordinate of each second position.

7. The display device according to claim 1, characterized in that, After the controller adjusts the first position of each human key point according to the offset corresponding to each human key point to obtain the second position of each human key point, it is further configured to: Obtain the probability value corresponding to each key point of the human body in the heat map; The probability value is compared with the probability threshold; Delete human key points whose probability value is lower than the probability threshold, and retain human key points whose probability value is higher than or equal to the probability threshold.

8. The display device according to claim 1, characterized in that, The controller, based on TensorFlow.js, runs the human pose recognition model in the browser via a web graphics library to obtain a heatmap containing multiple human key points and offsets. After this step, it is further configured to: Detect all key human body points included in the heat map; Clustering was performed on all the aforementioned key human body points to obtain multiple clusters; The number of clusters corresponding to the clusters after clustering is counted to obtain the number of users contained in the heatmap. When the number of users corresponds to one user, the first positions corresponding to multiple human body key points are determined according to the heat map; and the first positions of each human body key point are adjusted according to the offset corresponding to each human body key point to obtain the second position of each human body key point. When the number of users corresponds to multiple users, for each user's corresponding human body key point, the steps of determining the first position corresponding to the multiple human body key points according to the heat map, and adjusting the first position of each human body key point according to the offset corresponding to each human body key point, are performed to obtain the second position of each human body key point.

9. The display device according to claim 1, characterized in that, The controller, in response to a user posture detection command, invokes the camera via the browser's real-time web communication function to acquire the video stream data, specifically configured as follows: In response to a command to detect the user's posture, a request to acquire video stream data is initiated to the camera via the browser's real-time web communication function; Receive the video stream data returned by the camera in accordance with the acquisition request.

10. A method for determining the location of key points on the human body, applied to the display device according to any one of claims 1-9, the display device comprising a display, a camera, and a controller, characterized in that, The method includes: In response to a command to detect the user's posture, the camera is invoked via the browser's real-time web communication function to obtain the video stream data; The current video frame in the video stream data is input into the human pose recognition model based on TensorFlow.js integrated in the browser; Based on TensorFlow.js, the human pose recognition model is run in the browser using a web graphics library to obtain a heatmap and offset containing multiple human key points; the human pose recognition model is used to detect and locate the human key points; the heatmap is used to characterize the probability distribution of the human key points; the offset is used to correct the position of the human key points in the heatmap. The first positions corresponding to the multiple key points of the human body are determined based on the heat map; The first position of each human body key point is adjusted according to the offset corresponding to each human body key point to obtain the second position of each human body key point.

Images