Method and apparatus for fitness motion assessment, smart television

Abstract

The application relates to the technical field of smart televisions, and discloses a method for evaluating fitness actions, which is applied to a smart television and comprises the following steps: acquiring point cloud data of a user; generating a target tensor according to the point cloud data; inputting the target tensor into a preset skeleton point extraction network model to obtain skeleton point coordinates of each preset joint of the user; acquiring a total deviation value of the skeleton point coordinates relative to a standard action template according to the skeleton point coordinates and the preset standard action template; and evaluating the fitness actions of the user according to the total deviation value to obtain an evaluation result. The method can protect the privacy of the user while improving the accuracy of fitness action evaluation on the television side. The application further discloses a device for evaluating fitness actions and a smart television.

Description

Technical Field

[0001] This application relates to the field of smart TV technology, such as a method and device for fitness movement assessment, and a smart TV. Background Technology

[0002] With increasing health awareness and the widespread adoption of smart home devices, motion-sensing fitness applications based on smart TVs are gaining popularity. Current fitness movement evaluation and correction solutions can be categorized as follows: First, camera-based solutions use RGB or depth cameras to capture user movements and compare them to standard movements. This approach suffers from privacy leaks and poor low-light performance. Second, inertial measurement unit (IMU)-based solutions install accelerometers and gyroscopes on the user's body or clothing, using sensor fusion to calculate limb posture. This approach requires wearable devices, increasing user burden and discomfort; moreover, IMUs are inherently relative measurements, leading to cumulative drift and low posture estimation accuracy. These shortcomings prevent current technologies from achieving accurate fitness movement evaluation while protecting user privacy.

[0003] It should be noted that the information disclosed in the background section above is only used to enhance the understanding of the background of this application, and therefore may include information that does not constitute prior art known to those skilled in the art. Summary of the Invention

[0004] To provide a basic understanding of some aspects of the disclosed embodiments, a brief summary is given below. This summary is not intended as a general commentary, nor is it intended to identify key / important components or describe the scope of protection of these embodiments, but rather as a prelude to the detailed description that follows.

[0005] This disclosure provides a method and apparatus for evaluating fitness movements, as well as a smart TV, which can achieve more accurate evaluation of users' fitness movements while protecting user privacy.

[0006] In some embodiments, the method is applied to a smart TV, and the method includes: acquiring point cloud data of a user; generating a target tensor based on the point cloud data; inputting the target tensor into a preset skeletal point extraction network model to obtain skeletal point coordinates of each preset joint of the user; obtaining the total deviation value of the skeletal point coordinates relative to the preset standard movement template based on the skeletal point coordinates and the preset standard movement template; and evaluating the user's fitness movements based on the total deviation value to obtain an evaluation result.

[0007] In some embodiments, the device is applied to a smart TV, and the device includes: a point cloud data acquisition module configured to acquire point cloud data of a user; a target tensor generation module configured to generate a target tensor based on the point cloud data; a skeletal point coordinate acquisition module configured to input the target tensor into a preset skeletal point extraction network model to obtain skeletal point coordinates of each preset joint of the user; a comparison module configured to obtain a total deviation value of the skeletal point coordinates relative to the preset standard movement template based on the skeletal point coordinates and the preset standard movement template; and an evaluation module configured to evaluate the user's fitness movements based on the total deviation value to obtain an evaluation result.

[0008] In some embodiments, the apparatus includes a processor and a memory storing program instructions, the processor being configured to execute the above-described method for fitness movement assessment when the program instructions are executed.

[0009] In some embodiments, the smart TV includes: a smart TV body; and the aforementioned device for fitness movement assessment is installed on the smart TV body.

[0010] The method, apparatus, and smart TV for fitness movement evaluation provided in this disclosure can achieve the following technical effects: It obtains the skeletal coordinates of each preset joint of the user using the user's 3D point cloud data, and obtains the total deviation value of the skeletal coordinates relative to the preset standard movement template based on the skeletal coordinates and the template. This total deviation value is then used to evaluate the user's fitness movements. This eliminates the need to acquire the user's movements through a camera, avoiding privacy leaks. Furthermore, by calculating the difference between the skeletal coordinates of each preset joint and the standard movement template, a more accurate evaluation of the user's fitness movements can be achieved. Thus, while protecting user privacy, the accuracy of fitness movement evaluation on the TV is improved.

[0011] The above general description and the description below are exemplary and illustrative only and are not intended to limit this application. Attached Figure Description

[0012] One or more embodiments are illustrated by way of example with reference to the accompanying drawings. These illustrations and drawings do not constitute a limitation on the embodiments. Elements having the same reference numerals in the drawings are shown as similar elements. The drawings are not to be scaled. And wherein: Figure 1 This is a schematic diagram of a method for evaluating fitness movements provided in an embodiment of this disclosure; Figure 2 This is a schematic diagram of a method for generating point cloud data of a user provided in an embodiment of this disclosure; Figure 3This is a schematic diagram of a skeletal point extraction network model provided in an embodiment of this disclosure; Figure 4 This is a schematic diagram of a device for fitness movement assessment provided in an embodiment of this disclosure; Figure 5 This is a schematic diagram of another device for fitness movement assessment provided in an embodiment of this disclosure; Figure 6 This is a schematic diagram of a smart TV provided in an embodiment of this disclosure. Detailed Implementation

[0013] To provide a more detailed understanding of the features and technical content of the embodiments of this disclosure, the implementation of the embodiments of this disclosure will be described in detail below with reference to the accompanying drawings. The accompanying drawings are for illustrative purposes only and are not intended to limit the embodiments of this disclosure. In the following technical description, for ease of explanation, several details are used to provide a full understanding of the disclosed embodiments. However, one or more embodiments may still be implemented without these details. In other cases, well-known structures and devices may be simplified in their depiction to simplify the drawings.

[0014] The terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this disclosure are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate for the embodiments of this disclosure described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion.

[0015] Unless otherwise stated, the term "multiple" means two or more.

[0016] In this embodiment of the disclosure, the character " / " indicates that the objects before and after it are in an "or" relationship. For example, A / B means: A or B.

[0017] The term "and / or" describes an association between objects, indicating that three relationships can exist. For example, A and / or B means: A or B, or A and B.

[0018] The term "correspondence" can refer to an association or binding relationship. The correspondence between A and B means that there is an association or binding relationship between A and B.

[0019] In this embodiment of the disclosure, the smart TV incorporates microprocessors, sensor technology, and network communication technology, and features intelligent control, intelligent sensing, and intelligent applications. The operation of the smart TV often relies on the application and processing of modern technologies such as the Internet of Things, the Internet, and electronic chips.

[0020] The fitness movement assessment method provided in this disclosure is applied to smart TVs. The smart TV SoC (System on Chip) integrates heterogeneous computing units: CPU (Central Processing Unit), GPU (Graphics Processing Unit), NPU (Neural Processing Unit), and DSP (Digital Signal Processor), possessing an NPU computing power of 4-5 TOPS. In this disclosure, the DSP handles the signal processing pipeline of the radar front end, the NPU accelerates the inference of the skeletal point extraction network, and the CPU is responsible for standard movement template management, deviation quantification, and error correction decision engine. All calculations are completed within the smart TV SoC, requiring no external acceleration card or cloud service. Furthermore, users do not need to wear additional devices; they can obtain movement guidance and feedback solely through the TV's sensing module. This improves the accuracy of fitness movement assessment on the TV while protecting user privacy.

[0021] Combination Figure 1 As shown, this disclosure provides a method for evaluating fitness movements, including: Step S101: The smart TV acquires the user's point cloud data.

[0022] In one embodiment, the smart TV responds to a user's fitness movement assessment command by acquiring the user's point cloud data. The point cloud data is a set of three-dimensional spatial points, each point including three-dimensional spatial coordinates and echo intensity.

[0023] In step S102, the smart TV generates a target tensor based on the user's point cloud data.

[0024] In step S103, the smart TV inputs the target tensor into the preset skeletal point extraction network model to obtain the skeletal point coordinates of each preset joint of the user.

[0025] Among them, the skeletal point coordinates are a set of three-dimensional spatial coordinates of the human body's preset joints. The preset joints include multiple joints in the head, left shoulder, right shoulder, left elbow, right elbow, left wrist, right wrist, left hip, right hip, left knee, right knee, left ankle, right ankle, left foot, right foot, lumbar spine, and neck.

[0026] In step S104, the smart TV obtains the total deviation value of the skeletal point coordinates relative to the standard action template based on the skeletal point coordinates and the preset standard action template.

[0027] In step S105, the smart TV evaluates the user's fitness movements based on the total deviation value and obtains the evaluation results.

[0028] The method for fitness movement evaluation provided in this disclosure obtains the skeletal coordinates of each preset joint of the user through point cloud data. Based on the skeletal coordinates and a preset standard movement template, the total deviation value of the skeletal coordinates relative to the standard movement template is obtained, and then the user's fitness movement is evaluated using this total deviation value. This eliminates the need to capture the user's movements through a camera, avoiding privacy leaks. Furthermore, by calculating the differences between the skeletal coordinates of each preset joint and the standard movement template, a more accurate evaluation of the user's fitness movements can be achieved. Thus, while protecting user privacy, the accuracy of fitness movement evaluation on the television is improved.

[0029] The smart TV is equipped with a millimeter-wave radar module. In one embodiment, the millimeter-wave radar module is installed on the top or bottom bezel of the smart TV, adopts a 60GHz FMCW (Frequency Modulated Continuous Wave) system, includes a 2-transmit 4-receive antenna array (MIMO), and connects to the TV SoC via an SPI or USB interface, with a frame rate of 10-30fps.

[0030] Optionally, acquiring the user's point cloud data includes: transmitting a linear frequency modulated signal via a millimeter-wave radar module and receiving human body reflection echoes to obtain echo sampling data. Signal processing operations are then performed on the echo sampling data to generate the user's point cloud data. The point cloud data is a collection of three-dimensional spatial points, each point including its three-dimensional spatial coordinates and echo intensity.

[0031] Human body reflection data is detected using a millimeter-wave radar module, and then converted into point cloud data. Compared to cameras, this method offers advantages such as better privacy protection, immunity to lighting conditions, and the ability to penetrate clouds and fog. In some embodiments, signal processing of the echo sampling data is performed on the DSP core of the TV SoC.

[0032] In some embodiments, the millimeter-wave radar module transmits a linear frequency modulated (LFM) signal in each set period, such as 50 microseconds. The receiving antenna samples the echo reflected from the human body to obtain one-dimensional ADC (analog-to-digital converter) data. A frame contains a preset number of chirps (LFM signals), and the sampling data from multiple receiving antennas together constitute a three-dimensional data cube, i.e., echo sampling data. This three-dimensional data cube corresponds to the distance sampling point, the chirp sequence, and the multi-channel received data in the fast time dimension, slow time dimension, and antenna dimension, respectively.

[0033] Combination Figure 2 As shown, further signal processing operations are performed on the echo sampling data to generate the user's point cloud data, including the following steps: Step S201: The echo sampling data acquired by the millimeter-wave radar module is processed in the range dimension to obtain a range profile. The range profile contains multiple discrete range bins, each corresponding to a specific signal round-trip time, i.e., the target distance.

[0034] Step S202: Perform velocity dimension processing on the echo sampling data within the same distance cell to obtain Doppler velocity information.

[0035] Step S203: The range profile and Doppler velocity information are used to construct a range-Doppler map (RDM).

[0036] Step S204: Perform target detection on the range-Doppler graph (RDM) and extract valid reflection points.

[0037] Step S205: Cluster the extracted valid reflection points to obtain the reflection points belonging to the target human body, and determine the reflection points belonging to the target human body as the user's point cloud data.

[0038] Furthermore, the echo sampling data is processed in the range dimension to obtain a range profile, including: performing an FFT (Fast Fourier Transform) operation along the fast time dimension on the echo sampling data of each chirp to obtain the range profile. Each range unit (Bin) corresponds to a specific signal round-trip time, i.e., the target distance. This is achieved through calculation... Obtain the target distance, where, d For the target distance, c At the speed of light, f beat The difference frequency between the transmitted and received signals. S This represents the frequency modulation slope.

[0039] Furthermore, the echo sampling data within the same range cell is processed in the velocity dimension to obtain Doppler velocity information, including: performing FFT operation on the phase change sequence of the echo sampling data within the same range cell Bin along the slow time dimension to obtain Doppler velocity information.

[0040] Through calculation Obtain Doppler velocity information, where λ is the radar wavelength. f doppler This is the Doppler frequency shift.

[0041] A range-Doppler map (RDM) is a two-dimensional matrix obtained by performing range-dimensional FFT and Doppler-dimensional FFT on the echo sampled data. The horizontal axis corresponds to the range, and the vertical axis corresponds to the radial velocity. Each pixel value represents the echo energy within that range-velocity cell.

[0042] Furthermore, target detection is performed on the RDM to extract valid reflection points, including: using the ordered statistical constant false alarm rate (OS-CFAR) algorithm on the RDM for adaptive threshold detection to extract valid reflection points. The detection threshold is set to... ,in, X (k) For reference, the first [number] after sorting within the window k There are several values, where α is the threshold factor. Compared to the traditional cell-averaged CA-CFAR algorithm, the OS-CFAR algorithm has a lower false alarm rate and a higher detection probability in multi-target dense scenes.

[0043] Furthermore, the extracted effective reflection points are clustered to obtain reflection points belonging to the target human body. This includes: performing DBSCAN (density-based spatial clustering) processing on the effective reflection points, clustering point clouds belonging to the same human body into a single cluster, and filtering out clutter interference points to obtain reflection points belonging to the target human body. In this embodiment, the clustering neighborhood radius is [missing information]. eps =0.15m, minimum number of points minPts =5.

[0044] Optionally, a target tensor is generated based on the point cloud data, including: obtaining the point cloud density of the point cloud data; determining the voxel resolution based on the point cloud density; and performing voxelization on the point cloud data according to the voxel resolution to generate the target tensor.

[0045] Voxelization is the process of discretizing a continuous three-dimensional space into regular mesh cells. Unstructured point cloud data is transformed into a regular four-dimensional target tensor, i.e., three-dimensional spatial coordinates plus echo intensity, which serves as the input to the skeleton point extraction network model. The size of this target tensor is 64×64×64×4.

[0046] A denser point cloud indicates a closer user to the TV, while a sparser point cloud indicates a farther user. This transformation of point cloud data from an unstructured sparse set of points into a regular grid representation, and the adaptive adjustment of voxel resolution based on the point cloud density of the current frame, further eliminates the impact of the distance between the user and the smart TV on the overall deviation value.

[0047] Furthermore, determining the voxel resolution based on the point cloud density includes: performing a lookup operation in a preset first data table based on the point cloud density to find the voxel resolution corresponding to the point cloud density. The preset first data table stores the correspondence between point cloud density and voxel resolution.

[0048] Optionally, the preset skeletal point extraction network model has a two-layer architecture. The preset skeletal point extraction network model includes: a preset sparse convolutional network and a preset temporal modeling network. The sparse convolutional network is used to extract spatial features from the input target tensor and outputs a spatial feature map to the temporal modeling network. The temporal modeling network is used to model the temporal dependencies of the spatial feature map and outputs the coordinates of the skeletal points of each preset joint.

[0049] In some embodiments, the preset sparse convolutional network is a sparse 3D convolutional backbone network (SparseConvBackbone), which performs convolution computation only at non-zero voxel locations in the input target tensor, and does not compute empty voxels. Combined with Figure 3 As shown, the sparse 3D convolutional backbone network 301 comprises four cascaded downsampled residual blocks: Sparse ConvBlock 1, Sparse Conv Block 2, Sparse Conv Block 3, and Sparse Conv Block 4. Sparse ConvBlock 1 has 32 channels and a stride of 1. Sparse Conv Block 2 has 64 channels and a stride of 2. Sparse ConvBlock 3 has 128 channels and a stride of 2. Sparse Conv Block 4 has 256 channels and a stride of 2. Each residual block performs spatial downsampling of the feature map through sparse convolution, progressively expanding the receptive field and extracting high-dimensional semantic features, outputting a spatial feature map. The total number of parameters in this network is 1.6 million. The computational cost of employing a sparse 3D convolutional backbone network is linearly related only to the number of non-zero voxels, avoiding the problem of traditional dense 3D convolutions where the computational cost increases cubically with resolution (O(n³)) on high-resolution voxel grids. This makes real-time inference after 3D voxelization possible on TV SoCs. Compared to traditional dense 3D convolutions, the sparse 3D convolutional structure in this embodiment reduces the computational cost by approximately two orders of magnitude.

[0050] The preset temporal modeling network is a lightweight temporal transformer (LTT) network. This TLTT network is used to perform temporal dependency modeling on the spatial feature maps output by the sparse 3D convolutional backbone network. The TLTT network receives the spatial feature maps output by the sparse 3D convolutional backbone network and combines them with spatial feature maps from previous consecutive frames. Through a cross-frame attention mechanism, it captures the motion dependencies of bone points in the temporal dimension and outputs the coordinates of bone points for each preset joint in the current frame.

[0051] Combination Figure 3 As shown, the temporal Transformer network 302 includes a position encoding module, a Transformer encoder, and a fully connected output layer. The total number of parameters in the temporal Transformer network is 350,000.

[0052] The positional encoding module adds learnable positional codes to the spatial feature maps of each input frame. These learnable positional codes serve as trainable parameters of the network and are updated during training, rather than using a fixed sinusoidal positional code. They adaptively adjust the positional representation based on data distributions of different body sizes and movement ranges, enhancing the network's accuracy in extracting skeletal points from users of different body types.

[0053] The Transformer encoder is used to model cross-frame temporal dependencies of the spatial feature map. This Transformer encoder is a two-layer cascaded encoder. Each layer employs a four-head linear attention mechanism, which reduces the computational complexity of attention from quadratic (O(n²)) of the input sequence length to linear (O(n)), thereby reducing inference latency on TV SoCs.

[0054] The fully connected output layer is used to map the temporal features output by the encoder to the coordinates of the skeletal points of each preset joint. For example, if the number of preset joints is 15 to 25, and each joint corresponds to three-dimensional spatial coordinates (x, y, z), then the output dimension of the fully connected output layer is J×3, where J is the number of preset joints.

[0055] In some embodiments, the skeleton point extraction network model is deployed on the NPU of a smart TV SoC. This NPU has a computing power of 4 to 5 TOPS. To adapt to the limited computing power and memory bandwidth resources of the NPU and achieve real-time inference, this embodiment employs optimization strategies including INT8 quantization, operator fusion, and NPU-dedicated memory layout to optimize the skeleton point extraction network model. Specifically, INT8 quantization quantizes the 32-bit floating-point weights and activation values ​​of the skeleton point extraction network model into 8-bit integers. After INT8 quantization, the model size is compressed to approximately 25% of the original floating-point model, and the inference speed is increased by 2 to 3 times, enabling the network to complete single-frame inference with lower latency on the NPU. Operator fusion merges the convolutional layers (Conv), batch normalization layers (BN), and activation function layers (ReLU) in the skeleton point extraction network model into a single computational unit. When the fused computational unit is executed on the NPU, there is no need to write intermediate computation results back to memory, reducing memory bandwidth consumption and kernel startup overhead. The dedicated memory layout of the NPU adjusts the feature map tensor layout during the inference process of the skeletal point extraction network model to the NPU-friendly NHWC format (batch N, height H, width W, channels C). After the above optimization strategy, the single-frame inference time of the skeletal point extraction network model on the NPU does not exceed 15 milliseconds, and the total model file size is less than 2 megabytes. Therefore, the smart TV can process radar point cloud data and output skeletal point coordinates in real time at a frame rate of no less than 30 frames per second, meeting the real-time and response speed requirements of fitness motion correction scenarios.

[0056] Before obtaining the total deviation of the skeletal point coordinates from the standard motion template based on the skeletal point coordinates and the preset standard motion template, the process also includes obtaining the preset standard motion template. The preset standard motion template is the standard coordinate of the skeletal points of each joint under the target standard motion, and the target standard motion is the standard motion of the motion type corresponding to the user's skeletal point coordinates.

[0057] Optionally, the preset standard motion templates are obtained as follows: Skeletal point coordinates are input into a preset motion classification network model, and the corresponding motion type is output. Based on the motion type, a search operation is performed in a preset standard motion template library to find the standard motion template corresponding to the motion type. The standard motion template includes the standard coordinates of the skeletal points of each preset joint. The standard motion template library stores the correspondence between motion types and standard motion templates. For example, motion types include squats, push-ups, planks, lunges, burpees, and downward-facing dog in yoga.

[0058] In one embodiment, the action classification network model is a lightweight fully connected classification network. Its input is the 3D skeletal coordinates of each preset joint in the current frame, output by the skeletal point extraction network, and its output is the probability distribution of action categories. Specifically, the action classification network model includes: an input layer that receives a skeletal coordinate vector of dimension J×3; a global average pooling layer or flattening layer that converts the input vector into a fixed-length one-dimensional feature; at least one fully connected layer that maps the features to the action type space; and a softmax layer that outputs the probability of each action type. The action type with the highest probability output by the action classification network model, exceeding a preset threshold (e.g., 0.7), is determined as the action type corresponding to the user's skeletal coordinates.

[0059] Optionally, the preset standard action template is obtained in the following way: obtain the currently playing content and current timestamp of the smart TV, determine the current action type based on the currently playing content and current timestamp; perform a search operation in the preset standard action template library based on the current action type to find the standard action template corresponding to the action type.

[0060] In some embodiments, after obtaining a preset standard movement template, the method further includes: obtaining the user's shoulder width and height parameters; determining a lateral scaling factor based on the user's shoulder width parameters; determining a longitudinal scaling factor based on the user's height parameters; and anisotropically scaling the standard coordinates of the skeletal points of each preset joint in the standard movement template according to the lateral and longitudinal scaling factors. Since there are height differences among different users, directly comparing the extracted skeletal point coordinates with the standard coordinates of the skeletal points in the standard movement template will lead to severe distortion of the evaluation results. Even if a tall user performs the movement correctly, the absolute value of their skeletal point coordinates may far exceed the template value, while the value for a short user will be smaller. Therefore, in this embodiment, the standard coordinates of the skeletal points are adaptively scaled for height before matching them with the standard movement template. This allows for more precise location of specific errors at the joint level, making the evaluation of the user's fitness movements more accurate. The matching accuracy can be improved by approximately 20-30% after scaling.

[0061] Furthermore, the user's shoulder width and height parameters are obtained, including: calculating the 3D Euclidean distance between the left and right shoulder joints based on the skeletal point coordinates output by the skeletal point extraction network model to obtain the user's shoulder width parameters; and calculating the vertical distance from the top of the head to the soles of the feet to obtain the user's height parameters.

[0062] Furthermore, the lateral scaling factor is determined based on the user's shoulder width parameter, including: determining the lateral scaling factor by the ratio between the user's shoulder width parameter and the shoulder width parameter of the standard motion template.

[0063] Furthermore, the vertical scaling factor is determined based on the user's height parameters, including: determining the vertical scaling factor by comparing the user's height parameters with the height parameters of the standard motion template.

[0064] Furthermore, the standard coordinates of the skeletal points of each preset joint in the standard motion template are anisotropically scaled according to the horizontal and vertical scaling factors, including: multiplying the horizontal coordinate components of each preset joint in the standard motion template by the horizontal scaling factor; and multiplying the vertical coordinate components of each preset joint in the standard motion template by the vertical scaling factor.

[0065] Optionally, based on the skeletal point coordinates and a preset standard motion template, the total deviation value of the skeletal point coordinates relative to the standard motion template is obtained, including: calculating the three-dimensional spatial deviation value of each preset joint based on the skeletal point coordinates and the standard motion template; and obtaining the total deviation value of the skeletal point coordinates relative to the standard motion template based on each three-dimensional spatial deviation value.

[0066] Furthermore, based on the skeletal point coordinates and the standard motion template, the three-dimensional spatial deviation values ​​of each preset joint are calculated, including: by calculating... Obtain the three-dimensional spatial deviation values ​​of each preset joint. For the first i The three-dimensional spatial deviation value of a preset joint, For the first i The coordinates of the skeletal points of the preset joints and the first i The standard coordinates of a preset joint in three-dimensional space x Difference in direction; For the first i The coordinates of the skeletal points of the preset joints and the first i The standard coordinates of a preset joint in three-dimensional space y Difference in direction; For the first i The coordinates of the skeletal points of the preset joints and the first i The standard coordinates of a preset joint in three-dimensional space z Difference in direction; for x Weighting coefficients for the directional spatial dimension; for y Weighting coefficients for the directional spatial dimension; for z Weighting coefficients for the directional spatial dimension.

[0067] The weighting coefficients for each spatial dimension are set values. Since the ranging accuracy of millimeter-wave radar in the z-axis direction (depth direction) is typically lower than its measurement accuracy in the lateral plane (x-axis and y-axis), this embodiment sets... =1.0, =1.0, =0.5, to reduce the adverse impact of depth direction measurement error on deviation assessment.

[0068] Optionally, obtaining the total deviation value of the skeletal point coordinates relative to the standard motion template based on each three-dimensional spatial deviation value includes: obtaining the weight coefficient of each preset joint according to the motion type; and obtaining the total deviation value of the skeletal point coordinates relative to the standard motion template based on the three-dimensional spatial deviation value of each preset joint and the weight coefficient of each preset joint.

[0069] Furthermore, the weight coefficients of each preset joint are obtained according to the movement type, including: performing a lookup operation in a preset second data table based on the movement type to find the weight coefficients of each preset joint corresponding to the movement type. The preset second data table stores the correspondence between the weight coefficients of each preset joint and the weight coefficients of each preset joint. For example, if the movement type is squat, the weight coefficients of the left knee and right knee are both 3.0, the weight coefficients of the left ankle and right ankle are both 2.0, and the weight coefficients of the left shoulder and right shoulder are both 1.0, etc.

[0070] Furthermore, based on the three-dimensional spatial deviation values ​​of each preset joint and the weight coefficients of each preset joint, the total deviation value of the skeletal point coordinates relative to the standard motion template is obtained, including: by calculating... Obtain the total deviation value. Among them, This represents the total deviation of the user's skeletal point coordinates from the standard motion template. For the first i The weighting coefficients of each preset joint. J The preset number of joints. Overall deviation value. The smaller the value, the closer the user's current action is to the standard action, and the more standardized the action is; conversely, the larger the value, the greater the deviation in the action.

[0071] In some embodiments, user actions are categorized into different deviation levels based on the threshold range in which the overall deviation value falls. The evaluation result is the deviation level to which the user's current fitness action belongs. For example, deviation levels include slight deviation, moderate deviation, or severe deviation.

[0072] Optionally, the user's fitness movements are evaluated based on the total deviation value to obtain an evaluation result, including: if the total deviation value is less than a first deviation threshold, determining that the user's current fitness movement belongs to the slight deviation level; if the total deviation value is greater than or equal to the first deviation threshold and less than a second deviation threshold, determining that the user's current fitness movement belongs to the moderate deviation level; if the total deviation value is greater than or equal to the second deviation threshold, determining that the user's current fitness movement belongs to the severe deviation level. The first deviation threshold is less than the second deviation threshold. For example, the first deviation threshold is 5 cm, and the second deviation threshold is 10 cm.

[0073] Optionally, after obtaining the evaluation results, the method also includes: displaying the evaluation results.

[0074] Optionally, after obtaining the evaluation results, the process also includes: generating corresponding error correction instructions based on the evaluation results, and controlling the smart TV to output the error correction instructions.

[0075] If the evaluation result is at the level of slight deviation, indicating that the user's current fitness movement is close to the standard movement and no error correction intervention is required, the smart TV will display a comparison between the user's current posture and the standard movement posture on the display screen in the form of a semi-transparent skeletal line overlay, without triggering text or voice error correction prompts, so as to avoid excessively interfering with the user's fitness concentration.

[0076] If the evaluation result is at the medium deviation level, indicating that the user's current fitness movement deviates significantly from the standard movement, the joint with the largest deviation value in three-dimensional space is identified, and a yellow highlight circle is generated at the location of the joint on the TV screen to visually mark the joint that needs adjustment; preset text content is generated at the edge or unobstructed area of ​​the smart TV screen to provide error correction prompts, such as "open your knees 5cm more" or "straighten your back a little more"; the text content is automatically generated based on the type of joint with the largest deviation and the direction of the deviation.

[0077] If the assessment result is at the severe deviation level, indicating a significant deviation between the user's current fitness movement and the standard movement, the joint with the largest deviation value in three-dimensional space is identified. A flashing red warning circle is generated at the location of this joint on the TV screen to strongly alert the user. A full-screen semi-transparent overlay is superimposed on the TV screen, and preset text correction instructions are displayed above the overlay, such as "Severe knee valgus, please open your knee outward and keep your toes aligned with your knees." Optionally, the smart TV's audio system is also triggered to output voice prompts to assist the user in obtaining correction information through the auditory channel.

[0078] Optionally, after obtaining the evaluation result, the process further includes: obtaining the duration of the evaluation result, obtaining the user's skill level, obtaining the list of critical safety joints corresponding to the user's current fitness movement, and obtaining the type of fitness content currently being played on the smart TV. Based on one or more of the evaluation result, the duration of the evaluation result, the user's skill level, the list of critical safety joints, and the type of fitness content currently being played, corresponding error correction instructions are generated.

[0079] The duration for obtaining the evaluation result includes: calculating the cumulative number of frames with the same deviation level in consecutive frames and converting it into time length by combining the frame rate.

[0080] The system acquires the user's skill level by retrieving the current user's skill level tag from the smart TV's local storage or a cloud-based historical database. The skill level is dynamically assessed and generated based on the user's historical fitness data (such as historical movement scores, historical error correction trigger frequency, and the smoothness of completing movements), and is divided into three levels: "Beginner," "Intermediate," and "Advanced."

[0081] Retrieve the list of critical safety joints corresponding to the user's current fitness movement, including: retrieving the list of critical safety joints corresponding to the movement type from a preset safety joint configuration table. For example, the critical safety joints for squats include the left knee joint, right knee joint, and lumbar spine; the critical safety joints for push-ups include the left shoulder joint, right shoulder joint, left elbow joint, and right elbow joint.

[0082] Get the type tags of the fitness content currently playing on the smart TV, including instructional video mode, free training mode, and competition mode.

[0083] In some embodiments, if the assessment result is a severe deviation level, the duration exceeds a preset duration, and the list of critical safety joints includes the knee, lumbar spine, and cervical spine, indicating a high risk of sports injury for the current exercise, a full-screen semi-transparent overlay is overlaid on the TV screen displaying preset text correction guidance content, while simultaneously triggering the audio system to output a preset voice warning. For example, the preset duration is 3 seconds. In some embodiments, if the evaluation result is of medium deviation level and the current user's skill level is "beginner", it is determined that the user has a weak ability to perceive the standardization of the action. Then, the joint with the largest three-dimensional spatial deviation value is identified, and a red flashing warning circle is generated at the position of the joint on the TV screen. A full-screen semi-transparent mask is superimposed on the TV screen, and preset text correction guidance content is displayed above the mask.

[0084] In some embodiments, if the fitness content currently being played on the smart TV is in competition mode, the current evaluation results are recorded in the background log for post-competition analysis or replay annotation, without overlaying any error-correction visual elements or voice prompts on the foreground competition screen.

[0085] In some embodiments, when the same joint is repeatedly prompted with error correction within three consecutive frames, it is determined that the user has persistent difficulty adjusting that joint, and the repetitive wording can easily lead to user attention fatigue. In this case: increase the weight coefficient of that joint, for example, by 20%, so that it occupies a larger proportion in the calculation of the overall deviation value in subsequent frames; at the same time, switch the error correction wording and use different semantic expressions to provide error correction guidance, so as to avoid the user ignoring the prompts due to repetition.

[0086] In practical applications, the ADC data acquisition for one frame (containing multiple chirs) is completed by transmitting FMCW signals and receiving echoes through a millimeter-wave radar module, taking approximately 33 milliseconds (corresponding to a 30fps frame rate). On the TV SoC's DSP, the raw ADC data undergoes distance dimension processing, velocity dimension processing, target detection, and clustering to generate point cloud data, taking no more than 8 milliseconds. On the TV SoC's NPU, an INT8-quantized skeletal point extraction network model is run, performing forward inference and outputting skeletal point coordinates, with a single-frame inference time of no more than 15 milliseconds. Height-adaptive scaling, joint-level weighted deviation quantization, and error correction decision engine inference are performed on the skeletal point coordinates, taking no more than 2 milliseconds. The error correction instructions are rendered as visual overlay elements on the TV screen, such as skeletal lines, highlight circles, and semi-transparent masks, taking less than 16 milliseconds (corresponding to a 60fps display refresh rate). The total processing time for each step is approximately 74 milliseconds. To reduce end-to-end perception latency and achieve a smoother interactive experience, this embodiment employs a double-buffered pipeline parallel processing mechanism on the TV SoC. This reduces the end-to-end processing latency of the system, from radar signal acquisition to error correction command output, from 74ms serially per frame to no more than 50ms, making the feedback lag almost imperceptible to the user and enabling them to obtain an immersive and smooth interactive experience.

[0087] Combination Figure 4 As shown in the figure, this disclosure provides a device 400 for fitness movement evaluation, applied to a smart TV. The device includes: a point cloud data acquisition module 401, a target tensor generation module 402, a skeletal point coordinate acquisition module 403, a comparison module 404, and an evaluation module 405. The point cloud data acquisition module is configured to acquire point cloud data of a user. The target tensor generation module is configured to generate a target tensor based on the point cloud data. The skeletal point coordinate acquisition module is configured to input the target tensor into a preset skeletal point extraction network model to obtain the skeletal point coordinates of each preset joint of the user. The comparison module is configured to obtain the total deviation value of the skeletal point coordinates relative to the preset standard movement template based on the skeletal point coordinates and the template. The evaluation module is configured to evaluate the user's fitness movements based on the total deviation value to obtain an evaluation result.

[0088] Combination Figure 5As shown, this disclosure provides an apparatus 500 for fitness movement assessment, including a processor 504 and a memory 501 storing program instructions. Optionally, the apparatus may further include a communication interface 502 and a bus 503. The processor 504, communication interface 502, and memory 501 can communicate with each other via the bus 503. The communication interface 502 can be used for information transmission. The processor 504 can call the program instructions in the memory 501 to execute the fitness movement assessment method described in the above embodiment.

[0089] Furthermore, the logic instructions in the aforementioned memory 501 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium.

[0090] The memory 501, as a computer-readable storage medium, can be used to store software programs and computer-executable programs, such as program instructions / modules corresponding to the methods in the embodiments of this disclosure. The processor 504 executes functional applications and data processing by running the program instructions / modules stored in the memory 501, that is, it implements the method for fitness movement assessment in the above embodiments.

[0091] The memory 501 may include a program storage area and a data storage area. The program storage area may store the operating system and applications required for at least one function; the data storage area may store data created based on the use of the terminal device. Furthermore, the memory 501 may include high-speed random access memory and may also include non-volatile memory.

[0092] Combination Figure 6 As shown, this disclosure provides a smart TV 600, including a smart TV body and the aforementioned device 500 for fitness movement assessment. The device 500 for fitness movement assessment is installed in the smart TV body. The installation relationship described herein is not limited to placement inside the smart TV, but also includes installation connections with other components of the smart TV, including but not limited to physical connections, electrical connections, or signal transmission connections. Those skilled in the art will understand that the device 500 for fitness movement assessment can be adapted to any suitable smart TV body, thereby realizing other feasible embodiments.

[0093] The aforementioned computer-readable storage medium may be a transient computer-readable storage medium or a non-transitory computer-readable storage medium.

[0094] The technical solutions of this disclosure can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes one or more instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the method described in this disclosure. The aforementioned storage medium can be a non-transitory storage medium, including: a USB flash drive, a portable hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk, and other media capable of storing program code; it can also be a transient storage medium.

[0095] The foregoing description and accompanying drawings fully illustrate embodiments of this disclosure to enable those skilled in the art to practice them. Other embodiments may include structural, logical, electrical, procedural, and other changes. The embodiments represent only possible variations. Individual components and functions are optional unless explicitly required, and the order of operation may vary. Parts and features of some embodiments may be included in or replace parts and features of other embodiments. Moreover, the terminology used in this application is for describing embodiments only and is not intended to limit the claims. As used in the description of embodiments and claims, the singular forms “a,” “an,” and “the” are intended to equally include the plural forms unless the context clearly indicates otherwise. Similarly, the term “and / or” as used in this application means including one or more of the associated listed items and all possible combinations thereof. Additionally, when used in this application, the term "comprise" and its variations "comprises" and / or "comprising" refer to the presence of stated features, integrals, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components, and / or groups thereof. Without further limitations, an element defined by the phrase "comprises a..." does not exclude the presence of other identical elements in the process, method, or apparatus that includes said element. In this document, each embodiment may focus on the differences from other embodiments, and similar or identical parts between embodiments can be referred to mutually. For methods, products, etc., disclosed in the embodiments, if they correspond to the method section disclosed in the embodiments, the relevant parts can be referred to the description of the method section.

[0096] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the embodiments of this disclosure. Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0097] The methods and products disclosed in the embodiments herein (including but not limited to devices and equipment) can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For instance, the division of units may be merely a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces, and the indirect coupling or communication connection of devices or units may be electrical, mechanical, or other forms. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to implement this embodiment according to actual needs. In addition, the functional units in the embodiments of this disclosure may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0098] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than that shown in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. In the descriptions corresponding to the flowcharts and block diagrams in the accompanying drawings, the operations or steps corresponding to different blocks may also occur in a different order than disclosed in the description, and sometimes there is no specific order between different operations or steps. For example, two consecutive operations or steps may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. Each block in a block diagram and / or flowchart, and combinations of blocks in a block diagram and / or flowchart, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

Claims

1. A method for evaluating fitness movements, characterized in that, Applied to smart TVs, the method includes: Obtain the user's point cloud data; Generate a target tensor based on the point cloud data; The target tensor is input into a preset skeletal point extraction network model to obtain the skeletal point coordinates of each preset joint of the user. Based on the skeletal point coordinates and the preset standard motion template, obtain the total deviation value of the skeletal point coordinates relative to the standard motion template; The user's fitness movements are evaluated based on the total deviation value to obtain the evaluation result.

2. The method according to claim 1, characterized in that, The smart TV is equipped with a millimeter-wave radar module; the acquisition of the user's point cloud data includes: The millimeter-wave radar module transmits a linear frequency modulated signal and receives human body reflected echoes to obtain echo sampling data. The echo sampling data is processed to generate the user's point cloud data.

3. The method according to claim 1, characterized in that, Based on the point cloud data, a target tensor is generated, including: Obtain the point cloud density of the point cloud data; The voxel resolution is determined based on the point cloud density; The point cloud data is voxelized according to the voxel resolution to generate a target tensor.

4. The method according to claim 1, characterized in that, The preset skeletal point extraction network model includes: a preset sparse convolutional network and a preset temporal modeling network; The sparse convolutional network is used to extract spatial features from the target tensor and output a spatial feature map to the temporal modeling network. The temporal modeling network is used to perform temporal dependency modeling on the spatial feature map and output the coordinates of the skeletal points of each preset joint.

5. The method according to claim 1, characterized in that, Preset standard action templates are obtained through the following methods: Input the coordinates of the skeletal points into a preset action classification network model, and output the corresponding action type; Based on the action type, a search operation is performed in a preset standard action template library to find the standard action template corresponding to the action type; the standard action template includes the standard coordinates of each preset joint.

6. The method according to claim 5, characterized in that, Based on the skeletal point coordinates and a preset standard motion template, obtain the total deviation value of the skeletal point coordinates relative to the standard motion template, including: Based on the skeletal point coordinates and the standard motion template, calculate the three-dimensional spatial deviation value of each preset joint; Based on the deviation values ​​of each three-dimensional space, the total deviation value of the skeletal point coordinates relative to the standard motion template is obtained.

7. The method according to claim 6, characterized in that, Based on the deviation values ​​of each three-dimensional space, the total deviation value of the skeletal point coordinates relative to the standard motion template is obtained, including: Based on the action type, obtain the weight coefficients of each preset joint; The total deviation value is obtained based on the three-dimensional spatial deviation value of each preset joint and the weight coefficient of each preset joint.

8. A device for evaluating fitness movements, characterized in that, The device, used in smart TVs, includes: The point cloud data acquisition module is configured to acquire the user's point cloud data; The target tensor generation module is configured to generate a target tensor based on the point cloud data; The skeletal point coordinate acquisition module is configured to input the target tensor into a preset skeletal point extraction network model to obtain the skeletal point coordinates of each preset joint of the user. The comparison module is configured to obtain the total deviation value of the bone point coordinates relative to the standard action template based on the bone point coordinates and a preset standard action template; The evaluation module is configured to evaluate the user's fitness movements based on the total deviation value and obtain an evaluation result.

9. A device for evaluating fitness movements, comprising a processor and a memory storing program instructions, characterized in that, The processor is configured to perform the method for fitness movement assessment as described in any one of claims 1 to 7 when executing the program instructions.

10. A smart TV, characterized in that, include: The smart TV itself; The device for fitness movement assessment as described in claim 8 or 9 is installed on the smart TV body.

Images