A human posture recognition method and device for a body sensing game machine

Abstract

The application is suitable for the technical field of image recognition, and provides a human posture recognition method and device for a body sensing game machine, the human posture recognition method comprising: inputting a current video frame in a video frame sequence into a pre-trained posture estimation network to obtain initial heat maps of each key point of a human body in the current video frame, and taking the position of a main peak with the largest response value in each initial heat map as an initial position of each key point; matching a compensation path corresponding to an error mode, and correcting the initial position of each key point based on the compensation path to obtain a compensation position of each key point; weighting and fusing the compensation position of each key point and a corresponding motion extrapolation position according to a convergence weight to output a human posture recognition result of the body sensing game machine for the current video frame. The above scheme stably connects the positioning results of each key point in complex scenes such as occlusion, blur, interlacing and boundary, and finally improves the recognition accuracy.

Description

Technical Field

[0001] This invention belongs to the technical field of image recognition, and particularly relates to a method and device for human posture recognition in a motion-sensing game console. Background Technology

[0002] Motion-sensing game consoles are interactive terminals designed for indoor fitness and entertainment scenarios. They rely on cameras to track the user's full-body movements in real time and translate them into game control commands, undertaking functions such as human posture recognition, action semantic understanding, and game feedback. These devices collect continuous image sequences of the user's body areas through ordinary visible light cameras. The human posture estimation data link between the device and the host typically operates in complex indoor lighting and variable background environments. Such environments simultaneously accommodate various factors such as ambient light disturbances, furniture interference, multiple people in the same frame, limb self-occlusion, and extreme body movements, resulting in cluttered image backgrounds and drastically changing target shapes. If the human keypoint estimation process fails, it can cause minor issues such as lagging game character movements and mis-triggered commands, or even major issues such as abrupt changes in posture trajectory or complete loss of the target, directly threatening the smoothness of game operation and immersive experience.

[0003] To address the aforementioned issues, existing technologies for key point localization typically rely solely on the peak value of a single-frame heatmap response as the criterion. In dynamic scenarios involving occlusion, motion blur, multiple people intersecting, and image boundaries, misselection of the main peak and positioning drift are prone to occur. The continuous accumulation of interruptions in the trajectory of key points during periods of occlusion or going out of bounds exacerbates the jitter and distortion in pose recognition. Summary of the Invention

[0004] In view of this, embodiments of the present invention provide a human posture recognition method and device for a motion-sensing game console, in order to solve the technical problem in the prior art that key point localization usually only uses the peak value of a single frame heat map response as the judgment standard, which is prone to main peak misselection and positioning drift in dynamic scenes with occlusion, motion blur, multiple people intermingling and screen boundaries.

[0005] A first aspect of this invention provides a method for human posture recognition in a motion-sensing game console, the method comprising: A sequence of video frames of the human body is collected. The current video frame in the sequence is input into a pre-trained pose estimation network to obtain an initial heatmap of each key point of the human body in the current video frame. The position of the main peak with the largest response value in each initial heatmap is taken as the initial position of each key point. The morphological features of each initial heatmap are extracted, and the error patterns corresponding to each key point are matched based on the morphological features. The error patterns represent the categories of key point observations that fail due to different causes. The error patterns include occlusion diffusion type, blur split type, instance confusion type, out-of-bounds truncation type, and reliable observation type. The compensation path corresponding to the error pattern is matched, and the initial position of each key point is corrected based on the compensation path to obtain the compensation position of each key point; wherein, the compensation path is a processing flow that corrects the initial positioning of key points according to the cause of the corresponding error pattern. Based on the skeletal structure and motion relationships between the compensation positions of each key point, the consistency of the compensation positions of each key point is verified, and the convergence weight of each key point is determined based on the consistency verification results. The compensation positions of each key point and the corresponding motion extrapolation positions are weighted and fused according to the convergence weight to output the human posture recognition result of the motion-sensing game console in the current video frame; wherein, the motion extrapolation positions are obtained by extrapolating the historical trajectories of the corresponding key points in the video frame sequence before the current video frame.

[0006] Furthermore, the step of extracting the morphological features of each initial heatmap and matching the error patterns corresponding to each key point based on the morphological features includes: For each initial heatmap, the ratio between the peak value of the main peak in the initial heatmap and the average response value of the set of pixels in the initial heatmap whose response values ​​exceed a preset response threshold is calculated as the main peak sharpness. Extract the second peak whose response value is second only to the main peak in the initial heatmap, and calculate the ratio between the peak value of the second peak and the peak value of the main peak as the competition degree of the second peak; The number of pixels contained in the pixel set is counted and used as the area of ​​the supporting domain; The number of pixels in the pixel set that are adjacent to the image boundary of the current video frame is counted, and the ratio between the number of adjacent pixels and the area of ​​the supporting domain is calculated as the edge truncation ratio. When the competition level of the secondary peak is higher than the preset competition threshold, and the distance between the secondary peak and the primary peak is greater than the scale prior value of the limb segment where the key point is located, the error mode of the key point is classified as instance confusion type; wherein, the scale prior value represents the estimated length of the limb segment where the key point is located in the image. When the area of ​​the support domain is greater than the preset diffusion threshold and the sharpness of the main peak is lower than the preset sharpness threshold, the error mode of the key point is classified as occlusion diffusion type. When the edge truncation ratio is greater than the preset truncation threshold, the error pattern of the key point is classified as out-of-bounds truncation type. When the sharpness of the main peak is lower than the preset sharpness threshold and the area of ​​the support domain is not greater than the preset diffusion threshold, the error mode of the key point is classified as fuzzy splitting type. When the initial heatmap of the key point does not meet the classification conditions of instance confusion type, occlusion diffusion type, out-of-bounds truncation type and fuzzy split type, the error mode of the key point is classified as reliable observation type; When the initial heatmap of the key point simultaneously meets the classification conditions of multiple error modes, the key point is classified into the error mode with the highest priority according to the preset priority order.

[0007] Furthermore, when the error pattern is classified as occlusion diffusion type, the compensation path is a structural completion path; The step of matching the compensation path corresponding to the error pattern and correcting the initial position of each key point based on the compensation path to obtain the compensation position of each key point includes: Select key points that are located on the same motion chain as the current key point and whose error mode is reliable observation type as support key points; Based on the location of the support key point, the prior value of the bone length of the limb segment between the support key point and the current key point, and the feasible angle range of the joint at the support key point, multiple candidate locations of the current key point are generated; wherein, the prior value of the bone length is the preset average physiological length of the corresponding limb segment, and the feasible angle range is the preset physiological rotation angle range of the joint. For each candidate location, a local image patch of a preset size is extracted centered on the candidate location, and the edge response intensity of the local image patch in a preset edge direction is extracted; wherein, the preset edge direction is perpendicular to the direction from the support key point to the candidate location; Calculate the proportion of the edge response intensity of each candidate location in the sum of the edge response intensities of all candidate locations, and use it as the evidence score for each candidate location; Calculate the score difference between the evidence score of the candidate position with the highest evidence score and the evidence score of the candidate position with the second highest evidence score; When the score difference is greater than a preset interval threshold, the candidate position with the highest evidence score is used as the compensation position of the current key point. When the score difference is not greater than the preset interval threshold, the candidate positions are weighted and summed according to their respective evidence scores, and the resulting position is used as the compensation position of the current key point.

[0008] Furthermore, when the error pattern is classified as a blur-splitting type, the compensation path is a temporal sharpening path; The step of matching the compensation path corresponding to the error pattern and correcting the initial position of each key point based on the compensation path to obtain the compensation position of each key point includes: Select a predetermined number of consecutive historical video frames preceding the current video frame from the video frame sequence; For each historical video frame, the motion field between the current video frame and the historical video frame is calculated by optical flow estimation, and the initial position of the current key point in the historical video frame is determined according to the motion field. Spatial alignment is performed on the initial heatmap of the current key point in each historical video frame according to the offset between the corresponding position and the initial position. For each initial heatmap after spatial alignment, calculate the ratio between the maximum response value of the initial heatmap and the average response value of the pixels in the initial heatmap whose response value exceeds a preset threshold, and normalize the ratio corresponding to each initial heatmap to obtain the superposition weight corresponding to each frame's initial heatmap. The initial heatmaps of each spatially aligned historical video frame and the initial heatmap of the current video frame are superimposed pixel by pixel according to their respective superposition weights to obtain the sharpened heatmap of the current key point. The location with the largest response value in the sharpened heatmap is taken as the compensation location for the key point.

[0009] Furthermore, when the error pattern is classified as instance confusion type, the compensation path is the instance attribution arbitration path; The step of matching the compensation path corresponding to the error pattern and correcting the initial position of each key point based on the compensation path to obtain the compensation position of each key point includes: Human instance detection is performed on the current video frame to obtain the detection box for each human instance; Extract the second peak whose response value in the initial heatmap of the current key point is second only to the main peak, and match the positions of the main peak and the second peak with the detection boxes of each human instance. The number of keypoints with the error mode of reliable observation within the detection box of each human instance is counted, and the human instance with the largest number is taken as the target human instance to which the keypoint belongs. When the position of the main peak falls within the detection frame of the target human body instance, the position of the main peak is used as the compensation position of the current key point; When the position of the main peak does not fall within the detection frame of the target human instance, but the position of the secondary peak falls within the detection frame of the target human instance, the position of the secondary peak is used as the compensation position of the current key point. When neither the position of the main peak nor the position of the secondary peak falls within the detection frame of the target human body instance, the error mode of the current key point is reclassified as occlusion diffusion type, and the compensation path processing flow corresponding to the occlusion diffusion type is executed for the current key point.

[0010] Furthermore, when the error pattern being classified is out-of-bounds truncation, the compensation path is a boundary extrapolation path; The step of matching the compensation path corresponding to the error pattern and correcting the initial position of each key point based on the compensation path to obtain the compensation position of each key point includes: Select the other end of the limb segment to which the current key point belongs as the key point with a reliable observation type error pattern, and use it as the anchor key point; From the video frame sequence, determine the reference video frame in which the current key point was most recently classified as a reliable observation type, and record the limb segment image length between the anchored key point and the current key point in the reference video frame; Based on the orientation of the current key point in the reference video frame relative to the position of the anchor key point in the reference video frame and the motion direction of the anchor key point from the reference video frame to the current video frame, determine the extension direction vector of the limb segment toward the image boundary of the current video frame. Starting from the position of the anchor key point in the current video frame, extrapolate along the extension direction vector by a distance equal to the length of the limb segment image, and use the obtained position as the compensation position of the current key point; The compensation position of the current key point is marked as a low-confidence position; wherein, the key point marked as a low-confidence position is exempted from the consistency check of the consistency between the image length of the limb segment to which the current key point belongs and the prior value of the bone length, wherein the prior value of the bone length is the preset average physiological length of the corresponding limb segment.

[0011] Furthermore, the step of performing a consistency check on the compensation positions of each key point based on the skeletal structural and kinematic relationships between their compensation positions, and determining the convergence weight of each key point based on the consistency check results, includes: Determine the scale of the human body image; wherein the scale of the human body image is determined by the ratio between the image length of the reference limb segment formed by key points classified as reliable observation types in the current video frame and the prior value of the bone length of the reference limb segment, wherein the prior value of the bone length is the preset average physiological length of the corresponding limb segment. For each limb segment, the deviation between the image length between the compensated positions of the key points at both ends of the limb segment and the length of the limb segment after scaling by the human image scale is calculated, and this deviation is taken as the limb segment's bone length deviation. For each key point, the maximum value among the bone length deviations of each limb segment with the key point as the endpoint is taken as the bone length deviation of the key point. For each joint connecting two limb segments, calculate the included angle between the two limb segments at the joint, determined by the compensation position. When the included angle exceeds the feasible angle range of the joint, the excess amount is taken as the joint angle over-limit amount. For each key point, calculate the deviation between the compensated position of the key point and the extrapolated position of the key point's motion, and use it as the temporal motion deviation of the key point; For each key point, the bone length deviation of the key point, the joint angle overrun of the joint associated with the key point, and the temporal motion deviation of the key point are compared with their respective preset verification thresholds, and the number of inspection items that fail the comparison is counted. Calculate the ratio of the number of inspection items to the total number of inspection items, and use it as the failure rate of the verification of the key point; Calculate the product of the complement of the failure rate and the preset benchmark weight, and use the product as the convergence weight of the key point.

[0012] Further, the step of weightedly fusing the compensation positions of each key point with the corresponding motion extrapolation positions according to the convergence weight, and outputting the human pose recognition result of the motion-sensing game console for the current video frame includes: For each key point, calculate the product of the compensation position of the key point and the convergence weight, and use it as the first weighted position; The product of the extrapolated position of the key point and the difference between 1 and the convergence weight is calculated as the second weighted position; Add the first weighted position to the second weighted position to obtain the fused position of the key point; The fusion positions of each key point are connected according to the preset human skeleton connection relationship, and the convergence weight of each key point is output as the output confidence of the corresponding key point to obtain the human posture recognition result of the motion-sensing game machine.

[0013] A second aspect of the present invention provides a human posture recognition device for a motion-sensing game console, comprising: The acquisition unit is used to acquire a video frame sequence of the human body, input the current video frame in the video frame sequence into a pre-trained pose estimation network, obtain the initial heat map of each key point of the human body in the current video frame, and take the position of the main peak with the largest response value in each initial heat map as the initial position of each key point. The extraction unit is used to extract the morphological features of each initial heatmap and match the error patterns corresponding to each key point based on the morphological features; wherein, the error pattern represents the category of key point observation failure due to different causes, and the error pattern includes occlusion diffusion type, blur split type, instance confusion type, out-of-bounds truncation type and reliable observation type. A matching unit is used to match the compensation path corresponding to the error pattern, and correct the initial position of each key point based on the compensation path to obtain the compensation position of each key point; wherein, the compensation path is a processing flow that corrects the initial positioning of key points according to the cause of the corresponding error pattern. The verification unit is used to verify the consistency of the compensation positions of each key point based on the skeletal structure relationship and motion relationship between the compensation positions of each key point, and to determine the convergence weight of each key point based on the consistency verification result. The fusion unit is used to perform weighted fusion of the compensation position of each key point and the corresponding motion extrapolation position according to the convergence weight, and output the human posture recognition result of the motion-sensing game console in the current video frame; wherein, the motion extrapolation position is obtained by extrapolating the historical trajectory of the corresponding key point in the video frame sequence before the current video frame.

[0014] A third aspect of the present invention provides a terminal device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps in the human posture recognition method for the motion-sensing game console described in the first aspect.

[0015] A fourth aspect of the present invention provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps in the human posture recognition method for a motion-sensing game console described in the first aspect.

[0016] The beneficial effects of this invention compared to existing technologies are as follows: By acquiring continuous video frame sequences and inputting the current video frame into a pre-trained pose estimation network, initial heatmaps of each human keypoint are obtained, and the response peak positions of the main peaks of each heatmap are extracted as the initial positions of the keypoints. This provides a unified and quantifiable observation benchmark for subsequent localization correction, overcoming the limitation of existing methods that directly take the network's raw output, resulting in localization being dominated by single-frame noise. By extracting the response distribution morphology features of each initial heatmap, and accordingly subdividing the failure causes of each keypoint into various error modes such as occlusion diffusion, blurring and splitting, instance confusion, out-of-bounds truncation, and reliable observation, a refined attribution of the failure causes of keypoint observation is achieved. This overcomes the limitation of traditional methods that treat all failures the same and cannot distinguish failure types, providing a classification basis for differentiated processing. By matching targeted compensation paths to the causes of each error pattern and correcting the initial position, structural completion in occluded scenes, temporal sharpening in blurred scenes, attribution arbitration in multi-instance scenes, and boundary extrapolation in out-of-bounds scenes can each leverage their strengths. This overcomes the shortcomings of single compensation methods that cannot cover multiple failure causes and the lag in compensation intervention, directly reducing the positioning error of failure key points at the causal level. By verifying the consistency of the compensation position based on skeletal structure and motion relationships and determining the convergence weight of each key point accordingly, physiological structural constraints and motion continuity constraints jointly participate in the assessment of position reliability. This overcomes the shortcomings of existing methods that indiscriminately fuse observations and predictions and lack the ability to distinguish abnormal positioning, suppressing the impact of structural conflicts and motion jumps on the final result. By weighting and fusing the compensated positions of key points with the motion extrapolation positions obtained from historical trajectory extrapolation according to convergence weights, static observation information and dynamic motion information are combined in a complementary manner according to credibility. This avoids abrupt changes in positioning caused by constant weights or hard switching, and stably connects the positioning results of each key point in complex scenarios such as occlusion, blurring, intersection, and boundaries. Ultimately, this improves the accuracy of human posture recognition, the continuity of motion tracking, and the real-time performance of game control in complex indoor environments for motion-sensing game consoles. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 A schematic flowchart of a human posture recognition method for a motion-sensing game console provided by the present invention is shown; Figure 2 This diagram illustrates a human posture recognition device for a motion-sensing game console according to an embodiment of the present invention. Figure 3 A schematic diagram of a terminal device according to an embodiment of the present invention is shown; Figure 4 This invention provides a flowchart illustrating a specific implementation of step 102 in a human posture recognition method for a motion-sensing game console. Figure 5 The flowchart illustrates a specific implementation of step 103 in a human posture recognition method for a motion-sensing game console provided by the present invention. Figure 6 The flowchart illustrates a specific implementation of step 103 in a human posture recognition method for a motion-sensing game console provided by the present invention. Figure 7 The flowchart illustrates a specific implementation of step 103 in a human posture recognition method for a motion-sensing game console provided by the present invention. Figure 8 The flowchart illustrates a specific implementation of step 103 in a human posture recognition method for a motion-sensing game console provided by the present invention. Figure 9 The flowchart illustrates a specific implementation of step 104 in a human posture recognition method for a motion-sensing game console provided by the present invention. Figure 10 The flowchart illustrates a specific implementation of step 105 in a motion-sensing game console human posture recognition method provided by the present invention. Detailed Implementation

[0019] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of the invention. However, those skilled in the art will understand that the invention can be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods are omitted so as not to obscure the description of the invention with unnecessary detail.

[0020] This invention provides a method and apparatus for human posture recognition in a motion-sensing game console, which solves the technical problem in the prior art that key point localization usually only uses the peak value of a single frame heat map response as the judgment standard, which is prone to main peak misselection and positioning drift in dynamic scenes with occlusion, motion blur, multiple people intermingling and screen boundaries.

[0021] This embodiment provides a human posture recognition method for a motion-sensing game console. The motion-sensing game console continuously captures images of the user's body area at a certain frame rate using a camera, forming a continuous video frame sequence. The five main steps of this method are described in detail below with reference to the accompanying drawings.

[0022] Step 101: Collect a video frame sequence of the human body, input the current video frame in the video frame sequence into a pre-trained pose estimation network to obtain the initial heat map of each key point of the human body in the current video frame, and take the position of the main peak with the largest response value in each initial heat map as the initial position of each key point. The pose estimation network is a pre-trained deep convolutional neural network. The input is a single frame of a human image, and the output is a response heatmap of each key point on the human body. The deep convolutional neural network is existing technology and will not be elaborated further here. A response heatmap is a two-dimensional response map with the same resolution as the input image or after downsampling. The response value of each pixel in the heatmap reflects the probability of that key point appearing at that pixel location. This embodiment focuses on more than ten major joints and endpoints of the human body, including the head, neck, shoulders, elbows, wrists, hips, knees, and ankles. Each key point corresponds to an independent response heatmap.

[0023] Specifically, the pose estimation network is trained offline using a large number of video images with human keypoint annotations. Its main body employs a deep residual network as the backbone to extract image features, which are then progressively restored through several deconvolutional layers. Finally, it outputs a response heatmap for each keypoint on the coordinate axes. In actual deployment, the pose estimation network can adopt a lightweight design, such as replacing standard convolutions with depthwise separable convolutions and pruning and compressing feature channels, enabling single-frame inference to be completed in real-time on the embedded chip of a motion-sensing game console. Because the network applies Gaussian kernel smoothing to the response heatmap during training, even if there are sub-pixel offsets in the true location of the keypoints, the main peak of the heatmap can still fall relatively accurately near the true location.

[0024] For the current video frame, after the pose estimation network outputs initial heatmaps for each keypoint, this step takes the pixel with the largest response value in each initial heatmap as the response peak of that keypoint and uses the location of that peak as the initial position of that keypoint. If sub-pixel accuracy needs to be improved, a quadratic or Gaussian surface can be fitted onto the response distribution near the peak, and the extreme points of the surface can be used as more precise initial positions. This step outputs a set of initial positions corresponding one-to-one with each keypoint of the human body, which serve as input for error determination and position correction in subsequent steps.

[0025] As an optional embodiment of this application, the pose estimation network can also adopt a top-down estimation structure, that is, first use the detection network to circle the entire human target area, and then estimate the key points of the circled area separately; this structure has higher positioning accuracy in multi-person scenarios, but the computational load is larger than that of the bottom-up structure.

[0026] Step 102: Extract the morphological features of each initial heatmap, and match the error patterns corresponding to each key point based on the morphological features; The reliability of localization cannot be determined solely by the initial position, as the output of the attitude estimation network will exhibit different response patterns under different failure scenarios. This step extracts morphological features from the response distribution of each initial heatmap, and categorizes each keypoint into one of five error modes: occlusion diffusion, blurring and splitting, instance confusion, out-of-bounds truncation, and reliable observation. This provides a basis for subsequent selection of compensation paths based on their causes.

[0027] Error modes characterize the categories of keypoint observation failures due to different causes. Among them, occlusion diffusion refers to keypoints being covered by other body parts or occlusions, causing the thermal response to spread out over a large area. Blurring splitting refers to motion blur dulling the response peak, preventing the formation of a sharp main peak. Instance confusion refers to the main peak of the heatmap incorrectly falling on another person's body when multiple people are interspersed. Out-of-bounds truncation refers to keypoints moving to the edge of the image, causing the response to be truncated by the image boundary. Reliable observation refers to a clear and reliable main peak that requires no additional compensation.

[0028] Specifically, step 102 includes steps B1 to B10: Figure 4 The following is a flowchart illustrating a specific implementation of step 102 in a motion-sensing game console human posture recognition method provided by the present invention: Step B1: For each initial heatmap, calculate the ratio between the peak value of the main peak in the initial heatmap and the average response value of the set of pixels in the initial heatmap whose response values ​​exceed a preset response threshold, and use this ratio as the main peak sharpness. Peak sharpness is used to quantify the sharpness of a keypoint's response. For an initial heatmap, first determine its peak value Rpeak, then count the pixels in the image whose response values ​​exceed a preset response threshold T. The set of these pixels is denoted as the support region S, and the mean response value Rmean of all pixels within the support region S is calculated. Peak sharpness is defined as Sh = Rpeak / Rmean. This ratio characterizes the strength of the peak's prominence relative to its surrounding support region. The response threshold T can be a certain proportion of the maximum response value; for example, T is 0.1 times the peak value, used to delineate the significant response region connected to the peak. When a keypoint is occluded or blurred, the response diffuses over a large area, the peak is no longer significantly higher than its surroundings, and the peak sharpness decreases accordingly.

[0029] Step B2: Extract the second peak whose response value is second only to the main peak in the initial heatmap, and calculate the ratio between the peak value of the second peak and the peak value of the main peak as the competition degree of the second peak; Secondary peak competition is used to quantify the existence of a second significant response peak that can compete with the primary peak. Connectivity analysis is performed on the initial heatmap. Excluding the connected component containing the primary peak, the connected component with the second highest response peak is identified as the secondary peak, with its peak value denoted as Rsub. Secondary peak competition is defined as C = Rsub / Rpeak. When two or more similar instances appear interspersed in the image, the heatmap often shows response peaks with similar peak values ​​to the primary peak. In this case, the secondary peak competition is close to 1, indicating that the attribution of the primary peak may be questionable. If the heatmap contains only a single clear response peak, the secondary peak value is very low or even nonexistent, and the secondary peak competition is close to 0.

[0030] Step B3: Count the number of pixels contained in the pixel set, and use it as the area of ​​the supporting domain; The support domain area characterizes the spatial extent of the response diffusion, specifically the total number of pixels N_S contained in the set S of pixels whose response values ​​exceed a preset response threshold, as defined in the previous step. This area reflects the size of the range of the keypoint response spread across the image. When keypoints are largely occluded or severely blurred, the heatmap response often expands into a large diffuse area, significantly increasing the support domain area; conversely, when keypoints are clearly visible, the support domain shrinks into a compact small region surrounding the main peak, resulting in a smaller area.

[0031] Step B4: Count the number of pixels in the pixel set that are adjacent to the image boundary of the current video frame, and calculate the ratio between the number of adjacent pixels and the area of ​​the support domain as the edge truncation ratio; The edge truncation ratio is used to determine whether a keypoint response is truncated due to proximity to image boundaries. It is calculated by counting the number of pixels (N_edge) in the support region S that meet the four image boundaries of the current video frame. The edge truncation ratio is defined as E = N_edge / N_S. When a keypoint moves to the edge of the frame, its response half-region extends beyond the image and is partially cut off by the boundary. The main body of the support region then meets the image boundary, and the edge truncation ratio increases accordingly. This metric can detect signs of truncation even before the keypoint has completely left the frame.

[0032] Step B5: When the competition degree of the secondary peak is higher than the preset competition threshold, and the distance between the secondary peak and the primary peak is greater than the scale prior value of the limb segment where the key point is located, the error mode of the key point is classified as instance confusion type; wherein, the scale prior value represents the estimated length of the limb segment where the key point is located in the image. A secondary peak competition level exceeding the competition threshold indicates the existence of another candidate peak with a similar intensity to the primary peak. To distinguish between bimodal peaks caused by multiple overlapping peaks and noise-induced clutter, this step further examines the distance between the primary and secondary peaks. If the distance between the two peaks is greater than the scale prior value L_seg of the limb segment containing the keypoint, it indicates that the bimodal peaks belong to two independent and geographically distant body structures, more likely to be keypoints with the same name from two different instances, thus classifying it as an instance confusion type. The scale prior value characterizes the estimated length of the limb segment containing the keypoint in the image, which can be inferred from the projection distance between the joint and adjacent joints on two instances. For example, the competition threshold is set to 0.6, meaning that a strong competition peak is considered to exist when the secondary peak reaches 60% of the primary peak.

[0033] Step B6: When the area of ​​the support domain is greater than the preset diffusion threshold and the sharpness of the main peak is lower than the preset sharpness threshold, the error mode of the key point is classified as occlusion diffusion type. A support region area exceeding the diffusion threshold indicates that the response is spread out over a large spatial area. A peak sharpness below the sharpness threshold indicates that the peak is no longer prominent relative to its surroundings. When both conditions are met, the response is diffuse and spread out without a clear, sharp peak. This is a typical form of keypoint occlusion, hence it is classified as an occluded diffusion type. For example, the diffusion threshold is taken as a reference value several times the support region area when the peak is clearly visible, and the sharpness threshold is taken as 1.5, meaning that when the peak value is less than 1.5 times the average response value of the support region, the peak is considered insufficiently sharp.

[0034] Step B7: When the edge truncation ratio is greater than the preset truncation threshold, the error mode of the key point is classified as out-of-bounds truncation type; If the edge truncation ratio exceeds the truncation threshold, it indicates that the main body of the keypoint response is connected to the image boundary and is clearly truncated by the image edge, meaning the keypoint is at the edge of the frame. This is classified as an out-of-bounds truncation type. For example, a truncation threshold of 0.3 means that when more than 30% of the responses in the support domain are connected to the image boundary, the keypoint is considered to have entered the image edge region. The priority for determining out-of-bounds truncation types is after occlusion / diffusion types and before blur / split types.

[0035] Step B8: When the sharpness of the main peak is lower than the preset sharpness threshold and the area of ​​the support domain is not greater than the preset diffusion threshold, the error mode of the key point is classified as fuzzy splitting type. Low peak sharpness indicates that the peak is not prominent enough, but the supporting domain area does not significantly expand, suggesting that the response is not diffused over a large area, but rather the peak is blunted into a flat, broad peak. This is precisely the form of peak splitting caused by motion blur. This is classified as blur splitting type. Motion blur often occurs when keypoints move rapidly, with the keypoint displacement spanning several pixels within the exposure time. The response peak is stretched and smoothed out, exhibiting characteristics of decreased sharpness but little change in area.

[0036] Step B9: When the initial heatmap of the key point does not meet the classification conditions of instance confusion type, occlusion diffusion type, out-of-bounds truncation type and fuzzy split type, the error mode of the key point is classified as reliable observation type; When the initial heatmap does not meet the classification criteria for the first four error modes, it indicates that the main peak of the key point is clear, sharp, and unaffected by environmental interference, resulting in the highest quality reliability. In this case, it is classified as a reliable observation type. The compensation position for the reliable observation type directly adopts its initial position and serves as a reference benchmark for structural constraints in subsequent consistency checks. This is the key point category with the highest reliability in the entire processing chain.

[0037] Step B10: When the initial heatmap of the key point simultaneously meets the classification conditions of multiple error modes, the key point is classified into the error mode with the highest priority according to the preset priority order.

[0038] Since the classification criteria for each error mode are composed of different morphological indicators, multiple conditions may be met simultaneously in certain degradation scenarios, such as a keypoint being both occluded and close to the image boundary. In such cases, arbitration is performed according to a preset priority order. For example, the priority order is instance confusion type, occlusion diffusion type, out-of-bounds truncation type, blurry split type, and reliable observation type, that is, priority is given to classification based on the cause that causes greater damage to localization. Through this arbitration rule, each keypoint is ultimately determined to have one and only one error mode, avoiding duplicate processing or omissions.

[0039] The five error modes, their main decision conditions, and corresponding compensation paths used in this step can be summarized as follows for easy reference and configuration during engineering implementation. For example, the correspondence between error mode classification and compensation path is shown in Table 1 below: Table 1. Correspondence between error mode classification conditions and compensation paths In the embodiments corresponding to steps B1 to B10, through the cascaded processing of the above steps, this method transforms each initial heatmap output by the attitude estimation network into a precise localization and judgment of the failure cause of key points. Specifically, it discretizes the continuous heatmap grayscale distribution into five mutually exclusive error modes: occlusion diffusion, blurring and splitting, instance confusion, out-of-bounds truncation, and reliable observation. Four morphological features—primary peak sharpness, secondary peak competition, support domain area, and edge truncation ratio—characterize the heatmap morphology from four dimensions: sharpness, competition intensity, diffusion range, and boundary truncation, respectively. Combined with numbering and priority arbitration, the failure cause of each key point is uniquely and reasonably determined, providing a reliable classification basis for subsequent selection of compensation paths based on causes.

[0040] Step 103: Match the compensation path corresponding to the error pattern, and correct the initial position of each key point based on the compensation path to obtain the compensation position of each key point; After determining the error pattern of each keypoint, this step matches the corresponding compensation path according to the error pattern, and then corrects the initial position along the path to obtain the compensated position. The compensation path is a process that corrects the initial positioning of the keypoint based on the cause of the corresponding error pattern. This method designs four compensation paths for the first four error patterns: occlusion / diffuse type (corresponding to structure completion path), fuzzy / split type (corresponding to temporal sharpening path), instance confusion type (corresponding to instance attribution arbitration path), and out-of-bounds truncation type (corresponding to boundary extrapolation path). Reliable observation type requires no compensation; its compensation position is the initial position. The four compensation paths are explained in detail below.

[0041] Specifically, when the error pattern is classified as occlusion diffusion, the compensation path is a structural completion path, and step 103 specifically includes steps C1 to C7: Figure 5 The following is a flowchart illustrating a specific implementation of step 103 in a motion-sensing game console human posture recognition method provided by the present invention: Step C1: Select key points that are located on the same motion chain as the current key point and whose error mode is reliable observation type, as support key points; Structural completion paths are used to handle occluded and diffuse keypoints. The idea is to leverage the strong prior knowledge of the human skeletal structure to recover the possible locations of occluded keypoints. First, on the current skeletal motion chain, keypoints adjacent to the current keypoint and whose error patterns are determined to be reliable observation types are selected as support keypoints. A motion chain refers to a series of keypoints connected by skeletal relationships, such as the neck, shoulders, elbows, and wrists, which sequentially form the upper limb motion chain. Support keypoints, due to their high reliability, become reliable anchor points for inferring the location of occluded keypoints. For example, when the right elbow is occluded, the right shoulder or right wrist, which is located on the same upper limb motion chain and is clearly visible, can be selected as support keypoints.

[0042] Step C2: Based on the position of the support key point, the prior value of the bone length of the limb segment between the support key point and the current key point, and the feasible angle range of the joint at the support key point, generate multiple candidate positions of the current key point; wherein, the prior value of the bone length is the preset average physiological length of the corresponding limb segment, and the feasible angle range is the preset physiological rotation angle range of the joint. Based on the known location of the support key point, this step generates multiple candidate locations for the current key point using prior values ​​of limb segment bone length and feasible angle ranges of joints. The prior value of bone length, L_bone, is a preset average physiological length of the corresponding limb segment; for example, the prior value of forearm bone length can be the typical length normalized to the average height of the human body. The feasible angle range is a preset range of physiologically rotatable angles of the joint. Using the support key point as the center and the prior value of bone length as the radius, sampling is performed on this circle according to various orientation angles within the feasible angle range to obtain multiple candidate locations distributed around the circumference. For example, sampling can be performed within the feasible angle range at fixed angular steps, such as taking one candidate every 10 degrees, resulting in several evenly distributed candidate locations, allowing subsequent evidence evaluation to cover the orientation of the true location.

[0043] Step C3: For each candidate location, extract a local image patch of a preset size centered on the candidate location, and extract the edge response intensity of the local image patch in a preset edge direction; wherein, the preset edge direction is perpendicular to the direction from the support key point to the candidate location; To identify the candidate that best matches the human body appearance from multiple candidate locations, this step uses image edge information for scoring. For each candidate location, a local image patch of a preset size, such as a 16x16 pixel square, is extracted centered on that location, and the edge response intensity of this image patch in a preset edge direction is extracted. This preset edge direction is perpendicular to the direction from the supporting keypoints pointing to the candidate location. This direction is chosen because a limb appears as a strip of a certain width in the image, with its two contour lines parallel to the limb axis and its edge direction approximately perpendicular to the limb axis. When the candidate location falls exactly on the actual limb strip, the edge response intensity perpendicular to this direction is the highest, thus distinguishing the correct candidate from the incorrect one. The edge response intensity can be obtained by integrating along the specified direction using directional gradient filtering or Canny edge detection.

[0044] Step C4: Calculate the proportion of the edge response intensity of each candidate location in the sum of the edge response intensities of all candidate locations, and use it as the evidence score for each candidate location; The ratio of the edge response intensity of each candidate location to the sum of the edge response intensities of all candidate locations is used as the evidence score for that candidate location. The evidence score reflects the edge evidence weight that the candidate location occupies among all candidates; the higher the score, the more closely the candidate conforms to the boundary appearance of a human limb. For N candidate locations, let the edge response intensity of the i-th candidate be e_i, then its evidence score is s_i = e_i / Σe_j. The evidence score, after normalization, is between 0 and 1, facilitating subsequent threshold comparison and weighted fusion.

[0045] Step C5: Calculate the score difference between the evidence score of the candidate position with the highest evidence score and the evidence score of the candidate position with the second highest evidence score; After obtaining the evidence scores for each candidate, this step extracts the candidate with the highest score and the candidate with the second highest score, and calculates the difference between their evidence scores, obtaining the score difference Δ=s_max-s_2nd. This difference is used to measure the degree of leadership of the best candidate relative to the second best candidate. If the highest evidence score is significantly higher than the second highest, it indicates that the structural prior and image evidence strongly point to the same location, and the confidence in that location is high; if the two scores are close, it indicates that there is considerable marginal evidence in multiple directions, and the true location is uncertain, so it is not advisable to arbitrarily select one.

[0046] Step C6: When the score difference is greater than the preset interval threshold, the candidate position with the highest evidence score is used as the compensation position of the current key point; When the score difference exceeds a preset interval threshold, it indicates that the best candidate wins by a significant margin, and the candidate position with the highest evidence score is directly taken as the compensation position for the current key point. For example, the interval threshold ranges from 0.2 to 0.3, and can be 0.25, meaning that when the evidence score of the best candidate is more than 25% higher than the second-best candidate, its position is considered credible. At this point, the structural prior and image edge evidence corroborate each other, and the position of the occluded key point is accurately restored under physiological constraints.

[0047] Step C7: When the score difference is not greater than the preset interval threshold, the candidate positions are weighted and summed according to their respective evidence scores, and the obtained position is used as the compensation position of the current key point.

[0048] When the score difference is no greater than the interval threshold, the evidence for the best and second-best candidates is comparable, and choosing only one would be too arbitrary. In this case, all candidate positions are weighted and summed according to their respective evidence scores, i.e., the compensation position P_comp = Σs_i·P_i, where s_i is the evidence score of the i-th candidate and P_i is its position. Since the evidence scores are normalized and each candidate covers a feasible angle range, the compensation position obtained by the weighted summation actually falls at the weighted centroid of the evidence-dominant direction. This preserves the structural prior constraints while incorporating marginal evidence from multiple directions, providing a reasonable compensation position even when evidence is scattered.

[0049] In the embodiments corresponding to steps C1 to C7, through the cascaded processing of the above steps, the positions of occluded and diffuse keypoints are reconstructed within the physiological structural framework. Supporting keypoints provide credible anchor points, the prior value of bone length and the feasible angle range of joints jointly define the spatial distribution of candidate positions, the vertical edge response intensity scores the candidates as evidence, and the combination of score difference judgment and evidence weighted fusion makes the compensation result sharply locked when the evidence is clear and robustly compromised when the evidence is scattered, thereby restoring the originally diffuse and blurry positioning in the occluded scene to a structurally reasonable and aesthetically consistent compensation position.

[0050] Specifically, when the error pattern is classified as blur-splitting, the compensation path is a temporal sharpening path, and step 103 specifically includes steps C8 to C13: Figure 6 The flowchart illustrates a specific implementation of step 103 in a human posture recognition method for a motion-sensing game console provided by the present invention. Step C8: Select a predetermined number of consecutive historical video frames from the video frame sequence that precede the current video frame; The temporal sharpening path is used to handle blurred, split keypoints. Its idea is to use temporal information to compensate for peak blurring caused by motion blur in a single frame. This step first selects a predetermined number of consecutive historical video frames preceding the current video frame from the video frame sequence. For example, four historical frames are selected, meaning the four most recent historical frames preceding the current frame participate in subsequent processing along with the current frame, covering approximately a conservative temporal window. This ensures that temporary noise is not amplified by a window that is too narrow, nor is excessive pose variation introduced by a window that is too wide.

[0051] Step C9: For each historical video frame, calculate the motion field between the current video frame and the historical video frame through optical flow estimation, and determine the corresponding position of the initial position of the current key point in the historical video frame based on the motion field; To align the heatmaps of historical frames with the current frame, this step calculates the motion field between the current video frame and each historical video frame using optical flow estimation. Optical flow characterizes the displacement vector field of each pixel between two frames. Based on this motion field, the initial position of the current keypoint is back-mapped to its corresponding position in each historical video frame. For example, optical flow estimation can employ dense optical flow algorithms, such as iterative optical flow algorithms based on a pyramid structure, whose accuracy is sufficient to support sub-pixel-level alignment. Through the mapping of the motion field, even if the human body undergoes significant displacement between frames, the positions of keypoints with the same name as the current keypoint in historical frames can be accurately located.

[0052] Step C10: Spatially align the initial heatmap of the current key point in each historical video frame according to the offset between the corresponding position and the initial position; After obtaining the corresponding position of the current keypoint in each historical video frame, this step calculates the offset between that position and the initial position of the current frame. The initial heatmap of the current keypoint in each historical video frame is then spatially shifted according to this offset, aligning the main peak of each historical frame heatmap with the main peak of the current frame heatmap. Spatial alignment is achieved through a global shift of the heatmaps. After alignment, the same physical keypoint falls at the same coordinate position in all heatmaps, creating conditions for subsequent pixel-by-pixel overlay. For minor deviations caused by scale changes in the historical frame heatmaps, small-scale scaling corrections can be applied to the shift.

[0053] Step C11: For each initial heatmap after spatial alignment, calculate the ratio between the maximum response value of the initial heatmap and the average response value of the pixels in the initial heatmap whose response value exceeds a preset threshold, and normalize the ratio corresponding to each initial heatmap to obtain the superposition weight corresponding to each frame's initial heatmap. After spatial alignment, each initial heatmap represents observations of the same keypoint at different times. This step calculates the ratio between the maximum response value of each initial heatmap and the average pixel response value exceeding a preset threshold. This ratio is positively correlated with the peak sharpness of the heatmap frame, reflecting the confidence level of the keypoint in that frame. The ratios corresponding to each heatmap frame are normalized, summing to 1, thus obtaining the stacking weights for each initial heatmap frame. Sharp frames have higher sharpness ratios and receive higher weights, while blurry frames have lower sharpness ratios and receive lower weights, allowing sharper observations to dominate during stacking.

[0054] Step C12: The initial heatmaps of each spatially aligned historical video frame and the initial heatmap of the current video frame are superimposed pixel by pixel according to their respective superposition weights to obtain the sharpened heatmap of the current key point. The initial heatmaps of each historical video frame and the initial heatmap of the current video frame are weighted and summed pixel-by-pixel according to their respective superposition weights to obtain the sharpened heatmap of the current key point. Since the true peaks of the heatmaps of each frame are located at the same position after spatial alignment, the weighted superposition will cumulatively enhance the response at that position; while the diffusion components caused by blur are distributed in different positions in each frame, and are relatively thinned after superposition. Therefore, compared with the heatmap of the current frame alone, the main peaks of the sharpened heatmap are sharper and more prominent, and the peaks that were originally blunted and fragmented due to motion blur are effectively sharpened.

[0055] Step C13: Take the position with the largest response value in the sharpened heatmap as the compensation position of the key point.

[0056] Finally, the position with the largest response value in the sharpened heatmap is taken as the compensation position for the current key point. Since the main peak of the sharpened heatmap has been enhanced and locked to the real physical location during temporal overlay, its peak position is more accurate and stable than the initial position of a single blurred frame, thus restoring the localization of blurred split key points to a level close to that of a clear frame.

[0057] In the embodiments corresponding to steps C8 to C13, the localization of blurred split keypoints is compensated in the temporal dimension through the cascaded processing of the above steps. The optical flow motion field completes inter-frame mapping and spatial alignment, and the sharpness of the main peak in each frame is normalized into a superposition weight. The weighted superposition makes the real peak cumulatively enhanced across multiple frames and the diffuse component relatively thinned. Finally, a sharper and more stable compensation position is extracted from the sharpened heatmap, transforming the peak blunting problem introduced by motion blur into the complementary advantage of multi-frame evidence.

[0058] Specifically, when the error pattern is classified as instance confusion, the compensation path is the instance attribution arbitration path, and step 103 specifically includes steps C14 to C19: Figure 7 The flowchart illustrates a specific implementation of step 103 in a human posture recognition method for a motion-sensing game console provided by the present invention. Step C14: Perform human instance detection on the current video frame to obtain the detection box for each human instance; The instance attribution arbitration path is used to handle instance-confusing keypoints. Its idea is to determine which human instance in the image the dominant peak of the current keypoint's heatmap belongs to. When multiple people are interspersed, the heatmap of the same keypoint may show two response peaks falling on different people, and their attribution must be determined. This step first performs human instance detection on the current video frame to obtain the detection bounding boxes for each human instance in the image. For example, human instance detection can use an object detection network, such as a combination of an anchor-box-based detection structure and a deep feature extraction backbone. After offline training, it outputs the bounding box and its category confidence for each human instance. The detection boxes are represented by the top-left and bottom-right vertices in image coordinates, defining the area occupied by each human instance in the image.

[0059] Step C15: Extract the secondary peak whose response value in the initial heatmap of the current key point is second only to the main peak, and match the positions of the main peak and the secondary peak with the detection boxes of each human instance respectively; Extract the secondary peak whose response value is second only to the primary peak from the initial heatmap of the current key point, resulting in two candidate response peaks: the primary peak position and the secondary peak position. Match the positions of these two peaks with the detection boxes of each human instance obtained in step C14, i.e., determine whether each peak falls within a certain detection box. Position matching is determined by coordinate inclusion relationships: if the horizontal and vertical coordinates of the peak position both fall within the vertex range of the detection box, then the peak is considered to match that detection box. Since the primary and secondary peaks fall on different human bodies when multiple people are interspersed, this matching result provides a spatial basis for subsequent determination of the primary peak's attribution.

[0060] Step C16: Count the number of keypoints with reliable observation error mode within the detection box of each human instance, and take the human instance with the most keypoints as the target human instance to which the keypoints belong. To determine which human instance the current keypoint belongs to, this step counts the number of keypoints with a reliable observation error mode within the detection box of each human instance. Reliable observation keypoints are chosen because their location is credible, their attribution is clear, and they reliably characterize the identity and scope of the human instance to which they belong. The human instance with the most reliable observation keypoints is identified as the target human instance to which the current keypoint belongs; that is, the human instance to which the current keypoint logically belongs is the one with the more complete pose chain and the more sufficient evidence in the image.

[0061] Step C17: When the position of the main peak falls within the detection frame of the target human body instance, the position of the main peak is used as the compensation position of the current key point; When the main peak falls within the detection frame of the target human instance, it indicates that the main peak matches the attribution determination. The main peak represents the correct position of the current keypoint on that instance, and its position is directly taken as the compensation position of the current keypoint. In this case, the main peak already falls on the correct human, and its attribution can be confirmed through attribution arbitration without any issues.

[0062] Step C18: When the position of the main peak does not fall within the detection frame of the target human instance, but the position of the secondary peak falls within the detection frame of the target human instance, the position of the secondary peak is used as the compensation position of the current key point. When the primary peak does not fall within the detection bounding box of the target human instance, but the secondary peak does, it indicates that the primary peak has fallen on the wrong human instance, while the secondary peak corresponds to the same-named keypoint on the target instance. This is because when multiple people are interspersed, the primary peak in the heatmap is attracted by the high response of other people's bodies, and the actual target instance being identified is the secondary peak. In this case, the position of the secondary peak is taken as the compensation position of the current keypoint, thereby correcting the incorrect assignment of the primary peak.

[0063] Step C19: When the positions of the main peak and the secondary peak do not fall within the detection frame of the target human body instance, the error mode of the current key point is reclassified as occlusion diffusion type, and the compensation path processing flow corresponding to the occlusion diffusion type is executed for the current key point.

[0064] When neither the primary peak nor the secondary peak falls within the detection bounding box of the target human instance, it indicates that neither candidate peak corresponds to the key point on the target instance, and the current key point is actually occluded on the target instance with no reliable observation peak. At this point, the error mode of the current key point is reclassified as occlusion diffusion type, and the process switches to the structural completion path processing flow described in steps C1 to C7. The compensation position of the occluded key point is reconstructed using the prior knowledge of the human structural skeleton. Through this fallback mechanism, the instance-confusion type compensation path can smoothly degrade to the structural completion path when evidence is missing, ensuring that all failed key points have corresponding compensation methods.

[0065] In the embodiments corresponding to steps C14 to C19, the cascaded processing of the above steps clarifies the ambiguity of instance attribution in multi-person overlapping scenarios. Human instance detection delineates the spatial range of each instance, the matching of detection boxes for the main peak and secondary peaks indicates the spatial attribution of candidate peaks, the statistical analysis of the number of reliably observed keypoints determines the target instance, and the three-element branch for determining the fall of the main and secondary peaks corresponds to three treatments: normal, corrected, and downgraded. As a result, the keypoints of overlapping human bodies are correctly bound to their respective instances, the incorrect attribution of the main peak is corrected, and in the event of complete absence, a seamless fallback to the structural completion path is possible.

[0066] Specifically, when the error pattern is classified as out-of-bounds truncation, the compensation path is the boundary extrapolation path, and step 103 specifically includes steps C20 to C24: Figure 8 The flowchart illustrates a specific implementation of step 103 in a human posture recognition method for a motion-sensing game console provided by the present invention. Step C20: Select the other end of the limb segment to which the current keypoint belongs as a keypoint with a reliable observational error pattern, and use it as the anchor keypoint; Boundary extrapolation paths are used to handle out-of-bounds truncated keypoints. The idea is that when a keypoint is about to reach or has already reached the edge of the frame, its position is extrapolated beyond the frame boundary based on the length and orientation of its corresponding limb segment before it goes out of bounds, maintaining the continuity of the pose trajectory. This step first selects a keypoint with a reliable observation error mode at the other end of the limb segment to which the current keypoint belongs, as the anchor keypoint. For example, if the current keypoint is the right wrist and its corresponding limb segment is the forearm, the clearly visible right elbow is selected as the anchor keypoint. Anchor keypoints are located within the frame and have reliable positioning, providing a stable reference endpoint for subsequent extrapolation.

[0067] Step C21: Determine the reference video frame from the video frame sequence in which the current key point was most recently classified as a reliable observation type, and record the limb segment image length between the anchored key point and the current key point in the reference video frame; The most recently classified historical video frame as a reliable observation type for the current keypoint is identified as the reference video frame. In the reference video frame, the current keypoint is still within its bounds and its location is reliable. The length of the limb segment between the current keypoint and the current keypoint in the image is then determined, i.e., the limb segment image length L_img. This image length is recorded as the basis for extrapolation. Since the human body does not change much in scale over a short period, the length recorded in the reference frame can approximately represent the actual image length that the limb segment should have in the current frame.

[0068] Step C22: Based on the orientation of the current key point in the reference video frame relative to the position of the anchor key point in the reference video frame and the motion direction of the anchor key point from the reference video frame to the current video frame, determine the extension direction vector of the limb segment toward the image boundary of the current video frame. The direction of limb segment extrapolation is determined. This direction is obtained by combining two pieces of information: first, the orientation of the current keypoint relative to the anchor keypoint in the reference video frame, i.e., the original direction the limb segment was pointing; second, the direction of motion of the anchor keypoint from the reference video frame to the current video frame, reflecting the overall displacement trend of the human body. By combining the original orientation of the limb segment with the movement direction of the anchor keypoint, the extension direction vector of the limb segment currently located at the keypoint, pointing towards the edge of the screen, can be determined. For example, the two direction vectors can be added with certain weights and then normalized. The weight values ​​can be 0.6 to 0.4, which preserves the inherent orientation of the limb while also taking into account the traction of human body displacement.

[0069] Step C23: Starting from the position of the anchor key point in the current video frame, extrapolate along the extension direction vector by a distance equal to the length of the limb segment image, and use the obtained position as the compensation position of the current key point; Starting from the position of the anchor keypoint in the current video frame, the compensation position of the current keypoint is determined by extrapolating along the extension direction vector to a position equidistant from the length L_img of the limb segment image. This extrapolated position is outside the frame boundary and maintains the same limb segment length as the anchor keypoint in the reference frame, thus preserving the geometric continuity of the limb chain in three-dimensional space even after the keypoint goes out of bounds. Because the extension direction integrates the inherent orientation and movement trend of the limb, the extrapolated compensation position can closely follow the actual movement trajectory of the human body when it leaves the frame.

[0070] Step C24: Mark the compensation position of the current key point as a low-confidence position; wherein, the key point marked as a low-confidence position is exempted from the consistency check of the consistency between the image length of the limb segment to which the current key point belongs and the prior value of the bone length in the consistency check, wherein the prior value of the bone length is the preset average physiological length of the corresponding limb segment.

[0071] The extrapolated compensation positions are marked as low-confidence positions to distinguish them from keypoints actually observed within the frame. A low-confidence position is one obtained by extrapolation based on historical information and lacks direct image evidence from the current frame; its reliability is lower than that of keypoints observed within the frame. In the subsequent consistency check in step 104, keypoints marked as low-confidence positions are exempt from the consistency check between the prior value of the image length of their limb segment and the bone length. This is because the limb segment has partially exceeded the frame, and its image length cannot be reliably measured; forced verification would introduce false structural conflicts. Through this exemption, the extrapolated positions of out-of-bounds keypoints will not be misjudged as structural anomalies due to normal out-of-frame phenomena.

[0072] In the embodiments corresponding to steps C20 to C24, through the cascaded processing of the above steps, the location of the out-of-bounds truncated keypoints is smoothly extrapolated beyond the image boundary. Anchored keypoints provide reference endpoints within the image, the length of the limb segment image recorded in the reference frame provides the extrapolation distance, and the inherent orientation of the limb and the direction of human movement combine to form the extension direction. Extrapolating an equal distance along this direction yields a compensated position, which is marked as a low-confidence position to exempt it from subsequent structural consistency checks. The result is that the trajectory of the keypoint no longer breaks when it leaves the frame, but extends continuously beyond the image in a manner consistent with limb geometry.

[0073] In the embodiments corresponding to steps C1 to C24, through the cascaded processing of the above steps, differentiated compensation paths for the four failure causes are fully implemented. The structural completion path reconstructs occluded key points using skeletal priors, the temporal sharpening path sharpens blurred key points using multi-frame overlay, the instance attribution arbitration path corrects interleaved attributions using instance detection, and the boundary extrapolation path continues out-of-bounds trajectories using geometric extrapolation. The four paths cover all failure error modes classified in step 102, while reliable observation key points do not require compensation and directly retain their initial positions. Each key point thus obtains a compensation position that matches its own failure cause and has been specifically corrected, becoming the input for subsequent consistency verification and weighted fusion.

[0074] Step 104: Based on the skeletal structure and motion relationships between the compensation positions of each key point, perform a consistency check on the compensation positions of each key point, and determine the convergence weight of each key point based on the consistency check results; The compensation locations may still contain errors, such as conflicts between individual compensation locations and the normal human skeletal structure, or discontinuities with historical motion trajectories. This step verifies the consistency of the compensation locations for each keypoint from two dimensions: skeletal structure relationship and motion relationship, and assigns a convergence weight to each keypoint accordingly. The convergence weight characterizes the overall credibility of the compensation location for that keypoint; a higher weight indicates a greater tendency to accept the compensation location during final fusion, while a lower weight indicates a greater tendency to accept the motion extrapolation location. The verification also incorporates three types of constraints: bone length, joint angle, and motion continuity.

[0075] Specifically, step 104 includes steps D1 to D8: Figure 9 The following is a flowchart illustrating a specific implementation of step 104 in a human posture recognition method for a motion-sensing game console provided by the present invention: Step D1: Determine the human image scale; wherein the human image scale is determined by the ratio between the image length of the reference limb segment formed by key points classified as reliable observation types in the current video frame and the prior value of the bone length of the reference limb segment, and the prior value of the bone length is the preset average physiological length of the corresponding limb segment. Skeletal length deviation verification requires unifying two different units: image pixel length and prior bone length values. Therefore, the human image scale must first be determined. The human image scale represents approximately how many millimeters or how many times the prior bone unit corresponds to each pixel in the current image. This step selects a reference limb segment composed of reliable observational keypoints, and divides its image length in the current frame by the prior bone length value of that limb segment to obtain the human image scale. For example, the torso segment from the neck to the hip can be selected as the reference limb segment. Since the reference limb segment is composed of reliable keypoints and its length is reliable, the resulting human image scale can accurately reflect the current imaging scaling ratio of the human body.

[0076] Step D2: For each limb segment, calculate the deviation between the image length between the compensated positions of the key points at both ends of the limb segment and the length of the limb segment's bone length prior value after scaling by the human image scale, and use this deviation as the bone length deviation of the limb segment. For each limb segment, the image length between the compensated keypoints at its two ends is calculated. Then, the predicted image length of the limb segment's bone length, after scaling down to the human image scale, is calculated. The absolute value of the difference between the two is the bone length deviation of the limb segment. If there is a large error at one end of the compensation position, the limb segment length will deviate significantly from its physiological length, and the bone length deviation will increase accordingly. This deviation directly measures the degree of distortion at the structural scale of the compensation position and is one of the key indicators for verification.

[0077] Step D3: For each key point, take the maximum value among the bone length deviations of each limb segment with the key point as the endpoint, and use it as the bone length deviation of the key point. Each keypoint typically connects one or more limb segments; for example, the elbow connects the upper arm and forearm. The maximum skeletal length deviation of each limb segment associated with that keypoint is taken as the skeletal length deviation of that keypoint. The rationale for taking the maximum value is that if any associated limb segment is significantly distorted at the structural scale, it indicates that the compensation position of that keypoint is likely to be incorrect. The most severe distortion should be used as the structural verification result for that keypoint to avoid the distortion of individual limb segments being masked by the average of the remaining normal limb segments.

[0078] Step D4: For each joint connecting two limb segments, calculate the included angle between the two limb segments at the joint, determined by the compensation position. When the included angle exceeds the feasible angle range of the joint, the excess amount is taken as the joint angle over-limit amount. For each joint connecting two limb segments, the included angle at the joint is calculated based on the compensated positions of the two limb segments. The feasible angle range of the joint is a preset range of physiologically rotatable angles for that joint. When the calculated included angle exceeds the feasible angle range, the excess is taken as the joint angle overshoot. For example, the normal bendable range of the elbow joint is approximately 0 to 150 degrees. If the compensated position causes the elbow joint included angle to reach 170 degrees, the overshoot is 20 degrees. Joint angle verification identifies obvious structural inconsistencies from the perspective of physiological rotation constraints, representing another layer of structural constraint besides bone length verification.

[0079] Step D5: For each key point, calculate the deviation between the compensated position of the key point and the extrapolated position of the key point's motion, and use this deviation as the temporal motion deviation of the key point; The motion extrapolation position is obtained by extrapolating the historical trajectory of the keypoint in the video frame sequence prior to the current frame. This step calculates the spatial deviation between the compensated position and the extrapolated position, which is used as the temporal motion deviation. If the compensated position matches the predicted position extrapolated from the historical trajectory, it indicates that the current positioning is consistent with the motion history; if the deviation is too large, it indicates that the compensated position contradicts the motion continuity of the keypoint, and there may be a positioning jump. The temporal motion deviation verifies the compensated position from the motion dimension, complementing the skeletal structure verification.

[0080] Step D6: For each key point, compare the bone length deviation of the key point, the joint angle overrun of the joint associated with the key point, and the temporal motion deviation of the key point with their respective preset verification thresholds, and count the number of inspection items that fail the comparison. For each keypoint, its bone length deviation, the joint angle exceeding the limit of the associated joint, and the temporal motion deviation are compared with their respective preset verification thresholds to determine whether they exceed the thresholds, and the number of checks that fail the comparison is counted. The three verification thresholds are determined by the statistical distribution of human physiological data. For example, the bone length deviation threshold can be taken as 20% of the prior value of bone length after scaling the human image, the joint angle exceeding the limit threshold can be taken as 15 degrees, and the temporal motion deviation threshold can be taken as the displacement corresponding to the historical average motion velocity of the keypoint. For keypoints marked as low-confidence locations, the bone length check of the limb segment to which it belongs is exempted according to the convention of step C24 and is not counted as a failed item.

[0081] Step D7: Calculate the ratio of the number of inspection items to the total number of inspection items, as the failure rate of the key point verification; The failure rate is calculated by dividing the number of failed checks by the total number of checks. The total number of checks represents the number of indicators actually involved in the verification of that keypoint. For low-confidence keypoints exempt from bone length checks, the total number of checks is reduced accordingly. The failure rate ranges from 0 to 1, where 0 indicates that all checks passed and the compensation position is highly reliable, and 1 indicates that all checks failed and the compensation position is extremely unreliable. This ratio compresses the multi-dimensional verification results into a normalized confidence metric, facilitating subsequent weight calculations.

[0082] Step D8: Calculate the product of the complement of the failure rate and the preset benchmark weight, and use the product as the convergence weight of the key point.

[0083] The convergence weight is derived from the failure rate of validation. The failure rate is negativeed, and then 1 is added to obtain its complement, representing the proportion of points that passed validation. This complement is multiplied by a preset baseline weight to obtain the convergence weight of the keypoint. For example, the baseline weight can be 0.6, meaning that when all validation items of a keypoint pass, its convergence weight is 0.6. The convergence weight decreases linearly as the failure rate increases; compensation positions with higher confidence receive greater weights, thus gaining more acceptance in subsequent fusion.

[0084] In the embodiments corresponding to steps D1 to D8, through the cascaded processing of the above steps, the multidimensional consistency of each keypoint compensation position is uniformly transformed into a single convergence weight. The human image scale unifies the dimensions of structural verification. Skeletal length deviation and joint angle overrun constitute a triple verification from the structural dimension, and temporal motion deviation from the motion dimension. The ratio of the number of failed items to the total number gives the normalized verification failure rate, which is then converted to the baseline weight using the complement. The result is that reliable compensation positions receive high weights, while distorted compensation positions have lower weights, providing a differentiated credibility basis for the next step of weighted fusion.

[0085] Step 105: According to the convergence weight, the compensation position of each key point and the corresponding motion extrapolation position are weighted and fused to output the human posture recognition result of the motion-sensing game console in the current video frame; After obtaining the convergence weights of each keypoint, this step weights and fuses the compensated positions of each keypoint with their corresponding motion extrapolation positions according to the convergence weights to obtain the final fused position. The complete human pose recognition result is then output according to the preset human skeleton connection relationship. The essence of fusion is to combine highly reliable compensated observations and motion predictions in a complementary manner under the guidance of weights: when the compensated position is reliable, observation takes precedence; when the compensated position is questionable, motion extrapolation serves as a fallback, thus balancing positioning accuracy and motion continuity.

[0086] Specifically, step 105 includes steps E1 to E4: Figure 10 The following is a flowchart illustrating a specific implementation of step 105 in a motion-sensing game console human posture recognition method provided by the present invention: Step E1: For each key point, calculate the product of the compensation position of the key point and the convergence weight, and use it as the first weighted position; For each keypoint, its compensation position is multiplied by its convergence weight to obtain the first weighted position. The first weighted position is the contribution of the compensation observation after being scaled by confidence. When the convergence weight is 1, the first weighted position is the complete compensation position; as the weight decreases, its contribution decreases accordingly.

[0087] Step E2: Calculate the product between the extrapolated position of the key point and the difference between 1 and the convergence weight, and use it as the second weighted position; Subtracting the convergence weight of the keypoint from 1 yields the difference. This difference is then multiplied by the motion extrapolation position to obtain the second weighted position. The second weighted position represents the contribution of the motion extrapolation scaled according to the complementary weights. Since the sum of the weights and the difference is 1, the contributions of the compensation observation and motion extrapolation are distributed complementaryly according to their weights. Keypoints with higher weights are more heavily weighted towards observation, while those with lower weights are more heavily weighted towards motion extrapolation.

[0088] Step E3: Add the first weighted position to the second weighted position to obtain the fused position of the key point; The first weighted position is added to the second weighted position to obtain the fused position of the keypoint. Since the sum of the two weights is 1, the fused position is actually a convex combination of the compensated position and the motion extrapolation position, falling on the line connecting the two. The larger the weight, the closer the fused position is to the compensated position; the smaller the weight, the closer it is to the motion extrapolation position. Therefore, when the compensated position is verified as reliable, the fusion result is close to the observation; when the compensation reliability is low, the motion prediction smoothly takes over the fusion result, avoiding positioning jumps.

[0089] Step E4: Connect the fusion positions of each key point according to the preset human skeleton connection relationship, and output the convergence weight of each key point as the output confidence of the corresponding key point to obtain the human posture recognition result of the motion-sensing game machine.

[0090] According to the preset human skeleton connection relationship, the fusion positions of each key point are sequentially connected to form a complete human skeleton pose. Simultaneously, the convergence weight of each key point is output as the corresponding key point's output confidence score, allowing downstream applications to obtain not only the coordinates of each key point but also their respective reliability information. The final output is the human pose recognition result of the motion-sensing game console for the current video frame, including the fusion positions of each key point, the skeleton connection relationship, and the output confidence score of each key point, which can be used as direct input for the motion-sensing game console to drive the mapping of game character movements.

[0091] In the embodiments corresponding to steps E1 to E4, through the cascaded processing of the above steps, the compensation observations and motion extrapolations of each key point are organically fused according to convergence weights. The first weighted position and the second weighted position respectively carry the credible contributions of the observation and prediction, and the summation yields the fusion position of the convex combination. The skeleton connection and output confidence organize the fusion result into a complete and consumable posture recognition result. As a result, the compensation position is accepted when it is credible, and the motion extrapolation smoothly takes over when it is inaccurate. The two are naturally connected under different confidence levels, so that the final posture has both positioning accuracy, motion continuity and structural rationality.

[0092] To verify the technical effectiveness of this embodiment, a comparative experiment was conducted on a dataset containing 12,000 frames of real-world motion-sensing game scene video samples. The samples covered four key degradation scenarios: single-person occlusion, motion blur, multiple-person interleaving, and edge-out of the frame, as well as normal scenarios. The experiment compared two methods: ① the direct peak method, which only takes the single peak position of the pose estimation network's output heatmap as the keypoint localization result, without any compensation or fusion; ② the human pose recognition method of this embodiment. Evaluation metrics included keypoint localization accuracy (the proportion of keypoints falling into the correct neighborhood under a tolerance threshold of 20% of the torso diameter) and pose trajectory jump rate (the percentage of abrupt changes in keypoint displacement between adjacent frames). The experimental results are shown in the table below.

[0093] For example, the comparison results of key point localization and trajectory continuity are shown in Table 2 below: Table 2 Comparison of Keypoint Localization and Trajectory Continuity Using Different Methods The experimental results show that, compared with the direct main peak method, the method in this embodiment improves the overall key point localization accuracy by 9.3 percentage points, by 18.3 percentage points in occluded scenarios, and by 13.3 percentage points in blurred and interlaced scenarios. The trajectory jump rate is reduced from 12.6% to 3.2%, a decrease of 74.6%. This indicates that error pattern classification and targeted compensation paths effectively reduce localization errors in occluded, blurred, and interlaced scenarios. The convergence weight-guided compensation observation and motion extrapolation fusion significantly stabilize the attitude trajectory while maintaining localization accuracy. The improvements in each aspect are superimposed in complex scenarios, resulting in a significant improvement in both the accuracy and continuity of the motion-sensing game console's attitude recognition.

[0094] In the embodiments corresponding to steps 101 to 105, by acquiring a continuous sequence of video frames and inputting the current video frame into a pre-trained pose estimation network, initial heatmaps of each human keypoint are obtained, and the response peak positions of the main peaks of each heatmap are extracted as the initial positions of the keypoints. This provides a unified and quantifiable observation benchmark for subsequent localization correction, overcoming the limitation of existing methods that directly take the original network output, resulting in localization being dominated by single-frame noise. By extracting the response distribution morphology features of each initial heatmap, and accordingly subdividing the failure causes of each keypoint into various error modes such as occlusion diffusion, blurring and splitting, instance confusion, out-of-bounds truncation, and reliable observation, a refined attribution of the failure causes of keypoint observation is achieved. This overcomes the limitation of traditional methods that treat all failures the same and cannot distinguish failure types, providing a classification basis for differentiated processing. By matching targeted compensation paths to the causes of each error pattern and correcting the initial position, structural completion in occluded scenes, temporal sharpening in blurred scenes, attribution arbitration in multi-instance scenes, and boundary extrapolation in out-of-bounds scenes can each leverage their strengths. This overcomes the shortcomings of single compensation methods that cannot cover multiple failure causes and the lag in compensation intervention, directly reducing the positioning error of failure key points at the causal level. By verifying the consistency of the compensation position based on skeletal structure and motion relationships and determining the convergence weight of each key point accordingly, physiological structural constraints and motion continuity constraints jointly participate in the assessment of position reliability. This overcomes the shortcomings of existing methods that indiscriminately fuse observations and predictions and lack the ability to distinguish abnormal positioning, suppressing the impact of structural conflicts and motion jumps on the final result. By weighting and fusing the compensated positions of key points with the motion extrapolation positions obtained from historical trajectory extrapolation according to convergence weights, static observation information and dynamic motion information are combined in a complementary manner according to credibility. This avoids abrupt changes in positioning caused by constant weights or hard switching, and stably connects the positioning results of each key point in complex scenarios such as occlusion, blurring, intersection, and boundaries. Ultimately, this improves the accuracy of human posture recognition, the continuity of motion tracking, and the real-time performance of game control in complex indoor environments for motion-sensing game consoles.

[0095] like Figure 2This invention provides a human posture recognition device for a motion-sensing game console. Please refer to [link / reference]. Figure 2 , Figure 2 This diagram illustrates a human posture recognition device for a motion-sensing game console provided by the present invention, as shown below. Figure 2 The human posture recognition device of the motion-sensing game console shown includes: The acquisition unit 21 is used to acquire a video frame sequence of the human body, input the current video frame in the video frame sequence into a pre-trained pose estimation network, obtain the initial heat map of each key point of the human body in the current video frame, and take the position of the main peak with the largest response value in each initial heat map as the initial position of each key point. Extraction unit 22 is used to extract the morphological features of each initial heatmap and match the error patterns corresponding to each key point based on the morphological features; wherein, the error pattern represents the category of key point observation failure due to different causes, and the error pattern includes occlusion diffusion type, blur split type, instance confusion type, out-of-bounds truncation type and reliable observation type. The matching unit 23 is used to match the compensation path corresponding to the error pattern, and correct the initial position of each key point based on the compensation path to obtain the compensation position of each key point; wherein, the compensation path is a processing flow that corrects the initial positioning of key points according to the cause of the corresponding error pattern. The verification unit 24 is used to verify the consistency of the compensation positions of each key point based on the skeletal structure relationship and motion relationship between the compensation positions of each key point, and to determine the convergence weight of each key point based on the consistency verification result. The fusion unit 25 is used to perform weighted fusion of the compensation position of each key point and the corresponding motion extrapolation position according to the convergence weight, and output the human posture recognition result of the motion-sensing game console in the current video frame; wherein, the motion extrapolation position is obtained by extrapolating the historical trajectory of the corresponding key point in the video frame sequence before the current video frame.

[0096] This invention provides a human posture recognition device for a motion-sensing game console. By acquiring a continuous sequence of video frames and inputting the current video frame into a pre-trained posture estimation network, initial heatmaps of each human keypoint are obtained. The peak response position of each heatmap's main peak is extracted as the initial position of the keypoint, providing a unified and quantifiable observation benchmark for subsequent localization correction. This overcomes the limitation of existing methods that directly use the network's raw output, leading to localization being dominated by single-frame noise. By extracting the response distribution morphology features of each initial heatmap, and accordingly subdividing the failure causes of each keypoint into various error modes such as occlusion diffusion, blurring and splitting, instance confusion, out-of-bounds truncation, and reliable observation, a refined attribution of keypoint observation failure causes is achieved. This overcomes the limitation of traditional methods that treat all failures the same and cannot distinguish failure types, providing a classification basis for differentiated processing. By matching targeted compensation paths to the causes of each error pattern and correcting the initial position, structural completion in occluded scenes, temporal sharpening in blurred scenes, attribution arbitration in multi-instance scenes, and boundary extrapolation in out-of-bounds scenes can each leverage their strengths. This overcomes the shortcomings of single compensation methods that cannot cover multiple failure causes and the lag in compensation intervention, directly reducing the positioning error of failure key points at the causal level. By verifying the consistency of the compensation position based on skeletal structure and motion relationships and determining the convergence weight of each key point accordingly, physiological structural constraints and motion continuity constraints jointly participate in the assessment of position reliability. This overcomes the shortcomings of existing methods that indiscriminately fuse observations and predictions and lack the ability to distinguish abnormal positioning, suppressing the impact of structural conflicts and motion jumps on the final result. By weighting and fusing the compensated positions of key points with the motion extrapolation positions obtained from historical trajectory extrapolation according to convergence weights, static observation information and dynamic motion information are combined in a complementary manner according to credibility. This avoids abrupt changes in positioning caused by constant weights or hard switching, and stably connects the positioning results of each key point in complex scenarios such as occlusion, blurring, intersection, and boundaries. Ultimately, this improves the accuracy of human posture recognition, the continuity of motion tracking, and the real-time performance of game control in complex indoor environments for motion-sensing game consoles.

[0097] Figure 3 This is a schematic diagram of a terminal device provided in an embodiment of the present invention. Figure 3 As shown, a terminal device 3 in this embodiment includes: a processor 30, a memory 31, and a computer program 32 stored in the memory 31 and executable on the processor 30, such as a human posture recognition program for a motion-sensing game console. When the processor 30 executes the computer program 32, it implements the steps described in the various embodiments of the human posture recognition method for a motion-sensing game console, for example... Figure 1 Steps 101 to 105 are shown. Alternatively, when the processor 30 executes the computer program 32, it implements the functions of each unit in the above-described device embodiments, for example... Figure 2 The function of the unit shown.

[0098] For example, the computer program 32 can be divided into one or more units, which are stored in the memory 31 and executed by the processor 30 to complete the present invention. The one or more units can be a series of computer program instruction segments capable of performing a specific function, which describe the execution process of the computer program 32 in the terminal device 3. For example, the specific functions of each unit of the computer program 32 can be divided as follows: The acquisition unit is used to acquire a current image of the power equipment, extract the gradient direction of each pixel in the current image of the power equipment, accumulate votes along the gradient direction of each pixel to accumulate the location of the target to be identified, and take the location where the accumulated vote value meets the preset peak condition as the candidate location hypothesis; wherein, the candidate location hypothesis indicates the possible location of the target in the current image of the power equipment. The synthesis unit is used to synthesize the expected appearance image corresponding to each candidate location hypothesis based on the location and scale indicated by the candidate location hypothesis through the appearance prediction model. The calculation unit is used to compare the expected appearance image corresponding to the candidate location hypothesis with the observed image of the corresponding location in the current power equipment image, and calculate the imaging consistency score of each candidate location hypothesis; wherein, the imaging consistency score characterizes the degree of consistency between the expected appearance image and the observed image; The judgment unit is used to take the position indicated by the candidate position hypothesis with the highest imaging consistency score as the operation target of the power system operation robot.

[0099] The terminal device includes, but is not limited to, a processor 30 and a memory 31. Those skilled in the art will understand that... Figure 3 This is merely an example of a terminal device 3 and does not constitute a limitation on a terminal device 3. It may include more or fewer components than shown, or combine certain components, or different components. For example, the terminal device may also include input / output devices, network access devices, buses, etc.

[0100] The processor 30 can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor.

[0101] The memory 31 can be an internal storage unit of the terminal device 3, such as a hard disk or memory of the terminal device 3. The memory 31 can also be an external storage device of the terminal device 3, such as a plug-in hard disk, smart media card (SMC), secure digital card (SD), flash card, etc., equipped on the terminal device 3. Furthermore, the memory 31 can include both internal and external storage units of the terminal device 3. The memory 31 is used to store the computer program and other programs and data required by the roaming control device. The memory 31 can also be used to temporarily store data that has been output or will be output.

[0102] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0103] It should be noted that the information interaction and execution process between the above-mentioned devices / units are based on the same concept as the method embodiments of the present invention. For details on their specific functions and technical effects, please refer to the method embodiments section, which will not be repeated here.

[0104] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this invention. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0105] This invention also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps described in the various method embodiments above.

[0106] This invention provides a computer program product that, when run on a mobile terminal, enables the mobile terminal to implement the steps described in the above-described method embodiments.

[0107] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments of the present invention can be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include at least: any entity or device capable of carrying the computer program code to a photographing device / terminal device, a recording medium, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium. Examples include USB flash drives, portable hard drives, magnetic disks, or optical disks.

[0108] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0109] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0110] In the embodiments provided by this invention, it should be understood that the disclosed apparatus / network devices and methods can be implemented in other ways. For example, the apparatus / network device embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0111] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; they may be located in one place or distributed across multiple network units.

[0112] It should be understood that, when used in this specification and the appended claims, terms include indicating the presence of a described feature, integral, step, operation, element and / or component, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or collections thereof.

[0113] It should also be understood that the terms used in this specification and the appended claims refer to any combination of one or more of the associated listed items and all possible combinations, and include such combinations.

[0114] As used in this specification and the appended claims, the term "if" can be interpreted, depending on the context, as when, once, or in response to determination or in response to detection. Similarly, the phrase "if determined" or "if detected [the described condition or event]" can be interpreted, depending on the context, as once determined or in response to determination or detection [the described condition or event] or in response to detection [the described condition or event].

[0115] Furthermore, in the description of this invention and the appended claims, the terms first, second, third, etc., are used only for distinguishing descriptions and should not be construed as indicating or implying relative importance.

[0116] References to one or more embodiments described in this specification mean that a particular feature, structure, or characteristic described in connection with that embodiment is included in one or more embodiments of the invention. Therefore, phrases appearing in different parts of this specification as referring to one embodiment, some embodiments, some other embodiments, and others do not necessarily refer to the same embodiment, but rather mean one or more, but not all, embodiments, unless otherwise specifically emphasized. The terms include, encompass, have, and variations thereof mean including, but are not limited to, unless otherwise specifically emphasized.

[0117] The above-described embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.

Claims

1. A method for human posture recognition in a motion-sensing game console, characterized in that, The human posture recognition method of the motion-sensing game console includes: A sequence of video frames of the human body is collected. The current video frame in the sequence is input into a pre-trained pose estimation network to obtain an initial heatmap of each key point of the human body in the current video frame. The position of the main peak with the largest response value in each initial heatmap is taken as the initial position of each key point. The morphological features of each initial heatmap are extracted, and the error patterns corresponding to each key point are matched based on the morphological features. The error patterns represent the categories of key point observations that fail due to different causes. The error patterns include occlusion diffusion type, blur split type, instance confusion type, out-of-bounds truncation type, and reliable observation type. The compensation path corresponding to the error pattern is matched, and the initial position of each key point is corrected based on the compensation path to obtain the compensation position of each key point; wherein, the compensation path is a processing flow that corrects the initial positioning of key points according to the cause of the corresponding error pattern. Based on the skeletal structure and motion relationships between the compensation positions of each key point, the consistency of the compensation positions of each key point is verified, and the convergence weight of each key point is determined based on the consistency verification results. The compensation positions of each key point and the corresponding motion extrapolation positions are weighted and fused according to the convergence weight to output the human posture recognition result of the motion-sensing game console in the current video frame; wherein, the motion extrapolation positions are obtained by extrapolating the historical trajectories of the corresponding key points in the video frame sequence before the current video frame.

2. The human posture recognition method for a motion-sensing game console according to claim 1, characterized in that, The step of extracting the morphological features of each initial heatmap and matching the error patterns corresponding to each key point based on the morphological features includes: For each initial heatmap, the ratio between the peak value of the main peak in the initial heatmap and the average response value of the set of pixels in the initial heatmap whose response values ​​exceed a preset response threshold is calculated as the main peak sharpness. Extract the second peak whose response value is second only to the main peak in the initial heatmap, and calculate the ratio between the peak value of the second peak and the peak value of the main peak as the competition degree of the second peak; The number of pixels contained in the pixel set is counted and used as the area of ​​the supporting domain; The number of pixels in the pixel set that are adjacent to the image boundary of the current video frame is counted, and the ratio between the number of adjacent pixels and the area of ​​the supporting domain is calculated as the edge truncation ratio. When the competition level of the secondary peak is higher than the preset competition threshold, and the distance between the secondary peak and the primary peak is greater than the scale prior value of the limb segment where the key point is located, the error mode of the key point is classified as instance confusion type; wherein, the scale prior value represents the estimated length of the limb segment where the key point is located in the image. When the area of ​​the support domain is greater than the preset diffusion threshold and the sharpness of the main peak is lower than the preset sharpness threshold, the error mode of the key point is classified as occlusion diffusion type. When the edge truncation ratio is greater than the preset truncation threshold, the error pattern of the key point is classified as out-of-bounds truncation type. When the sharpness of the main peak is lower than the preset sharpness threshold and the area of ​​the support domain is not greater than the preset diffusion threshold, the error mode of the key point is classified as fuzzy splitting type. When the initial heatmap of the key point does not meet the classification conditions of instance confusion type, occlusion diffusion type, out-of-bounds truncation type and fuzzy split type, the error mode of the key point is classified as reliable observation type; When the initial heatmap of the key point simultaneously meets the classification conditions of multiple error modes, the key point is classified into the error mode with the highest priority according to the preset priority order.

3. The human posture recognition method for a motion-sensing game console according to claim 1, characterized in that, When the error pattern is classified as occlusion diffusion type, the compensation path is a structural completion path; The step of matching the compensation path corresponding to the error pattern and correcting the initial position of each key point based on the compensation path to obtain the compensation position of each key point includes: Select key points that are located on the same motion chain as the current key point and whose error mode is reliable observation type as support key points; Based on the location of the support key point, the prior value of the bone length of the limb segment between the support key point and the current key point, and the feasible angle range of the joint at the support key point, multiple candidate locations of the current key point are generated; wherein, the prior value of the bone length is the preset average physiological length of the corresponding limb segment, and the feasible angle range is the preset physiological rotation angle range of the joint. For each candidate location, a local image patch of a preset size is extracted centered on the candidate location, and the edge response intensity of the local image patch in a preset edge direction is extracted; wherein, the preset edge direction is perpendicular to the direction from the support key point to the candidate location; Calculate the proportion of the edge response intensity of each candidate location in the sum of the edge response intensities of all candidate locations, and use it as the evidence score for each candidate location; Calculate the score difference between the evidence score of the candidate position with the highest evidence score and the evidence score of the candidate position with the second highest evidence score; When the score difference is greater than a preset interval threshold, the candidate position with the highest evidence score is used as the compensation position of the current key point. When the score difference is not greater than the preset interval threshold, the candidate positions are weighted and summed according to their respective evidence scores, and the resulting position is used as the compensation position of the current key point.

4. The human posture recognition method for a motion-sensing game console according to claim 1, characterized in that, When the error pattern is classified as blur-split type, the compensation path is a temporal sharpening path; The step of matching the compensation path corresponding to the error pattern and correcting the initial position of each key point based on the compensation path to obtain the compensation position of each key point includes: Select a predetermined number of consecutive historical video frames preceding the current video frame from the video frame sequence; For each historical video frame, the motion field between the current video frame and the historical video frame is calculated by optical flow estimation, and the initial position of the current key point in the historical video frame is determined according to the motion field. Spatial alignment is performed on the initial heatmap of the current key point in each historical video frame according to the offset between the corresponding position and the initial position. For each initial heatmap after spatial alignment, calculate the ratio between the maximum response value of the initial heatmap and the average response value of the pixels in the initial heatmap whose response value exceeds a preset threshold, and normalize the ratio corresponding to each initial heatmap to obtain the superposition weight corresponding to each frame's initial heatmap. The initial heatmaps of each spatially aligned historical video frame and the initial heatmap of the current video frame are superimposed pixel by pixel according to their respective superposition weights to obtain the sharpened heatmap of the current key point. The location with the largest response value in the sharpened heatmap is taken as the compensation location for the key point.

5. The human posture recognition method for a motion-sensing game console according to claim 1, characterized in that, When the error pattern is classified as instance confusion type, the compensation path is the instance attribution arbitration path; The step of matching the compensation path corresponding to the error pattern and correcting the initial position of each key point based on the compensation path to obtain the compensation position of each key point includes: Human instance detection is performed on the current video frame to obtain the detection box for each human instance; Extract the second peak whose response value in the initial heatmap of the current key point is second only to the main peak, and match the positions of the main peak and the second peak with the detection boxes of each human instance. The number of keypoints with the error mode of reliable observation within the detection box of each human instance is counted, and the human instance with the largest number is taken as the target human instance to which the keypoint belongs. When the position of the main peak falls within the detection frame of the target human body instance, the position of the main peak is used as the compensation position of the current key point; When the position of the main peak does not fall within the detection frame of the target human instance, but the position of the secondary peak falls within the detection frame of the target human instance, the position of the secondary peak is used as the compensation position of the current key point. When neither the position of the main peak nor the position of the secondary peak falls within the detection frame of the target human body instance, the error mode of the current key point is reclassified as occlusion diffusion type, and the compensation path processing flow corresponding to the occlusion diffusion type is executed for the current key point.

6. The human posture recognition method for a motion-sensing game console according to claim 1, characterized in that, When the error pattern is classified as out-of-bounds truncation, the compensation path is a boundary extrapolation path. The step of matching the compensation path corresponding to the error pattern and correcting the initial position of each key point based on the compensation path to obtain the compensation position of each key point includes: Select the other end of the limb segment to which the current key point belongs as the key point with a reliable observation type error pattern, and use it as the anchor key point; From the video frame sequence, determine the reference video frame in which the current key point was most recently classified as a reliable observation type, and record the limb segment image length between the anchored key point and the current key point in the reference video frame; Based on the orientation of the current key point in the reference video frame relative to the position of the anchor key point in the reference video frame and the motion direction of the anchor key point from the reference video frame to the current video frame, determine the extension direction vector of the limb segment toward the image boundary of the current video frame. Starting from the position of the anchor key point in the current video frame, extrapolate along the extension direction vector by a distance equal to the length of the limb segment image, and use the obtained position as the compensation position of the current key point; The compensation position of the current key point is marked as a low-confidence position; wherein, the key point marked as a low-confidence position is exempted from the consistency check of the consistency between the image length of the limb segment to which the current key point belongs and the prior value of the bone length, wherein the prior value of the bone length is the preset average physiological length of the corresponding limb segment.

7. The human posture recognition method for a motion-sensing game console according to claim 1, characterized in that, The step of performing a consistency check on the compensation positions of each key point based on the skeletal structure and motion relationships between their compensation positions, and determining the convergence weight of each key point based on the consistency check results, includes: Determine the scale of the human body image; wherein the scale of the human body image is determined by the ratio between the image length of the reference limb segment formed by key points classified as reliable observation types in the current video frame and the prior value of the bone length of the reference limb segment, wherein the prior value of the bone length is the preset average physiological length of the corresponding limb segment. For each limb segment, the deviation between the image length between the compensated positions of the key points at both ends of the limb segment and the length of the limb segment after scaling by the human image scale is calculated, and this deviation is taken as the limb segment's bone length deviation. For each key point, the maximum value among the bone length deviations of each limb segment with the key point as the endpoint is taken as the bone length deviation of the key point. For each joint connecting two limb segments, calculate the included angle between the two limb segments at the joint, determined by the compensation position. When the included angle exceeds the feasible angle range of the joint, the excess amount is taken as the joint angle over-limit amount. For each key point, calculate the deviation between the compensated position of the key point and the extrapolated position of the key point's motion, and use it as the temporal motion deviation of the key point; For each key point, the bone length deviation of the key point, the joint angle overrun of the joint associated with the key point, and the temporal motion deviation of the key point are compared with their respective preset verification thresholds, and the number of inspection items that fail the comparison is counted. Calculate the ratio of the number of inspection items to the total number of inspection items, and use it as the failure rate of the verification of the key point; Calculate the product of the complement of the failure rate and the preset benchmark weight, and use the product as the convergence weight of the key point.

8. The human posture recognition method for a motion-sensing game console according to claim 1, characterized in that, The step of weightedly fusing the compensation positions of each key point with the corresponding motion extrapolation positions according to the convergence weight, and outputting the human posture recognition result of the motion-sensing game console for the current video frame includes: For each key point, calculate the product of the compensation position of the key point and the convergence weight, and use it as the first weighted position; The product of the extrapolated position of the key point and the difference between 1 and the convergence weight is calculated as the second weighted position; Add the first weighted position to the second weighted position to obtain the fused position of the key point; The fusion positions of each key point are connected according to the preset human skeleton connection relationship, and the convergence weight of each key point is output as the output confidence of the corresponding key point to obtain the human posture recognition result of the motion-sensing game machine.

9. A human posture recognition device for a motion-sensing game console, characterized in that, The human posture recognition device of the motion-sensing game console includes: The acquisition unit is used to acquire a video frame sequence of the human body, input the current video frame in the video frame sequence into a pre-trained pose estimation network, obtain the initial heat map of each key point of the human body in the current video frame, and take the position of the main peak with the largest response value in each initial heat map as the initial position of each key point. The extraction unit is used to extract the morphological features of each initial heatmap and match the error patterns corresponding to each key point based on the morphological features; wherein, the error pattern represents the category of key point observation failure due to different causes, and the error pattern includes occlusion diffusion type, blur split type, instance confusion type, out-of-bounds truncation type and reliable observation type. A matching unit is used to match the compensation path corresponding to the error pattern, and correct the initial position of each key point based on the compensation path to obtain the compensation position of each key point; wherein, the compensation path is a processing flow that corrects the initial positioning of key points according to the cause of the corresponding error pattern. The verification unit is used to verify the consistency of the compensation positions of each key point based on the skeletal structure relationship and motion relationship between the compensation positions of each key point, and to determine the convergence weight of each key point based on the consistency verification result. The fusion unit is used to perform weighted fusion of the compensation position of each key point and the corresponding motion extrapolation position according to the convergence weight, and output the human posture recognition result of the motion-sensing game console in the current video frame; wherein, the motion extrapolation position is obtained by extrapolating the historical trajectory of the corresponding key point in the video frame sequence before the current video frame.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the steps in the human posture recognition method of the motion-sensing game console as described in any one of claims 1 to 8.

Images