Human-computer interaction implementation method and system based on visual perception

Abstract

The invention discloses a human-computer interaction implementation method and system based on visual perception, and the method comprises the steps: extracting an action feature from a target region of a depth image as a first feature, extracting a key point as a second feature, and extracting an instantaneous deformation feature as a third feature; obtaining a three-dimensional motion model according to the second feature and the third feature; performing pose estimation according to the first feature and the second feature to obtain a key point descriptor; constructing an attention function based on the deviation between the key point descriptors; constructing a semantic segmentation network based on the key point descriptor and the attention function, and learning the characteristics of the interaction action by using the semantic segmentation network; and judging whether to interact based on the size of the overlapping region in the cluster, and if so, executing interaction based on the semantic segmentation network. The method improves the accurate recognition rate of interaction, has good interpretability, and can be directly applied to a man-machine interaction system.

Description

Technical Field

[0001] The present invention relates to the field of visual perception, and in particular to a method and system for realizing human-computer interaction based on visual perception. Background Art

[0002] With the development of technology and the improvement of user experience requirements, people expect more natural and intuitive human-computer interaction methods, directly interacting with computer systems through natural methods such as body movements, gestures, and expressions. This natural interaction method can improve interaction efficiency, reduce user learning costs, and provide convenient interaction experience in more scenarios.

[0003] Traditional human-computer interaction implementation methods often rely on command line interaction and model matching. Moreover, most traditional methods lack a deep understanding of scene semantic information and can only handle some pre-set and relatively fixed interaction modes.

[0004] In today's digital age, human-computer interaction plays a key role in many fields. Therefore, a human-computer interaction implementation method and system based on visual perception is needed to help accurately identify user intentions, make the interaction more natural and efficient, and improve the overall experience of users in different scenarios. Summary of the invention

[0005] The purpose of the present invention is to provide a method and system for realizing human-computer interaction based on visual perception.

[0006] To achieve the above object, the present invention is implemented according to the following technical solutions:

[0007] A first aspect of the present invention provides a method for implementing human-computer interaction based on visual perception, comprising:

[0008] S100 acquires a depth image through a visual sensor, extracts action features as a first feature, extracts key points as a second feature, and extracts instantaneous deformation features as a third feature from a target area of ​​the depth image;

[0009] S200 obtains a global key frame according to the second feature and the third feature, constructs a Gaussian subgraph, anchors the Gaussian subgraph to the corresponding global key frame, and obtains a three-dimensional action model;

[0010] S300 performs pose estimation according to the first feature and the second feature, evaluates a feature map associated with the three-dimensional motion model at a key point position, and obtains a key point descriptor; and constructs an attention function based on a deviation between the key point descriptors;

[0011] S400 constructs a semantic segmentation network based on the key point descriptor and the attention function, clusters the interactive actions, and uses the semantic segmentation network to learn feature representations of the interactive actions in the same cluster;

[0012] S500 obtains the user's interactive actions, maps the interactive actions to the cluster, determines whether to interact based on the size of the overlapping area in the cluster, and if interaction is performed, executes the interaction based on the semantic segmentation network.

[0013] As a further method, the method of extracting action features as the first feature, extracting key points as the second feature, and extracting instantaneous deformation features as the third feature from the target area of ​​the depth image includes:

[0014] Preprocess the depth image, including: noise reduction, illumination correction, histogram equalization and smoothing;

[0015] The target area is selected in the depth image in an interactive manner, the convolution kernel is slid in the target area for feature extraction, the action features are extracted using the optical flow algorithm, the key points are extracted using the HRNet network, and the instantaneous deformation features are extracted based on time series analysis.

[0016] As a further method, the method for obtaining a global key frame according to the second feature and the third feature includes:

[0017] The extracted second feature data and third feature data are normalized and aligned, and the second feature and the third feature are combined into a new feature vector by vector concatenation. Each key point has a corresponding instantaneous deformation feature. The new feature vector is clustered to obtain several clusters containing frames.

[0018] Calculate the degree of deviation of each frame relative to the cluster center. The expression is:

[0019]

[0020] Among them, ω1 and ω2 represent the weight coefficients for controlling the Mahalanobis distance and the degree of deviation of the Gaussian kernel function, respectively, x1 is the feature vector of the frame, μ is the feature vector of the cluster center, T represents the transposition operation, ∑ is the covariance matrix, and σ is the bandwidth parameter in the Gaussian kernel function;

[0021] The frames with the largest deviation are selected from each cluster and combined to form a global keyframe.

[0022] As a further method, the method of constructing a Gaussian subgraph and anchoring the Gaussian subgraph to the corresponding global key frame to obtain a three-dimensional motion model includes:

[0023] The key point coordinates and instantaneous deformation features in each global key frame are used as data points in the high-dimensional space. For each data point in the high-dimensional space, the kernel density estimate is calculated, and the expression is:

[0024]

[0025] Among them, m is the total number of data points, n is the dimension of the data points, ∑ is the covariance matrix, x2 is any data point in the high-dimensional space, and y i is the i-th data point in the high-dimensional space, and T represents the transposition operation;

[0026] Based on the kernel density estimation value, a numerical matrix representing the data distribution density is obtained as a Gaussian subgraph;

[0027] Construct a spatial transformation matrix according to the spatial changes of key points in the global key frame, and transform the constructed Gaussian subgraph according to the spatial transformation matrix;

[0028] The transformed Gaussian sub-image is merged with the global key frame through weighted averaging to form a 3D motion model.

[0029] As a further method, the method of performing pose estimation based on the first feature and the second feature, evaluating a feature map associated with the three-dimensional motion model at a key point position, and obtaining a key point descriptor includes:

[0030] A pose estimation model is constructed using a neural network model according to the first feature and the second feature, the action feature and the key point feature are used as input, and a preliminary pose estimation result is output;

[0031] Considering the depth image in time series, the preliminary pose estimation result of the current frame is fused with the preliminary pose estimation result of the previous frame by weighted averaging. The weight distribution is determined according to the reliability of the pose of the previous frame and the confidence of the current frame. The parameters of the pose estimation model are continuously updated until the processing of each frame of the depth image is completed.

[0032] According to the output results of the optimized pose estimation model, the positions of the key points in the three-dimensional motion model are determined, and the three-dimensional coordinates are converted into two-dimensional coordinates in the feature map;

[0033] In the feature map, the local area is defined with the position of the key point as the center. The size of the area is determined by the resolution of the feature map, and the expression is:

[0034]

[0035] Among them, α is the scale factor, W is the width of the feature map, and H is the height of the feature map;

[0036] Calculate the gradient features, texture features, and depth change features of the pixels in the local area, and connect them in the order of the gradient direction to form a high-dimensional key point descriptor, expressed as:

[0037]

[0038] Where D(ξ,σ1,θ) is the key point descriptor at position ξ, scale σ1 and gradient direction θ, N is the normalization factor, Ω(ξ,σ1) represents the local area centered on ξ and with scale σ1, u1 and v1 are the pixel position indexes in the local area, They are respectively represented as the image gradients at pixel positions u1 and v1, ⊙ represents the Hadanard product between vectors, h(·) is a nonlinear activation function, It is expressed as the inner product of the image gradient at pixel positions u1 and v1, which is used to extract texture features and depth change features. ω(ξ,σ1,θ) is a weighting function based on the gradient direction θ, which is used to emphasize the gradient consistent with the direction of the key point.

[0039] As a further method, the method for constructing an attention function based on the deviation between the key point descriptors includes:

[0040] The average value of the Euclidean distance between the key point descriptors is calculated as the deviation, and the attention function is constructed based on the deviation. The expression is:

[0041]

[0042] Among them, α ij is the adjustment coefficient between the i-th and j-th key point descriptors, W Q , W K and W V They are represented as the query vector, key vector and value vector of the key point descriptor, G i and G j They are respectively represented as the i-th and j-th key point descriptors, f(·) represents the exponential distance deviation adjustment function, φ is the nonlinear activation function sigmoid, β is the coefficient of the influence of the average deviation on attention, is the average value of the Euclidean distance between all key point descriptors, g(·) is the distance deviation norm function, softmax(·) is the normalized exponential function, softmax(tanh(W Q G i +W K G j )) ij It is represented as normalizing all attention outputs.

[0043] As a further method, the method for constructing a semantic segmentation network based on the key point descriptor and the attention function includes:

[0044] Use the deep learning framework to build a semantic segmentation network, including: input layer: take the key point descriptor as the input of the network; convolution layer: apply the convolution layer to extract the features of the input key point descriptor; pooling layer: insert the average pooling layer between the convolution layers to reduce the resolution of the feature map and increase the receptive field; attention mechanism fusion layer: apply the attention function to the output feature map of the convolution layer, and assign different weights to different feature areas according to the deviation information; semantic segmentation layer: upsampling layer, restore the feature map after attention weighting to the resolution corresponding to the input image; classification layer, add a fully connected layer at the end of the network to map the feature map to the semantic category; output layer, set the number of output nodes of the semantic segmentation network based on the number of instructions to be set;

[0045] The expression of the objective function of the semantic segmentation network is:

[0046]

[0047] in, is the total number of pixels in the image, ζ i is the category of the i-th pixel in the true label, To predict the category probability of the i-th pixel in the output, is the weight coefficient for balancing the cross entropy and regularization term, λ is the regularization coefficient, M is the number of channels in the feature map, K is the number of elements in each channel, and W A,jk is the attention weight matrix W A The value of the kth element of the jth channel in , is the mean attention weight of the jth channel;

[0048] Based on the attention function, a spatial branch is introduced into the semantic segmentation network. The expression of the spatial branch is:

[0049]

[0050] Among them, F SG is a fully connected layer, σ4 represents a nonlinear transformation of the output of the softmax operation, and F SM (·) is the softmax operator, σ2 represents the nonlinear transformation of the output of the global pooling operation, and F GP (·) is the global pooling operator, W q is the weight parameter of the convolution kernel, σ3 represents the nonlinear transformation of the output of the convolution operation, W v (·) is the convolution operator, X is the input feature map;

[0051] Obtain the data set, divide it into training set, validation set and test set in the ratio of 8:1:1, normalize it, and train the semantic segmentation network.

[0052] As a further method, the method of using the semantic segmentation network to learn feature representations of interactive actions in the same cluster includes:

[0053] The kernel density estimation method is used to calculate the probability density function of the interaction actions in the same cluster, and the expression is:

[0054]

[0055] in, is the estimated density function at point x3, n is the number of samples, d is the dimension of the data, x3 is the point in the d-dimensional space where the probability density needs to be estimated, and x i is the i-th d-dimensional sample point, H1 is the bandwidth matrix of the kernel function, and T represents the transposition operation. is the probability density estimation function that depends on the bandwidth matrix H1, X1 is the variable in the entire d-dimensional space, Expressed as constructing a probability density estimate When , the estimation function obtained from the i-th d-dimensional sample point is not used;

[0056] The kernel density estimate is used as the weight to perform weighted summation on the image features, allowing the semantic segmentation network to learn the distribution characteristics of the data within the cluster. At the same time, the convolutional layer in the semantic segmentation network is used to learn the characteristics of the cluster center vector and the covariance matrix within the cluster.

[0057] As a further method, the method of mapping the interactive action to the cluster and determining whether there is an interaction based on the size of the overlapping area in the cluster includes:

[0058] For each cluster, obtain the location of its cluster center. When the user's interactive action feature vector is obtained, calculate the distance d1 from the cluster center. Cross-sample action vectors at a ratio of 10% within the cluster as the action vector set, and calculate the distance d1 between each action vector in the set and the cluster center. si ;

[0059] Statistical distance d si The number of vectors m1 that is less than or equal to d1, if m1 exceeds half of the number of selected samples, it means that the interaction action is relatively central in the cluster and the size of the overlapping area exceeds the threshold, and it is determined to be interactive, otherwise, it is determined not to be interactive.

[0060] A second aspect of the present invention provides a human-computer interaction implementation system based on visual perception, comprising:

[0061] A feature extraction module, used to obtain a depth image through a visual sensor, extract action features as a first feature, extract key points as a second feature, and extract instantaneous deformation features as a third feature from a target area of ​​the depth image;

[0062] A three-dimensional construction module, used to obtain a global key frame according to the second feature and the third feature, construct a Gaussian subgraph, anchor the Gaussian subgraph to the corresponding global key frame, and obtain a three-dimensional action model;

[0063] A feature conversion module, configured to perform pose estimation based on the first feature and the second feature, evaluate a feature map associated with the three-dimensional motion model at a key point position, and obtain a key point descriptor; and construct an attention function based on a deviation between the key point descriptors;

[0064] A semantic network module, used to construct a semantic segmentation network based on the key point descriptor and the attention function, cluster the interaction actions, and use the semantic segmentation network to learn the feature representation of the interaction actions in the same cluster;

[0065] The interaction execution module is used to obtain the user's interaction actions, map the interaction actions to the cluster, determine whether to interact based on the size of the overlapping area in the cluster, and if interaction is performed, execute the interaction based on the semantic segmentation network.

[0066] Compared with the prior art, the embodiments of the present invention have at least the following advantages or beneficial effects:

[0067] (1) The present invention forms a three-dimensional model by merging the transformed Gaussian subgraph with the global key frame. The depth information of the key frame can be used to construct a three-dimensional action model that can reflect the spatial structure of the interactive action in the depth image and the data distribution characteristics through the Gaussian subgraph. This model can be used to describe the state of the object's action, posture, etc. in three-dimensional space.

[0068] (2) The present invention obtains the first Gaussian subgraph and the second Gaussian subgraph and constructs an attention function based on the deviation between the Gaussian subgraphs. The spatial attention mechanism enables the model to focus on the key areas in the image and ignore irrelevant areas, thereby improving the accuracy and efficiency of detection.

[0069] (3) The present invention clusters actions based on interaction needs, uses a semantic segmentation network to learn the same-cluster feature representation, and finally obtains the user interaction actions and maps them into clusters. The interaction is judged and executed based on the overlapping areas within the clusters. This method is more intelligent and accurate in processing user interaction actions, fully considering the diversity of different user actions, and can better adapt to users' operating habits and interaction needs. BRIEF DESCRIPTION OF THE DRAWINGS

[0070] Figure 1The present invention is a flowchart of a method for realizing human-computer interaction based on visual perception in an embodiment of the present invention. DETAILED DESCRIPTION

[0071] The technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0072] Reference Figure 1 As shown, the present invention provides a method for realizing human-computer interaction based on visual perception, comprising:

[0073] S100 acquires a depth image through a visual sensor, extracts action features as a first feature, extracts key points as a second feature, and extracts instantaneous deformation features as a third feature from a target area of ​​the depth image;

[0074] In one embodiment, a vision-based rehabilitation training assistance field is used to identify, evaluate and interactively feedback the patient's rehabilitation movements;

[0075] In the actual evaluation, an Intel RealSense D435 depth camera is used as a visual sensor. The depth image resolution of the camera is 1280×720 pixels, the field of view is 87° horizontally and 58° vertically, and the frame rate can be set to 30fps or 60fps. In this embodiment, the frame rate is set to 30fps to obtain a depth image sequence, and 20 rehabilitation patients (aged between 25 and 65 years old, including patients in various physical conditions and rehabilitation stages) are selected for rehabilitation training action data collection. The rehabilitation training actions include knee flexion and extension, elbow flexion and extension, and hip abduction and adduction. Each patient performs 15 repetitions for each action, and each demonstration lasts for 10-15 seconds. A total of 900 depth image sequences (20 patients×3 actions×15 repetitions) are collected;

[0076] In the actual evaluation, for the target area of ​​each depth image (i.e., the area where the relevant joints of the human body are located), multiple features are obtained. For the knee flexion and extension movement, the knee angle change rate is calculated, and the key points of the knee joint are identified by the optical flow algorithm. The angle change rate is calculated based on the difference in the knee angle between adjacent frames divided by the time interval. In the collected data, the knee angle change rate of the knee flexion and extension movement ranges from 0.1 to 0.5 radians per second, and the average angle change rate is about 0.3 radians per second. At the same time, the energy characteristics of the movement are calculated, and the changes in depth values ​​in the target area are analyzed and converted into energy values. For a knee flexion and extension movement, the energy value fluctuates between 50 and 200 joules during the entire movement, and the average energy value is 120 joules, which is combined as the first feature. The HRNet posture estimation model is used to extract the key points of the human body from the depth image. For knee flexion and extension, the key points of the knee, hip and ankle are focused on. For elbow flexion and extension, the key points of the elbow, shoulder and wrist are focused on. For hip abduction and adduction, the key points of the hip, knee and waist are focused on. Taking a certain frame of knee flexion and extension as an example, the depth image of the knee key point is (450,320), the hip key point is (400,280), and the ankle key point is (480,380). The depth value of each key point is recorded. The depth value of the knee key point is 1.5 meters, the hip key point is 1.4 meters, and the ankle key point is 1.6 meters, completing the extraction of the second feature. The instantaneous deformation features are calculated for the local area around each joint. In the knee flexion and extension movement, a circular area with a radius of 50 pixels is selected as the local area with the knee joint as the center, and the standard deviation of the depth value in the area is calculated as the instantaneous deformation feature. During the movement, this standard deviation varies between 0.05 and 0.2 meters. When the knee joint is bent to the maximum angle, the standard deviation reaches a maximum value of 0.2 meters. For the elbow flexion and extension movement, the standard deviation of the depth value of the circular area with a radius of 30 pixels with the elbow joint as the center is between 0.03 and 0.15 meters, with an average of 0.08 meters, completing the extraction of the third feature;

[0077] S200 obtains a global key frame according to the second feature and the third feature, constructs a Gaussian subgraph, anchors the Gaussian subgraph to the corresponding global key frame, and obtains a three-dimensional action model;

[0078] It should be explained that the purpose of obtaining the global keyframe is to filter out the key parts that summarize the action process from a large number of frame data. The purpose is to reduce the amount of data and summarize the action characteristics, which is convenient for subsequent analysis and processing and action type identification. The purpose of anchoring the Gaussian subgraph to the global keyframe is to establish an accurate connection between the local and the overall action. The purpose of obtaining the three-dimensional action model is to fully describe the action in three dimensions. The purpose is to fully present the dynamic process of the action and provide support for multi-dimensional analysis.

[0079] In the actual evaluation, the global key frame is determined based on the second feature (key point) and the third feature (instantaneous deformation feature). In the knee flexion and extension movement, the frame when the knee joint angle reaches the maximum or minimum (the angle is calculated by the key point coordinates) and the standard deviation of the depth value in the local area around the knee joint reaches the local maximum value (instantaneous deformation feature) is determined as the global key frame;

[0080] In the actual evaluation, for each global keyframe, a Gaussian subgraph is constructed. For the global keyframe of knee flexion and extension, a Gaussian subgraph is constructed with the knee key point as the center. The mean of the Gaussian subgraph is the coordinate of the knee key point. The covariance matrix is ​​determined according to the distribution of key points around the knee joint, and the standard deviation in the horizontal direction is 20 pixels and the standard deviation in the vertical direction is 15 pixels. These Gaussian subgraphs are anchored to the corresponding global keyframes to form a three-dimensional action model of knee flexion and extension. This model contains multiple global keyframes and their corresponding Gaussian subgraphs, which can describe the key forms and changes of knee flexion and extension in three-dimensional space.

[0081] Specifically, in a typical knee flexion and extension movement, the frames at the 3rd, 7th, and 11th seconds are determined as global key frames, because the knee joint angles at these moments reach the maximum extension, maximum bending, and maximum extension again, respectively, and the standard deviation of the depth value around the knee joint also meets the conditions;

[0082] S300 performs pose estimation according to the first feature and the second feature, evaluates a feature map associated with the three-dimensional motion model at a key point position, and obtains a key point descriptor; and constructs an attention function based on a deviation between the key point descriptors;

[0083] It should be explained that the purpose of evaluating the feature map associated with the three-dimensional motion model at the key point position to obtain the key point descriptor is to obtain a description that can more accurately represent the key point features. By combining the relationship between the feature map related to the three-dimensional motion model and the key points, a unique feature representation can be extracted at a specific key point position to provide more representative data for subsequent processing and construct an attention function. Because the deviation can reflect the difference between different feature distributions, the attention function can focus on the more critical parts of the action understanding or analysis based on this difference. In tasks such as action analysis and recognition, the attention mechanism helps to increase the focus on important features, thereby improving the overall analysis and recognition effect;

[0084] In the actual evaluation, the posture estimation is performed based on the first feature (action feature) and the second feature (key point). In the knee flexion and extension action, the knee angle change rate (action feature) and the key point coordinates of the knee, hip and ankle joints (second feature) are used to determine the posture of the leg. By calculating the angle between the knee joint and the hip and ankle joints and the distance relationship between the key points, the position and posture of the leg in three-dimensional space are accurately described.

[0085] In the actual evaluation, for the key points of the knee joint, in the three-dimensional action model of the knee flexion and extension, the depth value and depth gradient feature map information near the position (circular area with a radius of 30 pixels) are evaluated according to the position of the key points of the knee joint. The average depth value in this area is 1.52 meters, and the depth gradient varies between -0.1 and 0.1 meters / pixel. This information is combined into a key point descriptor, which is a 50-dimensional vector that contains depth value statistics, depth gradient statistics, and the relationship with adjacent key points.

[0086] In the actual evaluation, in the analysis of the athlete's knee flexion and extension movements, the key point descriptors of the knee, hip and ankle joints of multiple frames were obtained. After calculating the deviation norm of all frame pairs, the normalized weights were obtained. After being weighted by the attention function, the information of other frames was integrated and the pose estimation was performed. Finally, a more accurate leg pose was obtained, the initial deviation was corrected, and the accuracy of leg pose estimation in knee flexion and extension movements was improved.

[0087] S400 constructs a semantic segmentation network based on the key point descriptor and the attention function, clusters the interactive actions, and uses the semantic segmentation network to learn feature representations of the interactive actions in the same cluster;

[0088] It should be explained that when constructing a semantic segmentation network, the key point descriptor contains information such as the key structure of the target, such as the characteristics of various parts of the human body in human action analysis, and the attention function can focus on important feature areas. The combination of the two to construct a semantic segmentation network can improve the accuracy of image pixel classification and enable the network to better distinguish different semantic areas, such as distinguishing between human and background pixels. Using the semantic segmentation network to learn the same cluster feature representation can mine the common features of the same cluster actions. This feature representation helps to better understand and process the inherent laws of the same type of interactive actions;

[0089] In the actual evaluation, a semantic segmentation network is constructed based on key point descriptors and attention functions. The network structure adopts an improved ResNet-50 network. The input layer receives the key point descriptors and maps them to a 128-dimensional vector space. The middle layer contains multiple residual blocks. Each residual block contains a convolution layer, a batch normalization layer, and a ReLU activation function. In the convolution layer, the convolution kernel size is 3×3 and the step size is 2. The attention function is used to weight the feature map of the middle layer. For example, the attention function is applied to the output feature map of the third residual block, and each element in the feature map is weighted according to the attention weight. The last layer is a fully connected layer, which outputs the semantic category of the rehabilitation training action (knee flexion and extension, elbow flexion and extension, hip abduction and adduction);

[0090] S500 obtains the user's interactive actions, maps the interactive actions to the cluster, determines whether to interact based on the size of the overlapping area in the cluster, and if interaction is performed, executes the interaction based on the semantic segmentation network.

[0091] It should be explained that the reason for judging whether an action is interactive based on the size of the overlapping area within a cluster is that the actions within a cluster have some common feature areas. If the user's interactive action has a large overlap with the actions within the cluster in these feature areas, it means that the action has a high similarity with the actions within the cluster. If the overlapping area is small, it is not an interactive action.

[0092] In the actual evaluation, the actions were clustered according to the semantic categories of the rehabilitation training actions. The 900 collected action samples (depth image sequences) were clustered according to the feature representation learned by the semantic segmentation network, using a hierarchical clustering algorithm with Euclidean distance as the distance metric. In the clustering process, the distance between the key point descriptors of each action sample was calculated. After clustering, three clusters were obtained, corresponding to knee flexion and extension, elbow flexion and extension, and hip abduction and adduction actions, with a clustering accuracy of 98.42% (compared with manual annotations by professional rehabilitation doctors);

[0093] In the actual evaluation, the user's interactive actions are obtained (for example, the patient is performing knee flexion and extension movements), and the user's interactive actions are mapped to the cluster. By calculating the Euclidean distance between the key point descriptor of the user action and the key point descriptor of the clustered action samples, the user action is mapped to the nearest cluster (knee flexion and extension cluster), and the overlapping area between the user action and the action in the cluster in the feature space is calculated. The specific calculation method is to count the intersection ratio of the key point descriptor of the user action and the key point descriptor of the action in the cluster in each dimension. If this intersection ratio is greater than 60% (the set threshold), it is judged as an interaction;

[0094] Specifically, when it is determined that the user is performing knee flexion and extension interaction, the semantic segmentation network gives the user feedback through voice prompts based on the user's action characteristics (such as whether the knee angle meets the standard, whether the action energy is normal, etc.). If the knee angle changes too fast or too slow, the voice prompts are: "Your knee flexion and extension speed is too fast, please slow down appropriately" and "Your knee flexion and extension speed is too slow, please speed up appropriately";

[0095] The second aspect of the present invention further provides a human-computer interaction implementation system based on visual perception, comprising:

[0096] A feature extraction module, used to obtain a depth image through a visual sensor, extract action features as a first feature, extract key points as a second feature, and extract instantaneous deformation features as a third feature from a target area of ​​the depth image;

[0097] A three-dimensional construction module, used to obtain a global key frame according to the second feature and the third feature, construct a Gaussian subgraph, anchor the Gaussian subgraph to the corresponding global key frame, and obtain a three-dimensional action model;

[0098] A feature conversion module, configured to perform pose estimation based on the first feature and the second feature, evaluate a feature map associated with the three-dimensional motion model at a key point position, and obtain a key point descriptor; and construct an attention function based on a deviation between the key point descriptors;

[0099] A semantic network module, used to construct a semantic segmentation network based on the key point descriptor and the attention function, cluster the interaction actions, and use the semantic segmentation network to learn the feature representation of the interaction actions in the same cluster;

[0100] The interaction execution module is used to obtain the user's interaction actions, map the interaction actions to the cluster, determine whether to interact based on the size of the overlapping area in the cluster, and if interaction is performed, execute the interaction based on the semantic segmentation network.

[0101] In this embodiment, the method of extracting action features as the first feature, extracting key points as the second feature, and extracting instantaneous deformation features as the third feature from the target area of ​​the depth image includes:

[0102] Preprocess the depth image, including noise reduction, illumination correction, histogram equalization and smoothing;

[0103] The target area is selected in the depth image in an interactive manner, the convolution kernel is slid in the target area for feature extraction, the action features are extracted using the optical flow algorithm, the key points are extracted using the HRNet network, and the instantaneous deformation features are extracted based on time series analysis.

[0104] In this embodiment, the method for obtaining a global key frame according to the second feature and the third feature includes:

[0105] The extracted second feature data and third feature data are normalized and aligned, and the second feature and the third feature are combined into a new feature vector by vector concatenation. Each key point has a corresponding instantaneous deformation feature. The new feature vector is clustered to obtain several clusters containing frames.

[0106] Calculate the degree of deviation of each frame relative to the cluster center. The expression is:

[0107]

[0108] Among them, ω1 and ω2 represent the weight coefficients for controlling the Mahalanobis distance and the degree of deviation of the Gaussian kernel function, respectively, x1 is the feature vector of the frame, μ is the feature vector of the cluster center, T represents the transposition operation, ∑ is the covariance matrix, and σ is the bandwidth parameter in the Gaussian kernel function;

[0109] The frames with the largest deviation are selected from each cluster and combined to form a global keyframe.

[0110] In this embodiment, the method of constructing a Gaussian subgraph, anchoring the Gaussian subgraph to the corresponding global key frame, and obtaining a three-dimensional motion model includes:

[0111] The key point coordinates and instantaneous deformation features in each global key frame are used as data points in the high-dimensional space. For each data point in the high-dimensional space, the kernel density estimate is calculated, and the expression is:

[0112]

[0113] Among them, m is the total number of data points, n is the dimension of the data points, ∑ is the covariance matrix, x2 is any data point in the high-dimensional space, and y i is the i-th data point in the high-dimensional space, and T represents the transposition operation;

[0114] Based on the kernel density estimation value, a numerical matrix representing the data distribution density is obtained as a Gaussian subgraph;

[0115] Construct a spatial transformation matrix according to the spatial changes of key points in the global key frame, and transform the constructed Gaussian subgraph according to the spatial transformation matrix;

[0116] The transformed Gaussian sub-image is merged with the global key frame through weighted averaging to form a 3D motion model.

[0117] In this embodiment, the method of performing pose estimation according to the first feature and the second feature, evaluating a feature map associated with the three-dimensional motion model at a key point position, and obtaining a key point descriptor includes:

[0118] A pose estimation model is constructed using a neural network model according to the first feature and the second feature, the action feature and the key point feature are used as input, and a preliminary pose estimation result is output;

[0119] Considering the depth image in time series, the preliminary pose estimation result of the current frame is fused with the preliminary pose estimation result of the previous frame by weighted averaging. The weight distribution is determined according to the reliability of the pose of the previous frame and the confidence of the current frame. The parameters of the pose estimation model are continuously updated until the processing of each frame of the depth image is completed.

[0120] According to the output results of the optimized pose estimation model, the positions of the key points in the three-dimensional motion model are determined, and the three-dimensional coordinates are converted into two-dimensional coordinates in the feature map;

[0121] In the feature map, the local area is defined with the position of the key point as the center. The size of the area is determined by the resolution of the feature map, and the expression is:

[0122]

[0123] Among them, α is the scale factor, W is the width of the feature map, and H is the height of the feature map;

[0124] Calculate the gradient features, texture features, and depth change features of the pixels in the local area, and connect them in the order of the gradient direction to form a high-dimensional key point descriptor, expressed as:

[0125]

[0126] Where D(ξ,σ1,θ) is the key point descriptor at position ξ, scale σ1 and gradient direction θ, N is the normalization factor, Ω(ξ,σ1) represents the local area centered on ξ and with scale σ1, u1 and v1 are the pixel position indexes in the local area, They are respectively represented as the image gradients at pixel positions u1 and v1, ⊙ represents the Hadanard product between vectors, h(·) is a nonlinear activation function, It is expressed as the inner product of the image gradient at pixel positions u1 and v1, which is used to extract texture features and depth change features. ω(ξ,σ1,θ) is a weighting function based on the gradient direction θ, which is used to emphasize the gradient consistent with the direction of the key point.

[0127] In this embodiment, the method for constructing an attention function based on the deviation between the key point descriptors includes:

[0128] The average value of the Euclidean distance between key point descriptors is calculated as the deviation, and the attention function is constructed based on the deviation. The expression is:

[0129]

[0130] Among them, α ij is the adjustment coefficient between the i-th and j-th key point descriptors, W Q , W K and W V They are respectively represented as the query vector, key vector and value vector of the key point descriptor, G i and G j They are respectively represented as the i-th and j-th key point descriptors, f(·) represents the exponential distance deviation adjustment function, φ is the nonlinear activation function sigmoid, β is the coefficient of the influence of the average deviation on attention, is the average value of the Euclidean distance between all key point descriptors, g(·) is the distance deviation norm function, softmax(·) is the normalized exponential function, softmax(tanh(W Q G i +W K G j )) ij It is represented as normalizing all attention outputs.

[0131] In this embodiment, the method for constructing a semantic segmentation network based on the key point descriptor and the attention function includes:

[0132] Use the deep learning framework to build a semantic segmentation network, including: input layer: take the key point descriptor as the input of the network; convolution layer: apply the convolution layer to extract the features of the input key point descriptor; pooling layer: insert the average pooling layer between the convolution layers to reduce the resolution of the feature map and increase the receptive field; attention mechanism fusion layer: apply the attention function to the output feature map of the convolution layer, and assign different weights to different feature areas according to the deviation information; semantic segmentation layer: upsampling layer, restore the feature map after attention weighting to the resolution corresponding to the input image; classification layer, add a fully connected layer at the end of the network to map the feature map to the semantic category; output layer, set the number of output nodes of the semantic segmentation network based on the number of instructions to be set;

[0133] The expression of the objective function of the semantic segmentation network is:

[0134]

[0135] in, is the total number of pixels in the image, ζ i is the category of the i-th pixel in the true label, To predict the category probability of the i-th pixel in the output, is the weight coefficient for balancing the cross entropy and regularization term, λ is the regularization coefficient, M is the number of channels in the feature map, K is the number of elements in each channel, and W A,jk is the attention weight matrix W A The value of the kth element of the jth channel in , is the mean attention weight of the jth channel;

[0136] Based on the attention function, a spatial branch is introduced into the semantic segmentation network. The expression of the spatial branch is:

[0137] m=F SG [σ4(F SM (σ2(F GP (W q )))×σ3(W v (X)))]

[0138] Among them, F SG is a fully connected layer, σ4 represents a nonlinear transformation of the output of the softmax operation, and F SM (·) is the softmax operator, σ2 represents the nonlinear transformation of the output of the global pooling operation, and F GP (·) is the global pooling operator, W q is the weight parameter of the convolution kernel, σ3 represents the nonlinear transformation of the output of the convolution operation, W v (·) is the convolution operator, X is the input feature map;

[0139] Obtain the data set, divide it into training set, validation set and test set in the ratio of 8:1:1, normalize it, and train the semantic segmentation network.

[0140] In this embodiment, the method of using the semantic segmentation network to learn feature representations of interactive actions in the same cluster includes:

[0141] The kernel density estimation method is used to calculate the probability density function of the interaction actions in the same cluster, and the expression is:

[0142]

[0143] in, is the estimated density function at point x3, n is the number of samples, d is the dimension of the data, x3 is the point in the d-dimensional space where the probability density needs to be estimated, and x i is the i-th d-dimensional sample point, H1 is the bandwidth matrix of the kernel function, and T represents the transposition operation. is the probability density estimation function that depends on the bandwidth matrix H1, X1 is the variable in the entire d-dimensional space, Expressed as constructing a probability density estimate When , the estimation function obtained from the i-th d-dimensional sample point is not used;

[0144] The kernel density estimate is used as the weight to perform weighted summation on the image features, allowing the semantic segmentation network to learn the distribution characteristics of the data within the cluster. At the same time, the convolutional layer in the semantic segmentation network is used to learn the characteristics of the cluster center vector and the covariance matrix within the cluster.

[0145] In this embodiment, the method of mapping the interactive action to the cluster and determining whether there is an interaction based on the size of the overlapping area within the cluster includes:

[0146] For each cluster, obtain the location of its cluster center. When the user's interactive action feature vector is obtained, calculate the distance d1 from the cluster center. Cross-sample action vectors at a ratio of 10% within the cluster as the action vector set, and calculate the distance d1 between each action vector in the set and the cluster center. si ;

[0147] Statistical distance d si The number of vectors m1 that is less than or equal to d1, if m1 exceeds half of the number of selected samples, it means that the interaction action is relatively central in the cluster and the size of the overlapping area exceeds the threshold, and it is determined to be interactive, otherwise, it is determined not to be interactive.

[0148] The above contents are merely examples and explanations of the structure of the present invention. The technicians in this technical field may make various modifications or additions to the specific embodiments described or replace them in a similar manner. As long as they do not deviate from the structure of the invention or exceed the scope defined by the claims, they should all fall within the protection scope of the present invention.

Claims

1. A method for implementing human-computer interaction based on visual perception, characterized in that: The following steps are involved: Acquire a depth image through a visual sensor, extract action features as a first feature, extract key points as a second feature, and extract instantaneous deformation features as a third feature from a target area of ​​the depth image; Obtain a global key frame according to the second feature and the third feature, construct a Gaussian subgraph, anchor the Gaussian subgraph to the corresponding global key frame, and obtain a three-dimensional action model; Performing pose estimation based on the first feature and the second feature, evaluating a feature map associated with the three-dimensional motion model at a key point position, and obtaining a key point descriptor; Constructing an attention function based on the deviations between the key point descriptors; Building a semantic segmentation network based on the key point descriptor and the attention function, clustering the interactive actions, and using the semantic segmentation network to learn feature representations of the interactive actions in the same cluster; The user's interactive actions are obtained, the interactive actions are mapped to the clusters, and whether there is interaction is determined based on the size of the overlapping area in the clusters. If there is interaction, the interaction is performed based on the semantic segmentation network.

2. The method for realizing human-computer interaction based on visual perception according to claim 1, characterized in that: The method of extracting action features as the first feature, extracting key points as the second feature, and extracting instantaneous deformation features as the third feature from the target area of ​​the depth image includes: Preprocess the depth image, including noise reduction, illumination correction, histogram equalization and smoothing; The target area is selected in the depth image in an interactive manner, the convolution kernel is slid in the target area for feature extraction, the action features are extracted using the optical flow algorithm, the key points are extracted using the HRNet network, and the instantaneous deformation features are extracted based on time series analysis.

3. The method for realizing human-computer interaction based on visual perception according to claim 1, characterized in that: The method for obtaining a global key frame according to the second feature and the third feature includes: The extracted second feature data and third feature data are normalized and aligned, and the second feature and the third feature are combined into a new feature vector by vector concatenation. Each key point has a corresponding instantaneous deformation feature. The new feature vector is clustered to obtain several clusters containing frames. Calculate the degree of deviation of each frame relative to the cluster center. The expression is: Among them, ω1 and ω2 represent the weight coefficients for controlling the Mahalanobis distance and the degree of deviation of the Gaussian kernel function, respectively, x1 is the feature vector of the frame, μ is the feature vector of the cluster center, T represents the transposition operation, ∑ is the covariance matrix, and σ is the bandwidth parameter in the Gaussian kernel function; The frames with the largest deviation are selected from each cluster and combined to form a global keyframe.

4. The method for realizing human-computer interaction based on visual perception according to claim 1, characterized in that: The method of constructing a Gaussian subgraph, anchoring the Gaussian subgraph to the corresponding global key frame, and obtaining a three-dimensional motion model includes: The key point coordinates and instantaneous deformation features in each global key frame are used as data points in the high-dimensional space. For each data point in the high-dimensional space, the kernel density estimation value is calculated, and the expression is: Among them, m is the total number of data points, n is the dimension of the data points, ∑ is the covariance matrix, x2 is any data point in the high-dimensional space, and y i is the i-th data point in the high-dimensional space, and T represents the transposition operation; Based on the kernel density estimation value, a numerical matrix representing the data distribution density is obtained as a Gaussian subgraph; Construct a spatial transformation matrix according to the spatial changes of key points in the global key frame, and transform the constructed Gaussian subgraph according to the spatial transformation matrix; The transformed Gaussian sub-image is merged with the global key frame through weighted averaging to form a 3D motion model.

5. The method for realizing human-computer interaction based on visual perception according to claim 1, characterized in that: The method of performing pose estimation according to the first feature and the second feature, evaluating a feature map associated with the three-dimensional motion model at a key point position, and obtaining a key point descriptor comprises: A pose estimation model is constructed using a neural network model according to the first feature and the second feature, the action feature and the key point feature are used as input, and a preliminary pose estimation result is output; Considering the depth image in time series, the preliminary pose estimation result of the current frame is fused with the preliminary pose estimation result of the previous frame by weighted averaging. The weight distribution is determined according to the reliability of the pose of the previous frame and the confidence of the current frame. The parameters of the pose estimation model are continuously updated until the processing of each frame of the depth image is completed. According to the output results of the optimized pose estimation model, the positions of the key points in the three-dimensional motion model are determined, and the three-dimensional coordinates are converted into two-dimensional coordinates in the feature map; In the feature map, the local area is defined with the position of the key point as the center. The size of the area is determined by the resolution of the feature map, and the expression is: Among them, α is the scale factor, W is the width of the feature map, and H is the height of the feature map; Calculate the gradient features, texture features, and depth change features of the pixels in the local area, and connect them in the order of the gradient direction to form a high-dimensional key point descriptor, expressed as: Where D(ξ,σ1,θ) is the key point descriptor at position ξ, scale σ1 and gradient direction θ, N is the normalization factor, Ω(ξ,σ1) represents the local area centered on ξ and with scale σ1, u1 and v1 are the pixel position indexes in the local area, They are respectively represented as the image gradients at pixel positions u1 and v1, ⊙ represents the Hadanard product between vectors, h(·) is a nonlinear activation function, It is expressed as the inner product of the image gradient at pixel positions u1 and v1, which is used to extract texture features and depth change features. ω(ξ,σ1,θ) is a weighting function based on the gradient direction θ, which is used to emphasize the gradient consistent with the direction of the key point.

6. The method for realizing human-computer interaction based on visual perception according to claim 1, characterized in that: The method for constructing an attention function based on the deviation between the key point descriptors includes: The average value of the Euclidean distance between the key point descriptors is calculated as the deviation, and the attention function is constructed based on the deviation. The expression is: Among them, α ij is the adjustment coefficient between the i-th and j-th key point descriptors, W Q , W K and W V They are represented as the query vector, key vector and value vector of the key point descriptor, G i and G j They are respectively represented as the i-th and j-th key point descriptors, f(·) represents the exponential distance deviation adjustment function, φ is the nonlinear activation function sigmoid, β is the coefficient of the influence of the average deviation on attention, is the average value of the Euclidean distance between all key point descriptors, g(·) is the distance deviation norm function, softmax(·) is the normalized exponential function, softmax(tanh(W Q G i +W K G j )) ij It is represented as normalizing all attention outputs.

7. The method for realizing human-computer interaction based on visual perception according to claim 1, characterized in that: The method for constructing a semantic segmentation network based on the key point descriptor and the attention function comprises: Use the deep learning framework to build a semantic segmentation network, including: input layer: take the key point descriptor as the input of the network; convolution layer: apply the convolution layer to extract the features of the input key point descriptor; pooling layer: insert the average pooling layer between the convolution layers to reduce the resolution of the feature map and increase the receptive field; attention mechanism fusion layer: apply the attention function to the output feature map of the convolution layer, and assign different weights to different feature areas according to the deviation information; semantic segmentation layer: upsampling layer, restore the feature map after attention weighting to the resolution corresponding to the input image; classification layer, add a fully connected layer at the end of the network to map the feature map to the semantic category; output layer, set the number of output nodes of the semantic segmentation network based on the number of instructions to be set; The expression of the objective function of the semantic segmentation network is: in, is the total number of pixels in the image, ζ i is the category of the i-th pixel in the true label, To predict the category probability of the i-th pixel in the output, is the weight coefficient for balancing the cross entropy and regularization term, λ is the regularization coefficient, M is the number of channels in the feature map, K is the number of elements in each channel, and W A,jk is the attention weight matrix W A The value of the kth element of the jth channel in , is the mean attention weight of the jth channel; Based on the attention function, a spatial branch is introduced into the semantic segmentation network. The expression of the spatial branch is: m=F SG [σ4(F SM (σ2(F GP (W q )))×σ3(W v (X)))] Among them, F SG is a fully connected layer, σ4 represents a nonlinear transformation of the output of the softmax operation, and F SM (·) is the softmax operator, σ2 represents the nonlinear transformation of the output of the global pooling operation, and F GP (·) is the global pooling operator, W q is the weight parameter of the convolution kernel, σ3 represents the nonlinear transformation of the output of the convolution operation, W v (·) is the convolution operator, X is the input feature map; Obtain the data set, divide it into training set, validation set and test set in the ratio of 8:1:1, normalize it, and train the semantic segmentation network.

8. The method for realizing human-computer interaction based on visual perception according to claim 1, characterized in that: The method of using the semantic segmentation network to learn feature representations of interactive actions in the same cluster includes: The kernel density estimation method is used to calculate the probability density function of the interaction actions in the same cluster, and the expression is: in, is the estimated density function at point x3, n is the number of samples, d is the dimension of the data, x3 is the point in the d-dimensional space where the probability density needs to be estimated, and x i is the i-th d-dimensional sample point, H1 is the bandwidth matrix of the kernel function, and T represents the transposition operation. is the probability density estimation function that depends on the bandwidth matrix H1, X1 is the variable in the entire d-dimensional space, Expressed as constructing a probability density estimate When , the estimation function obtained from the i-th d-dimensional sample point is not used; The kernel density estimate is used as the weight to perform weighted summation on the image features, allowing the semantic segmentation network to learn the distribution characteristics of the data within the cluster. At the same time, the convolutional layer in the semantic segmentation network is used to learn the characteristics of the cluster center vector and the covariance matrix within the cluster.

9. The method for realizing human-computer interaction based on visual perception according to claim 1, characterized in that: The method of mapping the interactive action to the cluster and determining whether there is an interaction based on the size of the overlapping area in the cluster includes: For each cluster, obtain the location of its cluster center. When the user's interactive action feature vector is obtained, calculate the distance d1 from the cluster center. Cross-sample action vectors at a ratio of 10% within the cluster as the action vector set, and calculate the distance d1 between each action vector in the set and the cluster center. si ; Statistical distance d si The number of vectors m1 that is less than or equal to d1, if m1 exceeds half of the number of selected samples, it means that the interaction action is relatively central in the cluster and the size of the overlapping area exceeds the threshold, and it is determined to be interactive, otherwise, it is determined not to be interactive.

10. A human-computer interaction implementation system based on visual perception is used to execute a human-computer interaction implementation method based on visual perception according to claims 1 to 9, characterized in that: The system comprises: A feature extraction module, used to obtain a depth image through a visual sensor, extract action features as a first feature, extract key points as a second feature, and extract instantaneous deformation features as a third feature from a target area of ​​the depth image; A three-dimensional construction module, used to obtain a global key frame according to the second feature and the third feature, construct a Gaussian subgraph, anchor the Gaussian subgraph to the corresponding global key frame, and obtain a three-dimensional action model; A feature conversion module, configured to perform pose estimation based on the first feature and the second feature, evaluate a feature map associated with the three-dimensional motion model at a key point position, and obtain a key point descriptor; and construct an attention function based on a deviation between the key point descriptors; A semantic network module, used to construct a semantic segmentation network based on the key point descriptor and the attention function, cluster the interaction actions, and use the semantic segmentation network to learn the feature representation of the interaction actions in the same cluster; The interaction execution module is used to obtain the user's interaction actions, map the interaction actions to the cluster, determine whether to interact based on the size of the overlapping area in the cluster, and if interaction is performed, execute the interaction based on the semantic segmentation network.

Images