A method and system for realizing human-computer interaction based on visual perception

Abstract

The application discloses a human-computer interaction implementation method and system based on visual perception, comprising the following steps: extracting motion features from a target region of a depth image as first features, extracting key points as second features, and extracting instantaneous deformation features as third features; obtaining a three-dimensional motion model according to the second features and the third features; performing pose estimation according to the first features and the second features to obtain key point descriptors; constructing an attention function based on the deviation between the key point descriptors; constructing a semantic segmentation network based on the key point descriptors and the attention function, learning the features of interactive actions by using the semantic segmentation network; judging whether to interact based on the size of the overlapping region in the cluster, and if the interaction is to be performed, performing the interaction based on the semantic segmentation network. The method improves the accurate recognition rate of the interaction, has good explainability, and can be directly applied to a human-computer interaction system.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of visual perception, and in particular to a human-computer interaction implementation method and system based on visual perception. BACKGROUND

[0002] With the development of technology and the increasing demand for user experience, people expect a more natural and intuitive human-computer interaction mode, directly interacting with the computer system through natural ways such as body movements, gestures, and expressions. This natural interaction mode can improve interaction efficiency, reduce user learning costs, and provide convenient interaction experience in more scenarios.

[0003] Traditional human-computer interaction implementation methods often rely on command line interaction and model matching, and most traditional methods lack deep understanding of scene semantic information, and can only handle some pre-set and fixed interaction modes.

[0004] In today's digital age, human-computer interaction plays a key role in many fields. Therefore, a human-computer interaction implementation method and system based on visual perception are needed to accurately identify user intent, make interaction more natural and efficient, and improve the overall experience of users in different scenarios. SUMMARY

[0005] The purpose of the present application is to provide a human-computer interaction implementation method and system based on visual perception.

[0006] To achieve the above purpose, the present application is implemented according to the following technical solutions:

[0007] The present application provides a human-computer interaction implementation method based on visual perception, comprising:

[0008] S100 obtains a depth image through a visual sensor, extracts motion features from a target region of the depth image as first features, extracts key points as second features, and extracts instantaneous deformation features as third features;

[0009] S200 obtains a global key frame according to the second features and the third features, constructs a Gaussian subgraph, anchors the Gaussian subgraph to the corresponding global key frame, and obtains a three-dimensional motion model;

[0010] S300 performs pose estimation according to the first features and the second features, evaluates a feature map associated with the three-dimensional motion model at the key point position, and obtains a key point descriptor; and constructs an attention function based on the deviation between the key point descriptors;

[0011] S400 constructing a semantic segmentation network based on the key point descriptor and the attention function, clustering the interaction actions, and learning a feature representation of the interaction actions in the same cluster using the semantic segmentation network;

[0012] S500 obtaining the interaction actions of the user, mapping the interaction actions to the clusters, determining whether to interact based on a size of an overlapping area in the cluster, and performing the interaction based on the semantic segmentation network if the interaction is performed.

[0013] As a further method, the method of extracting an action feature from a target region of a depth image as a first feature, extracting a key point as a second feature, and extracting a transient deformation feature as a third feature includes:

[0014] The depth image is preprocessed, including denoising, illumination correction, histogram equalization, and smoothing processing.

[0015] A target region is selected in the depth image based on an interaction mode, a convolution kernel is slid in the target region to extract features, an action feature is extracted using an optical flow algorithm, a key point is extracted using an HRNet network, and a transient deformation feature is extracted based on time series analysis.

[0016] As a further method, the method of obtaining a global key frame based on the second feature and the third feature includes:

[0017] The extracted second feature data and third feature data are normalized and data-aligned, the second feature and the third feature are combined into a new feature vector using a vector splicing method, each key point has a corresponding transient deformation feature, the new feature vector is clustered, and a plurality of clusters containing frames are obtained.

[0018] The deviation degree of each frame relative to the cluster center is calculated, and the expression is:

[0019]

[0020] wherein, and and represent weight coefficients for controlling the Mahalanobis distance and the Gaussian kernel deviation degree, respectively, is a feature vector of the frame, is a feature vector of the cluster center, represents a transposition operation, is a covariance matrix, is a bandwidth parameter in the Gaussian kernel function;

[0021] The frame with the largest deviation degree is selected in each cluster, and the global key frame is formed by combining the frames.

[0022] As a further method, the method of constructing a Gaussian subgraph and anchoring the Gaussian subgraph to the corresponding global keyframe to obtain a 3D motion model includes:

[0023] The keypoint coordinates and instantaneous deformation features in each global keyframe are used as data points in a high-dimensional space. For each data point in the high-dimensional space, the kernel density estimate is calculated, expressed as:

[0024]

[0025] in, The total number of data points. For the dimensions of the data points, Let covariance matrix be the variance matrix. For any data point in high-dimensional space, For the first in high-dimensional space Data points, This indicates a transpose operation;

[0026] A numerical matrix representing the data distribution density is obtained based on the kernel density estimation, which serves as a Gaussian subgraph;

[0027] A spatial transformation matrix is ​​constructed based on the spatial changes of key points in the global keyframe, and the constructed Gaussian subgraph is transformed according to the spatial transformation matrix.

[0028] The transformed Gaussian subgraph is then merged with the global keyframes using a weighted average to form a 3D motion model.

[0029] As a further method, the method of performing pose estimation based on the first feature and the second feature, evaluating the feature map associated with the 3D motion model at the key point location, and obtaining key point descriptors includes:

[0030] Based on the first and second features, a pose estimation model is constructed using a neural network model. The action features and key point features are taken as inputs, and the preliminary pose estimation results are output.

[0031] Considering depth images in a time series, the preliminary pose estimation results of the current frame are fused with the preliminary pose estimation results of the previous frame by weighted averaging. The weight allocation is determined based on the reliability of the pose of the previous frame and the confidence of the current frame. The parameters of the pose estimation model are continuously updated until the processing of each frame of depth image is completed.

[0032] Based on the output of the optimized pose estimation model, the positions of key points in the 3D motion model are determined, and the 3D coordinates are converted into 2D coordinates in the feature map.

[0033] In the feature map, a local region is defined with the position of the key point as the center, and the region size is determined according to the resolution of the feature map, and the expression is:

[0034]

[0035] wherein, is a proportional coefficient, is the width of the feature map, is the height of the feature map;

[0036] The gradient feature, texture feature and depth change feature of the pixels in the local region are calculated, and they are connected in the order of the gradient direction to form a high-dimensional key point descriptor, and the expression is:

[0037]

[0038] wherein, is the key point descriptor at the position , the scale and the gradient direction , is a normalization factor, represents a local region with as the center and the scale , and are the pixel position indexes in the local region, respectively represent the image gradients at the pixel positions and , represents the Hadanard product between the vectors, is a nonlinear activation function, represents the inner product of the image gradients at the pixel positions and , which is used to extract the texture feature and the depth change feature, is a weighting function based on the gradient direction , which is used to emphasize the gradient consistent with the direction of the key point.

[0039] As a further method, the method of constructing the attention function based on the deviation between the key point descriptors comprises:

[0040] The average value of the Euclidean distance between the key point descriptors is calculated as the deviation, and the attention function is constructed based on the deviation, and the expression is:

[0041]

[0042] wherein, is the th an adjustment coefficient between key point descriptors, 、 and are denoted as query vector, key vector and value vector of key point descriptors respectively, and are denoted as the first and the second key point descriptors respectively, is denoted as an exponential distance deviation adjustment function, is a nonlinear activation function sigmoid, is a coefficient of the influence degree of average deviation on attention, is the average value of the Euclidean distance between all key point descriptors, is denoted as a distance deviation norm function, is a normalized exponential function, is denoted as a normalization processing on all attention outputs.

[0043] As a further method, the method of constructing a semantic segmentation network based on the key point descriptors and the attention function comprises:

[0044] The semantic segmentation network is constructed using a deep learning framework, comprising: an input layer: the key point descriptors are taken as the input of the network; a convolutional layer: the convolutional layer is applied to feature extraction of the input key point descriptors; a pooling layer: an average pooling layer is inserted between the convolutional layers to reduce the resolution of the feature map and increase the receptive field; an attention mechanism fusion layer: the attention function is applied to the output feature map of the convolutional layer, and different weights are assigned to different feature regions according to the deviation information; a semantic segmentation layer: an up-sampling layer, the feature map after attention weighting is restored to the resolution corresponding to the input image; a classification layer, a fully connected layer is added at the end of the network, and the feature map is mapped to a semantic category; an output layer, the number of output nodes of the semantic segmentation network is set based on the number of instructions to be set;

[0045] The expression of the objective function of the semantic segmentation network is:

[0046]

[0047] wherein, is the total number of pixels in the image, is the class of the pixel point in the real label, is the class probability of the pixel point in the predicted output, is a weight coefficient for balancing cross-entropy and regularization term, is a regularization coefficient, is the number of channels of the feature map, The number of elements in each channel Attention weight matrix The Middle The first channel The value of each element, For the first The average attention weight of each channel;

[0048] The attention function introduces a spatial branch into the semantic segmentation network. The expression for the spatial branch is:

[0049]

[0050] in, It is a fully connected layer. This represents a nonlinear transformation of the output of the softmax operation. This is the softmax operator. This represents a nonlinear transformation of the output of the global pooling operation. This is the global pooling operator. The weight parameters of the convolution kernel, This represents a nonlinear transformation of the output of the convolution operation. For convolution operators, The input feature map;

[0051] Obtain the dataset, divide it into training, validation and test sets in an 8:1:1 ratio, normalize it, and then train the semantic segmentation network.

[0052] As a further method, the method of learning feature representations of interaction actions in the same cluster using the semantic segmentation network includes:

[0053] The probability density function of interaction actions within the same cluster is calculated using the kernel density estimation method, and its expression is:

[0054]

[0055] in, For the point The estimated density function at that location, For the number of samples, For the dimensions of the data, In order to be in Points in a dimensional space for which probability density needs to be estimated For the first d-dimensional sample points, The bandwidth matrix of the kernel function. This indicates that a transpose operation is being performed. For bandwidth matrix dependent The probability density estimation function, For the whole Variables in 3D space This is expressed as in constructing probability density estimates When, do not use the first The estimated function obtained from d-dimensional sample points;

[0056] The kernel density estimate is used as a weight to perform a weighted summation of image features, allowing the semantic segmentation network to learn the distribution characteristics of data within clusters. At the same time, the convolutional layers in the semantic segmentation network are used to learn the features of the cluster center vector and the covariance matrix within the cluster.

[0057] As a further method, the method of mapping interactive actions to clusters and determining whether an interaction has occurred based on the size of the overlapping region within the cluster includes:

[0058] For each cluster, obtain its cluster center location. When the user's interaction action feature vector is obtained, calculate the distance to the cluster center. Within each cluster, action vectors are cross-sampled at a ratio of 10% to form an action vector set. The distance between each action vector in the set and the cluster center is then calculated. ;

[0059] Statistical distance Less than or equal to number of vectors ,like If the number of samples exceeds half of the selected sample size, it indicates that the interaction action is relatively central within the cluster and the size of the overlapping area exceeds the threshold, and an interaction is determined to be performed; otherwise, no interaction is determined to be performed.

[0060] A second aspect of the present invention provides a human-computer interaction system based on visual perception, comprising:

[0061] The feature extraction module is used to acquire depth images through a visual sensor, and extract motion features as the first feature, key points as the second feature, and instantaneous deformation features as the third feature from the target area of ​​the depth image.

[0062] A 3D construction module is used to obtain global keyframes based on the second feature and the third feature, construct a Gaussian subgraph, anchor the Gaussian subgraph to the corresponding global keyframe, and obtain a 3D motion model.

[0063] The feature transformation module is used to perform pose estimation based on the first feature and the second feature, evaluate the feature map associated with the 3D motion model at the key point location, and obtain key point descriptors; and construct an attention function based on the deviation between the key point descriptors.

[0064] a semantic network module configured to construct a semantic segmentation network based on the key point descriptor and the attention function, cluster the interactive actions, and learn feature representations of the interactive actions in the same cluster using the semantic segmentation network;

[0065] an interaction execution module configured to obtain the interactive actions of the user, map the interactive actions to the clusters, determine whether to interact based on a size of an overlapping area in the cluster, and execute the interaction based on the semantic segmentation network if the interaction is to be performed.

[0066] Compared with the prior art, the embodiments of the present application have at least the following advantages or beneficial effects:

[0067] The present application can construct a three-dimensional action model that can reflect the spatial structure of the interactive actions in the depth image and embody the data distribution characteristics through the Gaussian subgraph by using the depth information of the key frame, and the model can be used to describe the state of the object in the three-dimensional space, such as the action and the posture.

[0068] The present application can improve the accuracy and efficiency of detection by obtaining the first Gaussian subgraph and the second Gaussian subgraph and constructing the attention function based on the deviation between the Gaussian subgraphs, and the spatial attention mechanism enables the model to focus on the key areas in the image and ignore irrelevant areas.

[0069] The present application can improve the accuracy and efficiency of detection by obtaining the first Gaussian subgraph and the second Gaussian subgraph and constructing the attention function based on the deviation between the Gaussian subgraphs, and the spatial attention mechanism enables the model to focus on the key areas in the image and ignore irrelevant areas. BRIEF DESCRIPTION OF DRAWINGS

[0070] Figure 1 A flowchart of the steps of the method for realizing human-computer interaction based on visual perception in the embodiments of the present application. DETAILED DESCRIPTION

[0071] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, and not all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.

[0072] Referring to Figure 1 The present application provides a method for realizing human-computer interaction based on visual perception, which comprises:

[0073] S100 obtains a depth image through a visual sensor, extracts a motion feature as a first feature, extracts a key point as a second feature, and extracts a transient deformation feature as a third feature from a target region of the depth image;

[0074] In one embodiment, the application is applied to the field of vision-based rehabilitation training assistance, and is used for recognizing, evaluating and interactively feeding back rehabilitation actions of a patient;

[0075] In actual evaluation, an Intel RealSense D435 depth camera is used as a visual sensor, the depth image resolution of the camera is 1280x720 pixels, the field of view angle is 87° horizontally and 58° vertically, and the frame rate can be set to 30fps or 60fps. In this embodiment, the frame rate is set to 30fps to obtain a depth image sequence. Twenty rehabilitation patients (aged 25-65 years old, including patients with various physical conditions and rehabilitation stages) are selected to collect rehabilitation training action data. The rehabilitation training actions include knee joint flexion and extension, elbow joint flexion and extension and hip joint abduction and adduction. Each patient demonstrates each action for 15 times, and each demonstration lasts for 10-15 seconds. A total of 900 depth image sequences (20 patients x 3 actions x 15 repetitions) are collected;

[0076] In the actual evaluation, for the target region of each depth image (i.e. the region where the human-related joint parts are located), multiple features are obtained, for the knee flexion and extension action, the knee angle change rate is calculated, the key points of the knee joint are identified through the optical flow algorithm, the angle change rate is calculated according to the difference value of the knee joint angle between adjacent frames divided by the time interval, in the collected data, the knee angle change rate of the knee flexion and extension action is in the range of 0.1-0.5 rad / s, and the average angle change rate is about 0.3 rad / s; meanwhile, the energy feature of the action is calculated, the change of the depth value in the target region is analyzed, the change of the depth value is converted into the energy value, for a knee flexion and extension action, the energy value fluctuates between 50-200 joules in the whole action process, and the average energy value is 120 joules, which are combined as the first feature. The HRNet pose estimation model is used to extract the key points of the human body from the depth image, for the knee flexion and extension action, the key points of the knee joint, hip joint and ankle joint are focused on, for the elbow flexion and extension action, the key points of the elbow joint, shoulder joint and wrist joint are focused on, and for the hip abduction and adduction action, the key points of the hip joint, knee joint and waist are focused on, taking a depth image of a knee flexion and extension action as an example, the knee joint key point coordinates are (450, 320), the hip joint key point coordinates are (400, 280), and the ankle joint key point coordinates are (480, 380), and the depth values of each key point are recorded, the knee joint key point depth value is 1.5 meters, the hip joint key point depth value is 1.4 meters, and the ankle joint key point depth value is 1.6 meters, and the extraction of the second feature is completed. The instantaneous deformation feature is calculated for the local region around each joint part, in the knee flexion and extension action, the knee joint is taken as the center, a circular region with a radius of 50 pixels is selected as the local region, and the standard deviation of the depth value in the region is calculated as the instantaneous deformation feature, in the action process, the standard deviation changes between 0.05-0.2 meters, and when the knee joint is bent to the maximum angle, the standard deviation reaches the maximum value of 0.2 meters, for the elbow flexion and extension action, the elbow joint is taken as the center, a circular region with a radius of 30 pixels, and the depth value standard deviation thereof is between 0.03-0.15 meters, and the average is 0.08 meters, and the extraction of the third feature is completed.

[0077] S200 obtains a global key frame according to the second feature and the third feature, constructs a Gaussian subgraph, anchors the Gaussian subgraph to the global key frame corresponding to the Gaussian subgraph, and obtains a three-dimensional action model;

[0078] It needs to be explained that the purpose of obtaining global key frames is to filter out key parts of the motion process from a large number of frame data, which is useful for reducing data volume and summarizing motion characteristics, facilitating subsequent analysis and motion type recognition. The purpose of anchoring the Gaussian subgraph to the global key frame is to accurately link the local and global motions, and the purpose of obtaining the three-dimensional motion model is to comprehensively and stereoscopically describe the motion, which is useful for complete presentation of the motion dynamic process and support for multi-dimensional analysis.

[0079] In actual evaluation, the global key frames are determined according to the second feature (key point) and the third feature (instantaneous deformation feature). In the knee flexion and extension motion, the frames in which the knee angle reaches the maximum or minimum (the angle is calculated by the key point coordinates) and the standard deviation of the depth value of the local area around the knee reaches the local maximum (instantaneous deformation feature) are determined as the global key frames.

[0080] In actual evaluation, for each global key frame, a Gaussian subgraph is constructed. For the global key frames of the knee flexion and extension motion, a Gaussian subgraph is constructed with the knee key point as the center. The mean of the Gaussian subgraph is the knee key point coordinates, and the covariance matrix is determined according to the distribution of the key points around the knee. The standard deviation in the horizontal direction is 20 pixels, and the standard deviation in the vertical direction is 15 pixels. These Gaussian subgraphs are anchored to the corresponding global key frames to form a three-dimensional motion model of the knee flexion and extension motion. This model contains multiple global key frames and their corresponding Gaussian subgraphs, and can describe the key shapes and changes of the knee flexion and extension motion in three-dimensional space.

[0081] Specifically, in a typical knee flexion and extension motion, the frames at the 3rd, 7th and 11th seconds are determined as global key frames, because at these times the knee angle reaches the maximum extension, maximum bending and maximum extension again, and at the same time the standard deviation of the depth value around the knee also meets the conditions.

[0082] S300 performs pose estimation according to the first feature and the second feature, evaluates the feature map associated with the three-dimensional motion model at the key point position, and obtains a key point descriptor; an attention function is constructed based on the deviation between the key point descriptors;

[0083] It needs to be explained that the feature map associated with the three-dimensional action model is evaluated at the key point position to obtain the key point descriptor, the purpose is to obtain a more accurate representation of the key point feature description. By combining the relationship between the three-dimensional action model related feature map and the key point, a unique feature representation can be extracted at a specific key point position, providing more representative data for subsequent processing, constructing an attention function, because the bias can reflect the difference between different feature distributions, and the attention function can focus on the more critical part of the action understanding or analysis. In the action analysis, recognition and other tasks, the attention mechanism helps to improve the attention to important features, thereby improving the overall analysis and recognition effect;

[0084] In actual evaluation, according to the first feature (action feature) and the second feature (key point), the pose estimation is carried out, in the knee flexion and extension action, the knee angle change rate (action feature) and the knee joint, hip joint, ankle joint key point coordinates (second feature) are used to determine the leg pose, by calculating the included angle of the line connecting the knee joint and the hip joint, ankle joint and the distance relationship between the key points, the position and attitude of the leg in three-dimensional space are accurately described;

[0085] In actual evaluation, for the knee key point, in the three-dimensional action model of the knee flexion and extension action, according to the position of the knee key point, the depth value, depth gradient feature map information in the vicinity of the position (a circular area with a radius of 30 pixels) is evaluated, the average value of the depth value in this area is 1.52 meters, and the depth gradient changes between -0.1-0.1 meters / pixel. These information is combined into a key point descriptor, which is a 50-dimensional vector containing depth value statistical information, depth gradient statistical information and relationship with adjacent key points;

[0086] In actual evaluation, in the analysis of the knee flexion and extension action of the athlete, the knee joint, hip joint and ankle joint key point descriptors of multiple frames are obtained, the normalized weight is obtained after calculating the deviation norm of all frames, and the other frame information is integrated after weighted by the attention function, and then the pose estimation is carried out, and finally the more accurate leg pose is obtained, the initial deviation is corrected, and the accuracy of the leg pose estimation in the knee flexion and extension action is improved;

[0087] S400 constructs a semantic segmentation network based on the key point descriptor and the attention function, clusters the interactive actions, and learns the feature representation of the interactive actions in the same cluster using the semantic segmentation network;

[0088] It needs to be explained that when constructing the semantic segmentation network, the key point descriptor contains the information of the target key structure, such as the features of each part of the human body in human motion analysis, and the attention function can focus on the important feature area. The combination of the two can improve the accuracy of image pixel classification and make the network better distinguish different semantic areas, such as distinguishing human body and background pixels. Using the semantic segmentation network to learn the same cluster feature representation can mine the common features of the same cluster motion. This feature representation is helpful for better understanding and processing the internal rules of the same type of interactive motion.

[0089] In actual evaluation, based on the key point descriptor and the attention function, a semantic segmentation network is constructed, and the network structure adopts an improved ResNet-50 network. The input layer receives the key point descriptor and maps it to a 128-dimensional vector space. The middle layer includes multiple residual blocks, each of which includes a convolution layer, a batch normalization layer and a ReLU activation function. In the convolution layer, the convolution kernel size is 3x3 and the step is 2. The attention function is used to weight the feature maps in the middle layer. For example, the attention function is applied to the output feature map of the third residual block, and each element in the feature map is weighted according to the attention weight. The last layer is a fully connected layer, which outputs the semantic category of the rehabilitation training motion (knee flexion and extension, elbow flexion and extension, hip abduction and adduction).

[0090] S500 obtains the interactive motion of the user, maps the interactive motion to the cluster, and determines whether the interaction is based on the size of the overlapping area in the cluster. If the interaction is performed, the interaction is performed based on the semantic segmentation network.

[0091] It needs to be explained that whether the interaction is based on the size of the overlapping area in the cluster is because the actions in the cluster have some common feature areas. If the user's interactive motion has a large overlap with the actions in the cluster in these feature areas, it means that the motion has a high similarity with the actions in the cluster. If the overlapping area is small, it is not an interactive motion.

[0092] In actual evaluation, the actions are clustered according to the semantic category of the rehabilitation training motion. The 900 action samples (depth image sequences) collected are clustered according to the feature representation learned by the semantic segmentation network, and the hierarchical clustering algorithm is used with Euclidean distance as the distance measure. In the clustering process, the distance between the key point descriptors of each action sample is calculated. After clustering, three clustering clusters are obtained, corresponding to the knee flexion and extension, elbow flexion and extension, and hip abduction and adduction motions, respectively. The clustering accuracy reaches 98.42% (compared with the artificial annotation of professional rehabilitation doctors);

[0093] In the actual evaluation, the interactive action of the user (for example, the patient is performing a knee flexion and extension action) is obtained, the interactive action of the user is mapped into a cluster, the user action is mapped to the nearest clustering cluster (knee flexion and extension cluster) by calculating the Euclidean distance between the key point descriptors of the user action and the key point descriptors of the clustered action samples, and the overlapping area of the user action and the action in the cluster in the feature space is calculated. The specific calculation method is to count the intersection ratio of the key point descriptors of the user action and the key point descriptors of the actions in the cluster in each dimension. If the intersection ratio is greater than 60% (a set threshold), it is judged that the interaction is performed.

[0094] Specifically, when it is judged that the user is performing a knee flexion and extension interaction, the semantic segmentation network gives feedback to the user through voice prompts according to the user action features (such as whether the knee angle meets the standard, whether the action energy is normal, etc.). If the knee angle changes too fast or too slow, the voice prompts are: "Your knee flexion and extension speed is too fast, please slow down appropriately" and "Your knee flexion and extension speed is too slow, please speed up appropriately".

[0095] The second aspect of the present application also provides a human-computer interaction implementation system based on visual perception, comprising:

[0096] A feature extraction module is configured to acquire a depth image through a visual sensor, extract an action feature as a first feature, extract a key point as a second feature, and extract a transient deformation feature as a third feature from a target region of the depth image.

[0097] A three-dimensional construction module is configured to obtain a global key frame according to the second feature and the third feature, construct a Gaussian subgraph, anchor the Gaussian subgraph to the global key frame, and obtain a three-dimensional action model.

[0098] A feature conversion module is configured to perform pose estimation according to the first feature and the second feature, evaluate a feature map associated with the three-dimensional action model at a key point position, obtain a key point descriptor, and construct an attention function based on the deviation between the key point descriptors.

[0099] A semantic network module is configured to construct a semantic segmentation network based on the key point descriptor and the attention function, cluster an interaction action, and learn a feature representation of the interaction action in the same cluster using the semantic segmentation network.

[0100] An interaction execution module is configured to obtain an interactive action of a user, map the interactive action into a cluster, judge whether to interact based on the size of the overlapping area in the cluster, and perform the interaction based on the semantic segmentation network if the interaction is performed.

[0101] In the embodiment, the method for extracting action features from a target region in a depth image as first features, extracting key points as second features, and extracting instantaneous deformation features as third features includes the following steps.

[0102] The depth image is preprocessed, including denoising, illumination correction, histogram equalization, and smoothing processing.

[0103] The target region is selected in the depth image based on an interaction mode, a convolution kernel is slid in the target region for feature extraction, action features are extracted using an optical flow algorithm, key points are extracted using an HRNet network, and instantaneous deformation features are extracted based on time series analysis.

[0104] In the embodiment, the method for obtaining a global key frame based on the second features and the third features includes the following steps.

[0105] The extracted second feature data and third feature data are normalized and data-aligned, the second features and the third features are combined into a new feature vector using a vector splicing method, each key point has a corresponding instantaneous deformation feature, the new feature vector is clustered, and a plurality of clusters containing frames are obtained.

[0106] The deviation degree of each frame relative to the cluster center is calculated, and the expression is as follows:

[0107]

[0108] wherein, and and represent weight coefficients for controlling Mahalanobis distance and Gaussian kernel deviation degree respectively, is a feature vector of a frame, is a feature vector of a cluster center, represents a transposition operation, is a covariance matrix, is a bandwidth parameter in a Gaussian kernel function;

[0109] The frame with the largest deviation degree is selected in each cluster, and the global key frame is formed by combination.

[0110] In the embodiment, the method for constructing a Gaussian subgraph, anchoring the Gaussian subgraph to the global key frame, and obtaining a three-dimensional action model includes the following steps.

[0111] The key point coordinates and the instantaneous deformation features in each global key frame are taken as data points in a high-dimensional space, and for each data point in the high-dimensional space, a kernel density estimation value is calculated, and the expression is as follows:

[0112]

[0113] wherein, is the total number of data points, is the dimension of the data points, is the covariance matrix, is any data point in the high-dimensional space, is the i-th data point in the high-dimensional space, is the i-th data point in the high-dimensional space, denotes the transpose operation;

[0114] Based on the kernel density estimate, a numerical matrix representing the density of the data distribution is obtained as a Gaussian subgraph;

[0115] According to the spatial change of the key points in the global key frame, a spatial transformation matrix is constructed, and the constructed Gaussian subgraph is transformed according to the spatial transformation matrix;

[0116] The transformed Gaussian subgraph is merged with the global key frame by weighted average to form a three-dimensional action model.

[0117] In this embodiment, the method of estimating pose according to the first feature and the second feature, evaluating the feature map associated with the three-dimensional action model at the key point position, and obtaining the key point descriptor, comprises:

[0118] According to the first feature and the second feature, a neural network model is used to construct a pose estimation model, and the action feature and the key point feature are used as input, and the preliminary pose estimation result is output;

[0119] Considering the depth images in the time sequence, the preliminary pose estimation result of the current frame and the preliminary pose estimation result of the previous frame are fused by weighted average, the weight distribution is determined according to the reliability of the previous frame pose and the confidence of the current frame, and the parameters of the pose estimation model are continuously updated until the processing of each frame of depth image is completed;

[0120] According to the output result of the optimized pose estimation model, the position of the key point in the three-dimensional action model is determined, and the three-dimensional coordinates are converted into two-dimensional coordinates in the feature map;

[0121] In the feature map, a local area is defined with the position of the key point as the center, and the size of the area is determined according to the resolution of the feature map, and the expression is:

[0122]

[0123] wherein, is a proportionality coefficient, is the width of the feature map, is the height of the feature map;

[0124] The gradient feature, texture feature and depth change feature of the pixels in the local region are calculated, and are connected in the order of the gradient direction to form a high-dimensional key point descriptor, and the expression is:

[0125]

[0126] wherein, is the key point descriptor at position , scale and gradient direction , is a normalization factor, represents a local region with as the center and as the scale, and are pixel position indexes in the local region, respectively represent image gradients at pixel positions and , represents the Hadanard product between vectors, is a nonlinear activation function, represents the inner product of image gradients at pixel positions and , used to extract texture features and depth change features, is a weighting function based on the gradient direction , used to emphasize the gradient consistent with the direction of the key point.

[0127] In the embodiment, the method for constructing the attention function based on the deviation between the key point descriptors comprises:

[0128] The average value of the Euclidean distance between the key point descriptors is calculated as the deviation, and the attention function is constructed based on the deviation, and the expression is:

[0129]

[0130] wherein, is the adjustment coefficient between the th and the th key point descriptor, , and respectively represent the query vector, the key vector and the value vector of the key point descriptor, and respectively represent the th and the th key point descriptor, represents an exponential distance deviation adjustment function, is a nonlinear activation function sigmoid, is the degree coefficient of the influence of the average deviation on attention, is the average value of the Euclidean distance between all key point descriptors, is represented as a distance deviation norm function, is a normalized exponential function, is represented as a normalization process on all attention outputs.

[0131] In the embodiment, the method for constructing a semantic segmentation network based on the key point descriptor and the attention function comprises:

[0132] The semantic segmentation network is constructed using a deep learning framework, which comprises: an input layer, taking the key point descriptor as the input of the network; a convolutional layer, applying a convolutional layer to feature extraction of the input key point descriptor; a pooling layer, inserting an average pooling layer between the convolutional layers to reduce the resolution of the feature map and increase the receptive field; an attention mechanism fusion layer, applying the attention function to the output feature map of the convolutional layer, and assigning different weights to different feature regions according to the deviation information; a semantic segmentation layer, an upsampling layer, restoring the feature map after attention weighting to the resolution corresponding to the input image; a classification layer, adding a fully connected layer at the end of the network, mapping the feature map to a semantic category; and an output layer, setting the number of output nodes of the semantic segmentation network based on the number of instructions to be set.

[0133] The expression of the objective function of the semantic segmentation network is:

[0134]

[0135] wherein, is the total number of pixels in the image, is the class of the th pixel point in the real label, is the class probability of the th pixel point in the predicted output, is the weight coefficient for balancing the cross-entropy and the regularization term, is the regularization coefficient, is the number of channels of the feature map, is the number of elements in each channel, is the value of the th element in the th channel of the attention weight matrix , is the average value of the attention weight of the th channel;

[0136] The spatial branch is introduced into the semantic segmentation network based on the attention function, and the expression of the spatial branch is:

[0137]

[0138] in, It is a fully connected layer. This represents a nonlinear transformation of the output of the softmax operation. This is the softmax operator. This represents a nonlinear transformation of the output of the global pooling operation. This is the global pooling operator. The weight parameters of the convolution kernel, This represents a nonlinear transformation of the output of the convolution operation. For convolution operators, The input feature map;

[0139] Obtain the dataset, divide it into training, validation and test sets in an 8:1:1 ratio, normalize it, and then train the semantic segmentation network.

[0140] In this embodiment, the method for learning feature representations of interactive actions within the same cluster using the semantic segmentation network includes:

[0141] The probability density function of interaction actions within the same cluster is calculated using the kernel density estimation method, and its expression is:

[0142]

[0143] in, For the point The estimated density function at that location, For the number of samples, For the dimensions of the data, In order to be in Points in a dimensional space for which the probability density needs to be estimated For the first d-dimensional sample points, The bandwidth matrix of the kernel function. This indicates that a transpose operation is being performed. For bandwidth matrix dependent The probability density estimation function, For the whole Variables in 3D space This is expressed as in constructing probability density estimates When, do not use the first The estimated function obtained from d-dimensional sample points;

[0144] The kernel density estimate is used as a weight to perform a weighted summation of image features, allowing the semantic segmentation network to learn the distribution characteristics of data within clusters. At the same time, the convolutional layers in the semantic segmentation network are used to learn the features of the cluster center vector and the covariance matrix within the cluster.

[0145] In this embodiment, the method of mapping interactive actions to clusters and determining whether an interaction has occurred based on the size of the overlapping region within the cluster includes:

[0146] For each cluster, obtain its cluster center location. When the user's interaction action feature vector is obtained, calculate the distance to the cluster center. Within each cluster, action vectors are cross-sampled at a ratio of 10% to form an action vector set. The distance between each action vector in the set and the cluster center is then calculated. ;

[0147] Statistical distance Less than or equal to number of vectors ,like If the number of samples exceeds half of the selected sample size, it indicates that the interaction action is relatively central within the cluster and the size of the overlapping area exceeds the threshold, and an interaction is determined to be performed; otherwise, no interaction is determined to be performed.

[0148] The above description is merely an example and illustration of the structure of the present invention. Those skilled in the art can make various modifications or additions to the specific embodiments described, or use similar methods to replace them, as long as they do not deviate from the structure of the invention or exceed the scope defined in the claims, all of which should fall within the protection scope of the present invention.

Claims

1. A method for realizing human-computer interaction based on visual perception, characterized in that, The method comprises the following steps: obtaining a depth image through a visual sensor, extracting motion features as first features, extracting key points as second features, and extracting instantaneous deformation features as third features from a target region of the depth image; obtaining a global key frame according to the second features and the third features, constructing a Gaussian subgraph, anchoring the Gaussian subgraph to the global key frame, and obtaining a three-dimensional motion model; performing pose estimation according to the first features and the second features, evaluating a feature map associated with the three-dimensional motion model at a key point position, and obtaining a key point descriptor; constructing an attention function based on deviations between the key point descriptors; constructing a semantic segmentation network based on the key point descriptors and the attention function, clustering interactive actions, and learning feature representations of the interactive actions in the same cluster using the semantic segmentation network; obtaining interactive actions of a user, mapping the interactive actions to a cluster, determining whether to interact based on a size of an overlapping region in the cluster, and performing interaction based on the semantic segmentation network if the interaction is to be performed.

2. The method of claim 1, wherein, The method for extracting motion features as first features, extracting key points as second features, and extracting instantaneous deformation features as third features from a target region of a depth image comprises: performing preprocessing on the depth image, including denoising, illumination correction, histogram equalization, and smoothing processing; selecting a target region in the depth image based on an interactive mode, performing feature extraction by sliding a convolution kernel in the target region, extracting motion features using an optical flow algorithm, extracting key points using an HRNet network, and extracting instantaneous deformation features based on time series analysis.

3. The method of claim 1, wherein the method further comprises: The method for obtaining a global key frame according to the second features and the third features comprises: performing normalization and data alignment processing on the extracted second feature data and third feature data, combining the second features and the third features into a new feature vector using a vector splicing method, each key point having a corresponding instantaneous deformation feature, clustering the new feature vector, and obtaining a plurality of clusters containing frames; calculating a deviation degree of each frame relative to a clustering center, the expression being: wherein, and respectively represent weight coefficients that control the Mahalanobis distance and the degree of deviation of the Gaussian kernel function, is a feature vector of a frame, is a feature vector of a cluster center, denotes a transposition operation, is a covariance matrix, is a bandwidth parameter in the Gaussian kernel function; selecting a frame with the largest deviation degree in each cluster, and combining the frame to form a global key frame.

4. The method of claim 1, wherein, The method for constructing a Gaussian subgraph, anchoring the Gaussian subgraph to the global key frame, and obtaining a three-dimensional motion model comprises: taking key point coordinates and instantaneous deformation features in each global key frame as data points in a high-dimensional space, calculating a kernel density estimation value for each data point in the high-dimensional space, the expression being: wherein, is the total number of data points, is the dimension of the data points, is the covariance matrix, is any data point in the high dimensional space, is the i-th data point in the high dimensional space, is the i-th data point in the high dimensional space, denotes the transpose operation; obtaining a numerical matrix representing data distribution density based on the kernel density estimation value, as a Gaussian subgraph; constructing a spatial transformation matrix according to spatial changes of the key points in the global key frame, transforming the constructed Gaussian subgraph according to the spatial transformation matrix; merging the transformed Gaussian subgraph and the global key frame through weighted averaging, and forming a three-dimensional motion model.

5. The method of claim 1, wherein, The method for performing pose estimation according to the first features and the second features, evaluating a feature map associated with the three-dimensional motion model at a key point position, and obtaining a key point descriptor comprises: According to the first feature and the second feature, a pose estimation model is constructed using a neural network model, action features and key point features are taken as inputs, and a preliminary pose estimation result is output; Considering the depth images on the time sequence, the preliminary pose estimation result of the current frame is fused with the preliminary pose estimation result of the previous frame through weighted average, the weight distribution is determined according to the reliability of the previous frame pose and the confidence of the current frame, the parameters of the pose estimation model are continuously updated until the processing of each frame of depth image is completed; According to the output result of the optimized pose estimation model, the positions of the key points in the three-dimensional action model are determined, and the three-dimensional coordinates are converted into two-dimensional coordinates in the feature map; In the feature map, a local area is defined with the position of the key point as the center, and the area size is determined according to the resolution of the feature map, and the expression is: wherein, is a proportionality coefficient, is the width of the feature map, is the height of the feature map; The gradient features, texture features and depth change features of the pixels in the local area are calculated, and they are connected in the order of gradient direction to form a high-dimensional key point descriptor, and the expression is: wherein is a keypoint descriptor at position , scale and gradient direction , is a normalization factor, denotes a local region centered at with scale , and are pixel position indices within the local region, denote image gradients at pixel positions and respectively, denotes a Hadanard product between the vectors, is a non-linear activation function, denotes an inner product of image gradients at pixel positions and for extracting texture and depth variation features, is a weighting function based on gradient direction for emphasizing gradients consistent with the keypoint direction.

6. The method of claim 1, wherein the method further comprises: The method for constructing an attention function based on the deviation between the key point descriptors comprises: The average value of the Euclidean distance between the key point descriptors is calculated as the deviation, and the attention function is constructed based on the deviation, and the expression is: wherein, is the and the adjustment coefficient between the , and represent the query vector, key vector and value vector of the keypoint descriptor respectively, and represent the and the keypoint descriptor respectively, represents the exponential distance bias adjustment function, is the nonlinear activation function sigmoid, is the average bias-to-attention influence degree coefficient, is the average of the Euclidean distance between all keypoint descriptors, represents the distance bias norm function, is the normalized exponential function, represents the normalization processing on all attention outputs.

7. The method of claim 1, wherein, The method for constructing a semantic segmentation network based on the key point descriptors and the attention function comprises: A semantic segmentation network is constructed using a deep learning framework, which comprises: an input layer: taking the key point descriptors as the input of the network; a convolution layer: applying the convolution layer to feature extraction of the input key point descriptors; a pooling layer: inserting an average pooling layer between the convolution layers to reduce the resolution of the feature map and increase the receptive field; an attention mechanism fusion layer: applying the attention function to the output feature map of the convolution layer to assign different weights to different feature regions according to the deviation information; a semantic segmentation layer: an upsampling layer, which restores the feature map after attention weighting to the resolution corresponding to the input image; a classification layer, which adds a fully connected layer at the end of the network to map the feature map to a semantic category; and an output layer, which sets the number of output nodes of the semantic segmentation network based on the number of instructions to be set; The expression of the objective function of the semantic segmentation network is: wherein, is the total number of pixels in the image, is the class of the pixel in the ground truth, is the class probability of the pixel in the prediction output, is the weight coefficient balancing the cross-entropy and the regularization term, is the regularization coefficient, is the number of channels of the feature map, is the number of elements in each channel, is the value of the element in the channel of the attention weight matrix is the attention weight mean of the channel.​ A spatial branch is introduced into the semantic segmentation network based on the attention function, and the expression of the spatial branch is: wherein, is a fully connected layer, denotes a non-linear transformation on the output of a softmax operation, is a softmax operator, denotes a non-linear transformation on the output of a global pooling operation, is a global pooling operator, is a weight parameter of a convolution kernel, denotes a non-linear transformation on the output of a convolution operation, is a convolution operator, is an input feature map; A data set is obtained, which is divided into a training set, a validation set and a test set in a ratio of 8:1:1, and is normalized, and the semantic segmentation network is trained.

8. The method of claim 1, wherein, The method for learning the feature representation of the interactive action in the same cluster using the semantic segmentation network comprises: The probability density function of the interactive action in the same cluster is calculated using the kernel density estimation method, and the expression is: in, For the point The estimated density function at that location, For the number of samples, For the dimensions of the data, In order to be in Points in a dimensional space for which the probability density needs to be estimated For the first d-dimensional sample points, The bandwidth matrix of the kernel function. This indicates that a transpose operation is being performed. For bandwidth matrix dependent The probability density estimation function, For the whole Variables in 3D space This is expressed as in constructing probability density estimates When, do not use the first The estimated function obtained from d-dimensional sample points; The kernel density estimation value is taken as a weight for weighted summation of the image features, so that the semantic segmentation network learns the distribution features of the data in the cluster, and the convolution layer in the semantic segmentation network is used to learn the center vector of the cluster and the features of the covariance matrix in the cluster.

9. The method of claim 1, wherein, The method for mapping the interactive action into the cluster and judging whether the interaction is based on the size of the overlapping area in the cluster comprises: For each cluster, the cluster center position is obtained, when the user's interactive action feature vector is obtained, the distance from the cluster center is calculated, and in the cluster, the action vectors are cross-selected according to a 10% proportion to form an action vector set, and the distance from each action vector in the set to the cluster center is calculated ; statistical distance less than or equal to number of vectors if if more than half of the selected sample number, it means that the interaction action is in the relative center within the cluster and the overlapping area size exceeds the threshold value, it is determined to interact, otherwise, it is determined not to interact.

10. A system for implementing a visual perception based human-computer interaction to execute the method of any one of claims 1-9. The system comprises: The feature extraction module is configured to acquire a depth image through a visual sensor, extract motion features from a target region of the depth image as first features, extract key points as second features, and extract instantaneous deformation features as third features; The three-dimensional construction module is configured to obtain a global key frame according to the second features and the third features, construct a Gaussian subgraph, anchor the Gaussian subgraph to the global key frame, and obtain a three-dimensional motion model; The feature conversion module is configured to perform pose estimation according to the first features and the second features, evaluate feature maps associated with the three-dimensional motion model at key point positions to obtain key point descriptors, and construct an attention function based on deviations between the key point descriptors; The semantic network module is configured to construct a semantic segmentation network based on the key point descriptors and the attention function, cluster interactive motions, and learn feature representations of the interactive motions in the same cluster using the semantic segmentation network; The interaction execution module is configured to acquire an interactive motion of a user, map the interactive motion to a cluster, determine whether to interact based on a size of an overlapping region in the cluster, and perform interaction based on the semantic segmentation network if the interaction is to be performed.

Images