Embodied intelligent human-computer interaction system prediction method and device based on action recognition

Abstract

The application relates to the technical field of human-computer interaction, in particular to a method and device for predicting an embodied intelligent human-computer interaction system based on motion recognition, wherein the method comprises the following steps: collecting three-dimensional coordinates of a plurality of skeleton key points of a human body and time frames corresponding to the coordinates of each skeleton joint to generate initial skeleton data; preprocessing the skeleton data to obtain final skeleton data meeting a condition; performing skeleton coding on the final skeleton data to extract at least one skeleton feature; and training a prediction model based on the at least one skeleton feature, wherein the prediction model comprises a Kronecker product layer, a Kernel Matrix layer and a multilayer perception machine layer to identify a future motion trajectory. Thus, the problem that, in the prior art, due to a large model parameter quantity, strong data dependency and difficulty in intuitively displaying the relationship between variables, the effectiveness and interpretability of the model in actual application are limited, and the trust of a user in an identification result is affected, is solved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of human-computer interaction, and in particular to a method and device for predicting an embodied intelligent human-computer interaction system based on motion recognition. BACKGROUND

[0002] In related technologies, human-computer interaction based on motion recognition usually collects video or image information through a camera, then performs image preprocessing (such as but not limited to denoising, binarization, and edge detection), extracts a motion region of interest, extracts motion features that can effectively represent the motion from the preprocessed image, and then uses a large model and algorithm to recognize the motion.

[0003] However, in related technologies, a large model has a large number of parameters, resulting in high consumption of computing resources, long training period, high deployment and operation cost, and complex training optimization; has strong data dependency, requires a large amount of manpower and material resources to obtain labeled data, and the quality of the data is difficult to guarantee; and it is difficult to intuitively display the relationship between variables, and the high-dimensional distribution of model parameters makes the interaction relationship unclear, which needs to be improved. SUMMARY

[0004] The present application provides a method and device for predicting an embodied intelligent human-computer interaction system based on motion recognition, to solve the problem in related technologies that a large model has a large number of parameters, strong data dependency, and difficulty in intuitively displaying the relationship between variables, thereby limiting the effectiveness and interpretability of the model in actual application and affecting user trust in the recognition result.

[0005] The first aspect of the present application provides a method for predicting an embodied intelligent human-computer interaction system based on motion recognition, comprising the following steps: collecting three-dimensional coordinates of a plurality of skeletal key points of a human body and time frames corresponding to coordinates of each skeletal joint, to generate initial skeletal data; preprocessing the skeletal data to obtain final skeletal data that satisfies a preset processing condition; skeleton coding the final skeletal data to extract at least one skeletal feature; and training a prediction model based on the at least one skeletal feature, wherein the prediction model comprises a Kronecker product layer, a Kernel Matrix layer, and a multilayer perception layer to identify a future motion trajectory.

[0006] By the technical solutions, the human body action information can be comprehensively acquired by collecting human body skeleton key point data; the data quality is improved by preprocessing operation, and the model robustness is enhanced; the skeleton features are extracted by skeleton coding, the action behavior is effectively reflected, and the calculation complexity is low; based on the prediction model including the Kronecker product layer, the Kernel Matrix layer and the multi-layer perception layer, the Kronecker product layer can capture high-order nonlinear relationships, the Kernel Matrix layer can assist in processing, and the multi-layer perception layer can output classification results, so that the future action trajectory can be accurately identified, accurate action prediction can be provided for embodied intelligent human-computer interaction, the accuracy, fluency and naturalness of interaction are improved, and the computing resource demand is reduced and the operation efficiency is improved.

[0007] Optionally, in an embodiment of the present application, the preprocessing of the skeleton data to obtain final skeleton data satisfying a preset processing condition comprises: converting the initial skeleton data from a camera coordinate system to a human body coordinate system to obtain first skeleton data; performing enhancement processing on the first skeleton data, and performing cropping and / or flipping to obtain second skeleton data; and performing normalization processing on the second skeleton data to obtain the final skeleton data.

[0008] By the technical solutions, the initial skeleton data can be converted to the human body coordinate system, the influence of the view angle change is effectively eliminated, the relative motion features between joints are highlighted, and the data can better reflect the essence of human body action; the data is subjected to enhancement processing such as cropping and flipping, the data diversity is increased, the adaptability and robustness of the model to different situations are improved, the normalization processing scales the data coordinates to a specific range, the model training efficiency is improved, the input data has more stable gradients, the magnitude difference between features is reduced, the model performance is further optimized, a solid data foundation is laid for subsequent accurate action recognition and prediction, and the reliability and accuracy of the entire embodied intelligent human-computer interaction system based on action recognition are improved.

[0009] Optionally, in an embodiment of the present application, the skeleton coding of the final skeleton data to extract at least one skeleton feature comprises: mapping x, y and z coordinates to three channels of RGB respectively, and converting time series data into spatial data to change the final skeleton data into an RGB color image.

[0010] By the technical solution, the embodiment of the application can map the x, y and z coordinates to the RGB channel to convert the time series data into spatial data, realize effective conversion of the data form, represent the multi-dimensional information of the skeleton data in an intuitive and compact form, and make the originally complex time series skeleton data easier to process and analyze. The embodiment of the application not only reduces the complexity of data processing, but also effectively extracts at least one skeleton feature, which can accurately reflect the action behavior of a user, provides a high-quality and representative data basis for subsequent feature-based model training and action trajectory recognition, and thus helps to improve the performance and accuracy of the whole embodiment intelligent human-computer interaction system based on action recognition.

[0011] Optionally, in an embodiment of the application, the identifying the future action trajectory comprises: performing a multi-order Kronecker product operation on the input matrix based on the Kronecker product layer to form an N-order input matrix, and designing an X N ignoring high-order repeated items; defining N kernel functions based on the Kernel Matrix layer to obtain an output corresponding to an order number by using each order input matrix corresponding to the order number; and splicing the output matrix through the multi-layer perception layer and outputting through a pure linear multi-layer perception to obtain the future action trajectory.

[0012] Through the technical solution, the embodiment of the application can capture high-order nonlinear relationships of the system through the multi-order operation of the Kronecker product layer, the designed arrangement operation can effectively ignore high-order repeated items and reduce the complexity of the model; the Kernel Matrix layer defines the kernel function to obtain the output by using the input matrix, which enhances the processing capability of the model on the data features; the multi-layer perception layer splices the output matrix and outputs through a pure linear function, discards the traditional activation function, and still effectively captures the nonlinear characteristics of the system, enhances the interpretability of the model, and overall realizes accurate identification of the future action trajectory, improves the performance of the embodiment intelligent human-computer interaction system based on action recognition, and provides the system with an efficient, accurate and interpretable action prediction capability.

[0013] Optionally, in an embodiment of the application, after training the prediction model, the method further comprises: evaluating the actual performance of the prediction model by using a preset test data set until a preset condition is reached to form a final prediction model.

[0014] By the technical solution, the embodiment of the present application can evaluate the actual performance of the prediction model by using the test data set, and timely find the deficiencies of the model in the training process, such as the problems in prediction accuracy, stability, etc. Continuous evaluation until a certain condition is reached can ensure that the model is continuously optimized and improved, avoiding overfitting or underfitting, etc., thereby forming a final prediction model with high reliability and high precision, providing more stable and effective motion trajectory recognition and prediction ability for the embodied intelligent human-computer interaction system based on motion recognition, and improving the performance and user experience of the entire system in actual application.

[0015] The second aspect embodiment of the present application provides a device for predicting an embodied intelligent human-computer interaction system based on motion recognition, comprising: a data acquisition module for collecting three-dimensional coordinates of a plurality of skeletal key points of a human body and time frames corresponding to coordinates of each skeletal joint, to generate initial skeletal data; a data preprocessing module for preprocessing the skeletal data to obtain final skeletal data that meets a preset processing condition; a feature extraction module for skeleton coding the final skeletal data to extract at least one skeletal feature; and a recognition module for training a prediction model based on the at least one skeletal feature, wherein the prediction model comprises a Kronecker product layer, a Kernel Matrix layer, and a multi-layer perception layer to identify future motion trajectories.

[0016] By the technical solution, the embodiment of the present application can collect human skeletal key point data to comprehensively obtain human motion information; improve data quality through preprocessing operations to enhance model robustness; extract skeletal features through skeleton coding to effectively reflect motion behavior with low computational complexity; based on the prediction model comprising the Kronecker product layer, the Kernel Matrix layer, and the multi-layer perception layer, the Kronecker product layer can capture high-order nonlinear relationships, the Kernel Matrix layer assists in processing, and the multi-layer perception layer outputs classification results to accurately identify future motion trajectories, providing accurate motion prediction for embodied intelligent human-computer interaction, improving the accuracy, smoothness, and naturalness of interaction, while reducing the demand for computing resources and improving running efficiency.

[0017] Optionally, in an embodiment of the present application, the data preprocessing module comprises: a conversion unit for converting the initial skeletal data from a camera coordinate system to a human body coordinate system to obtain first skeletal data; an enhancement unit for performing enhancement processing on the first skeletal data and performing cropping and / or flipping to obtain second skeletal data; and a normalization unit for performing normalization processing on the second skeletal data to obtain the final skeletal data.

[0018] By the technical solution, the initial skeleton data can be converted into a human body coordinate system, the influence of the change of the visual angle is effectively eliminated, the relative motion characteristics between joints are highlighted, the data can better reflect the essence of human body motion, the data is subjected to enhancement processing such as cropping and flipping operation, the data diversity is increased, the adaptability and robustness of the model to different situations are improved, the data coordinates are scaled to a specific range through normalization processing, the model training efficiency is improved, the input data has more stable gradient, the magnitude difference between features is reduced, the model performance is further optimized, a solid data foundation is laid for subsequent accurate motion recognition and prediction, and the reliability and accuracy of the whole embodied intelligent human-computer interaction system based on motion recognition are improved.

[0019] Optionally, in an embodiment of the present application, the feature extraction module comprises: mapping x, y, z coordinates to RGB three channels respectively, and converting time series data into spatial data, so as to change the final skeleton data into an RGB color image.

[0020] By the technical solution, the x, y, z coordinates can be mapped to the RGB channel to convert the time series data into spatial data, the effective conversion of the data form is realized, the multi-dimensional information of the skeleton data can be represented in an intuitive and compact form, the originally complex time series skeleton data is easier to process and analyze, the complexity of data processing is reduced, at least one skeleton feature can be effectively extracted, the features can accurately reflect the motion behavior of the user, a high-quality and representative data foundation is provided for subsequent feature-based model training and motion trajectory recognition, and thus the performance and accuracy of the whole embodied intelligent human-computer interaction system based on motion recognition are improved.

[0021] Optionally, in an embodiment of the present application, the recognition module comprises: a first calculation unit configured to perform multi-order Kronecker product operation on the input matrix based on the Kronecker product layer to form an N-order input matrix, and design X N to ignore high-order repeated items; a second calculation unit configured to define N kernel functions based on the Kernel Matrix layer to obtain the output of the corresponding order by using the input matrix of the corresponding order; and a third calculation unit configured to splice the output matrix through the multi-layer perception layer and output through a pure linear multi-layer perception to obtain the future motion trajectory.

[0022] By the technical solution, the embodiment of the application can capture high-order nonlinear relationships of a system through multi-order operation by the Kron product layer, the designed arrangement operation can effectively ignore high-order repeated items, and reduce model complexity; the KernelMatrix layer defines a kernel function to obtain an output by using an input matrix, and enhances processing capability of the model on data features; the multi-layer perception layer splices the output matrix and outputs by pure linearity, discards a traditional activation function, and still effectively captures nonlinear characteristics of the system, enhances interpretability of the model, and realizes accurate recognition of future motion trajectories, and improves performance of the embodied intelligent human-computer interaction system based on motion recognition, and provides the system with efficient, accurate, and interpretable motion prediction capability.

[0023] Optionally, in an embodiment of the application, after the prediction model is trained, the method further includes: evaluating actual performance of the prediction model by using a preset test data set, until a preset condition is reached, and a final prediction model is formed.

[0024] By the technical solution, the embodiment of the application can evaluate actual performance of the prediction model by using a test data set, discover deficiencies of the model in a training process in time, such as problems in prediction accuracy and stability, continuously evaluate until a certain condition is reached, ensure continuous optimization and improvement of the model, avoid overfitting or underfitting, and thus form a final prediction model with high reliability and high precision, provide the embodied intelligent human-computer interaction system based on motion recognition with more stable and effective motion trajectory recognition and prediction capability, and improve performance and user experience of the entire system in actual application.

[0025] An embodiment of the third aspect of the application provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor executes the program to implement the method for predicting the embodied intelligent human-computer interaction system based on motion recognition as described in the above embodiments.

[0026] An embodiment of the fourth aspect of the application provides a computer readable storage medium, which stores a computer program, and the program is executed by a processor to implement the method for predicting the embodied intelligent human-computer interaction system based on motion recognition as described above.

[0027] An embodiment of the fifth aspect of the application provides a computer program product, including a computer program, and the computer program is executed to implement the method for predicting the embodied intelligent human-computer interaction system based on motion recognition as described above.

[0028] Additional aspects and advantages of the application will be in part apparent and in part pointed out hereinafter. BRIEF DESCRIPTION OF DRAWINGS

[0029] The above and / or additional aspects and advantages of the present application will become apparent and more readily appreciated from the following description of embodiments, taken in conjunction with the accompanying drawings, in which:

[0030] Figure 1 A flow chart of a method for predicting an embodied intelligent human-computer interaction system based on action recognition according to an embodiment of the present application;

[0031] Figure 2 A schematic diagram of data flow according to an embodiment of the present application;

[0032] Figure 3 A schematic diagram of network structure according to an embodiment of the present application;

[0033] Figure 4 A schematic diagram of a data recognition method according to a specific embodiment of the present application;

[0034] Figure 5 A structural schematic diagram of a device for predicting an embodied intelligent human-computer interaction system based on action recognition according to an embodiment of the present application;

[0035] Figure 6 A structural example diagram of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0036] Embodiments of the present application are described in detail below with reference to the attached drawing figures, wherein the same or like component have the same or similar designations. The embodiments described below are intended to be exemplary only, and are not to be construed as limiting the present application.

[0037] A method and device for predicting an embodied intelligent human-computer interaction system based on motion recognition are described below with reference to the accompanying drawings. In view of the problems in the related art that a large model has a large number of parameters, strong data dependency, and difficulty in intuitively displaying the relationship between variables, thereby limiting the effectiveness and interpretability of the model in actual application and affecting user trust in the recognition result, the present application provides a method for predicting an embodied intelligent human-computer interaction system based on motion recognition. In the method, human body motion information can be comprehensively obtained by collecting human body skeleton key point data. Data quality is improved and model robustness is enhanced through preprocessing operations. Skeleton features are extracted through skeleton coding, effectively reflecting action behavior and having low computational complexity. Based on a prediction model including a Kronecker product layer, a Kernel Matrix layer, and a multilayer perceptron layer, the Kronecker product layer can capture high-order nonlinear relationships, the Kernel Matrix layer can assist in processing, and the multilayer perceptron layer can output classification results, accurately identifying future motion trajectories and providing accurate motion prediction for embodied intelligent human-computer interaction, improving the accuracy, smoothness, and naturalness of the interaction, while reducing the demand for computing resources and improving the running efficiency. Thus, the problem in the related art that a large model has a large number of parameters, strong data dependency, and difficulty in intuitively displaying the relationship between variables, thereby limiting the effectiveness and interpretability of the model in actual application and affecting user trust in the recognition result is solved.

[0038] Specifically, Figure 1 A flowchart of a method for predicting an embodied intelligent human-computer interaction system based on motion recognition provided by an embodiment of the present application is shown in FIG. 1.

[0039] As shown in Figure 1 , the method for predicting an embodied intelligent human-computer interaction system based on motion recognition includes the following steps:

[0040] In step S101, three-dimensional coordinates of a plurality of human body skeleton key points and time frames corresponding to coordinates of each skeleton joint are collected to generate initial skeleton data.

[0041] An embodiment of the present application uses a Kinect V2 (Microsoft Kinect for Windows v2, a depth perception device developed by Microsoft) depth camera for data collection. The depth camera has a unique hardware structure, including an RGB camera, a depth (infrared) camera, an infrared projector, and a microphone array. Its depth perception capability gives it a significant advantage in human body skeleton data acquisition.

[0042] In actual execution, the three-dimensional coordinates of 25 human skeleton key points and the time frame corresponding to the coordinates of each skeleton joint are collected by the depth camera. These skeleton key points cover key parts of the human body, such as the head, torso, limbs, etc. By accurately obtaining the three-dimensional coordinates, the posture and action of the human body can be comprehensively and meticulously described. At the same time, the time frame corresponding to the coordinates of each joint is recorded, providing a data basis for subsequent analysis of the timing and dynamic changes of the action, so that the system can capture the evolution process of the human action in the time dimension, which helps to more accurately understand and predict the action behavior.

[0043] Specifically, compared with traditional RGB images, the skeleton information collected by the depth camera has stronger robustness. In actual application scenarios, RGB images are easily disturbed by factors such as background and light, for example, under strong light, it may produce reflection, causing inaccurate recognition of part of the limbs; complex background may cover the human body contour, affecting the posture judgment. The depth camera can focus on the human skeleton structure itself through its depth perception technology, effectively ignoring the influence of background and light changes, so as to more accurately represent the human action.

[0044] In particular, the above data collection method does not require the user to wear additional equipment such as sensors, markers, etc. In some traditional motion capture systems, in order to obtain more accurate action data, the user may be required to wear multiple sensors or markers, but this will bring inconvenience to the user, increase the burden of the user, and may affect the natural action performance of the user. The non-contact, non-wearing equipment data collection method of the Kinect V2 depth camera greatly reduces the complexity of collecting data, reduces the restrictions on the user, and also reduces the resource threshold, which has obvious advantages in terms of device cost and use convenience.

[0045] The embodiment of the present application can collect human skeleton data through the Kinect V2 depth camera, which has depth perception advantages due to its unique hardware structure. The 25 key point three-dimensional coordinates and time frames collected can comprehensively and meticulously describe the human posture and action, providing a basis for analyzing timing and dynamic changes. Compared with RGB images, the skeleton information has strong robustness and is not disturbed by background and light. Moreover, the user does not need to wear additional equipment, which reduces the complexity of data collection and the resource threshold, improves the device cost and use convenience, and helps to accurately understand and predict the action behavior.

[0046] In step S102, the skeleton data is preprocessed to obtain final skeleton data that meets a preset processing condition.

[0047] It can be understood that preprocessing the skeleton data can improve data quality and enhance model performance, so that the model can adapt to different scenarios and needs.

[0048] Optionally, in an embodiment of the present application, the skeleton data is preprocessed to obtain final skeleton data satisfying a preset processing condition, including: converting the initial skeleton data from a camera coordinate system to a human body coordinate system to obtain first skeleton data; performing enhancement processing on the first skeleton data and performing cropping and / or flipping to obtain second skeleton data; performing normalization processing on the second skeleton data to obtain the final skeleton data.

[0049] In actual execution, a view-independent transformation is performed to convert the skeleton data from a camera coordinate system to a human body coordinate system. In actual data collection, the camera coordinate system is established based on the position and direction of the camera itself. Different camera angles will cause differences in the coordinate representation of the skeleton data, which will interfere with the subsequent accurate analysis of human motion features. By converting it to a human body coordinate system, the influence of the change in the angle of view can be effectively eliminated, making the description of the skeleton data more stable and unified, and not affected by the change in the position of the camera, highlighting the relative motion features between the joints. In the human body coordinate system, the coordinates of the skeleton joints can more directly reflect the relative motion relationship between the various parts of the human body, rather than the relative position relationship with the camera. This helps to more accurately capture the essence of human motion, for example, in the human body coordinate system, the stretching motion of the arm can be more intuitively represented by the relative displacement between the joints, without being disturbed by the camera angle, providing a more valuable data basis for subsequent analysis and model training based on motion features.

[0050] Further, the first skeleton data is subjected to enhancement processing, including performing random cropping and flipping operations. Random cropping is to randomly select part of the area for retention and discard the other areas. For example, for image data containing human motion, a local area containing key motion parts can be randomly cropped, which can increase the diversity of the data, so that the model can learn the relationship between different local features and overall motion. The flipping operation is to flip the image or data in the horizontal or vertical direction, such as horizontally flipping an image of a left-hand waving motion to become an image of a right-hand waving motion, thereby expanding the sample size and diversity of the data set. The main purpose of these operations is to increase the diversity of the data and the robustness of the model. By increasing the diversity of the data, the model can be exposed to more different forms of samples during training, avoiding overfitting, i.e., the model will not only learn specific data set features, but will be able to learn more extensive motion feature patterns, thereby improving the ability to recognize new, unseen motion data. The enhanced model can more stably and accurately perform motion recognition in the face of various complex situations that may occur in actual applications, such as different angles, different part occlusions, etc., and exhibits stronger robustness.

[0051] Further, the second bone data is normalized to scale the bone data coordinate values between 0 and 1. The specific calculation method of normalization is to map each coordinate value to the interval of 0 to 1 according to the range of coordinate values in the data set. This operation is mainly to improve the efficiency of model training. During model training, the magnitude difference of different features may affect the convergence speed and effect of the gradient descent algorithm. If the numerical range of a certain feature is too large, it may dominate the update of other features when calculating the gradient, causing the model training to be unstable or slow to converge. By normalizing, all features have similar magnitudes, which can ensure that each feature gets a reasonable weight distribution when calculating the gradient, thereby speeding up the model training process, improving training efficiency, and making the model converge to better parameter values faster.

[0052] Overall, as shown in FIG. 10, the three-dimensional coordinates of the 25 skeletal key points of the human body collected in step S101 and the time frames corresponding to the coordinates of each skeletal joint are preprocessed and then input into the model. After the model processes the data, it outputs the recognized and classified skeletal joint coordinates. Figure 2

[0053] In terms of data preprocessing, data cleaning, standardization, and missing value processing are required to ensure data consistency and accuracy. During data cleaning, obviously incorrect or abnormal data points are removed, such as incorrect coordinate values caused by sensor failure or interference. Standardization ensures consistency in format, units, etc., such as unifying the representation format of all coordinate data, so that data from different sources or collected at different times can be processed under the same framework. Missing value processing can use various methods such as mean filling and interpolation, and appropriate methods are selected to supplement missing coordinate values or time frame information based on the characteristics and distribution of the data to ensure data integrity.

[0054] After preprocessing the data, the data set is divided into an 80% training set and a 20% test set to fully capture the dynamic characteristics of the skeleton sequence data and evaluate the model performance. The training set is used for model training, and the model learns the skeletal data features and motion patterns in the training set to adjust its parameters to achieve accurate recognition of different motions. The test set is used to evaluate the performance of the trained model on unseen data, which can objectively reflect the generalization ability of the model, i.e., the recognition accuracy of the model for new motion data in different scenarios.

[0055] ​The embodiments of the present application can eliminate the influence of the viewing angle through coordinate system conversion, highlight the relative motion characteristics of the joints, provide a more optimal data basis for motion analysis and model training; the data enhancement operation increases data diversity and model robustness, avoids overfitting, and improves the stability and accuracy of motion recognition in complex situations; normalization processing improves model training efficiency and accelerates convergence; data cleaning, standardization and missing value processing ensure data consistency and integrity; reasonable data set division can effectively evaluate the model generalization ability, so that the model can accurately recognize different motions, the entire preprocessing process comprehensively improves data quality, optimizes model performance, and enhances system reliability and adaptability.

[0056] In step S103, the final skeleton data is skeleton coded to extract at least one skeleton feature.

[0057] It can be understood that the skeleton feature is a key element for describing and distinguishing different human motions. In embodied intelligent human-computer interaction systems based on motion recognition, models recognize different motion trajectories by learning and analyzing these skeleton features, thereby achieving understanding and response to user motion intentions. Accurate and effective skeleton features can improve the accuracy of motion recognition, enabling the system to interact with users more accurately, such as playing an important role in virtual reality, intelligent monitoring, rehabilitation training, etc.

[0058] In some embodiments, the final skeleton data is skeleton coded to extract at least one skeleton feature, including: mapping the x, y, z coordinates to the RGB three channels respectively, and converting the time series data into spatial data, so as to change the final skeleton data into an RGB color image. For example, for a certain skeleton joint at a certain moment, the x coordinate value is mapped to the red channel, the y coordinate value is mapped to the green channel, and the z coordinate value is mapped to the blue channel. In this way, a three-dimensional coordinate information is converted into a color value of a pixel point in a color image. With the passage of time, the coordinates of the skeleton joints at different moments are mapped to the image in turn, forming a series of color image sequences that can reflect the spatial and temporal changes of human motion, which can effectively reflect the user's action behavior, and has low computational complexity and can reduce the demand for computing resources.

[0059] The embodiments of the present application can extract skeleton features by skeleton coding the final skeleton data, convert time series data into spatial data by mapping x, y, z coordinates to RGB three channels, and form color image sequences that can reflect the spatio-temporal changes of human motion. Not only can it effectively reflect the user's action behavior, help the model accurately recognize the motion trajectory to understand and respond to the user's intention, and improve the accuracy of human-computer interaction, but also has low computational complexity and can reduce the demand for computing resources, which is valuable in virtual reality, intelligent monitoring, rehabilitation training and other fields.

[0060] In step S104, a prediction model is trained based on at least one skeletal feature, wherein the prediction model includes a Kronecker layer, a Kernel Matrix layer, and a multilayer perceptron layer to identify future motion trajectories.

[0061] The network structure of the prediction model is as follows Figure 3 As shown, it includes three layers: Kronecker layer, Kernel Matrix layer and multilayer perceptron layer. A Rotation Mask and kernel function are also designed to achieve accurate identification of the three-dimensional skeletal coordinates of the human body in space through lightweight design, while reducing the computational load and storage space requirements of the model. It is suitable for resource-constrained environments and improves the efficiency of model operation.

[0062] Optionally, in one embodiment of this application, a multi-order Kronecker product operation is performed on the input matrix based on the Kronecker product layer to form an N-order input matrix, and a method is designed to process the X after the N-order Kronecker product. N The process involves tidying up the data to ignore higher-order repetitions; defining N kernel functions based on the Kernel Matrix layer to obtain the corresponding order output using the input matrices of each order; and concatenating the output matrix through a multilayer perceptron layer and outputting it through a purely linear multilayer perceptron to obtain the future motion trajectory.

[0063] Specifically, firstly, the Kronecker layer performs multi-order Kronecker product operations on the input matrix to form an N-order input matrix, capturing the higher-order nonlinear relationships of the system. Simultaneously, a Rotation Mask is designed to tidy up the input matrix after the N-order Kronecker product, ignoring higher-order duplicate terms. Then, N kernel functions are defined in the Kernel Matrix layer, applied to the corresponding order of the input matrix to obtain the corresponding order of the output. Finally, the output matrix is ​​concatenated through a multilayer perceptron layer and output through a purely linear multilayer perceptron to obtain the final classification result.

[0064] The following describes step S104 in detail using a specific embodiment, such as... Figure 4 As shown, the input data, after being transformed into an input matrix by the input layer, will pass through the Kronecker layer. The Kronecker layer can represent the higher-order nonlinear relationships of the system. By passing the input matrix through an N-order Kronecker product, an N-order input matrix can be generated. For any input matrix X, the index mapping relationship before and after the expansion is as follows: X(i,j) = X e (m), where m = (i-1)*N+j. X ​​after the Nth order Krone product is denoted as Xi. N X N If the shape is (length, ..., length) and the dimension is N, then the following relationship holds: XN (i, j,..., n) = X e (i) x X e (j) x... x X e (n), where X N is the result of the Kronecker product of N X e matrices, so there are a large number of repeated high-order terms in the result.

[0065] Further, by solving the problem of repeated elements in the high-order case of the model through the Mask, the embodiment creates an N-order Rotation Mask, which has the same form as the N-order input matrix X N , where each element can be obtained by the following formula:

[0066]

[0067] The algorithm defines N kernel functions, which are applied to the corresponding order matrix, and the corresponding order output is obtained. The output matrix is further processed in the multi-layer perception layer after splicing. It should be noted that, unlike the traditional multi-layer perception, the multi-layer perception used in the embodiment does not contain a nonlinear activation function. The output matrix obtained by the previous stage is directly fused through a linear layer using a linear superposition method. After linear superposition of the corresponding order, the classification result is finally obtained and output. The following is the mathematical expression of the model structure:

[0068] y(k) = b + k1 x y (1) (k) +... + k n x y (n) (k),

[0069] where y m (k) is the output result of any order obtained by the foregoing operation, as follows:

[0070] y (m) (k) = h m x X N ,

[0071] It is worth noting that the x here is redefined by the einsum operation, and the einsum description is: a..., b... → ab.

[0072] On this structure, the embodiment captures the nonlinear characteristics of the system and the correlation between the covariates through the kernel function, and discards the traditional method of capturing nonlinear characteristics through the activation function. Experiments prove that the method is valid and effective.

[0073] Optionally, in an embodiment of the present application, after training the prediction model, further comprising: evaluating the actual performance of the prediction model by using a preset test data set, until a preset condition is reached, forming a final prediction model.

[0074] After training the model, the actual performance of the prediction model is evaluated by using a test data set. Calculating classification error, accuracy and other indicators is an important means to evaluate the performance of the model. Classification error can intuitively reflect the difference between the model prediction result and the true result, for example, using MAE (Mean Absolute Error) and MSE (Mean Squared Error) method, wherein the calculation method of MAE is: The calculation method of MSE is: Wherein yi is the identified target, is the identification value.

[0075] The embodiment of the present application can capture high-order nonlinear relationships through the Kronecker product layer multi-order operation, ignore high-order repeated items by using Rotation Mask to reduce the amount of calculation and storage space requirement, which is suitable for resource limited environment and improves the running efficiency; the kernel function of the Kernel Matrix layer can mine the nonlinear characteristics and covariant correlation of the system; the multilayer perception layer adopts a pure linear output mode to enhance the explainability. After training, the performance is evaluated by using a preset test data set, and the difference is measured by using MAE, MSE and other indicators, and the final model is continuously optimized, which is helpful for accurately identifying future motion trajectory and improving the accuracy and practicality of the model.

[0076] In summary, the embodiment of the present application can stably provide the following functions:

[0077] 1) Accurate classification of action behavior and expression: the model should be able to process a small amount of short-term action data, and can perform long-term action recognition and classification to reduce the cost of data acquisition. By effectively using these variables, the model should achieve high-precision classification effect, and ensure that reliable classification results can be provided in various scenes and different users.

[0078] 2) Explainability of the model: it can not only accurately identify and classify different actions and expressions, but also intuitively express the relationship between each human action key point variable. At the same time, the model parameters can reflect the specific contribution and mutual influence of each action key point on the identification result, helping users understand the internal mechanism of the model. And can carry out detailed influence analysis on the variables, quantifying the influence degree of different variables on long-term action behavior. Through feature importance analysis and partial dependence diagram technology, the role and function of each variable in identification are clear.

[0079] 3) Interaction strategy formulation and validation: Utilizing the variable relationships and impact analysis provided by the model, specific interaction strategies are developed, such as controlling the robot to perform corresponding actions. Through simulation and practical application, the effectiveness of these interaction strategies is verified, enhancing the smooth and natural interaction between users and robots.

[0080] According to the embodiment of the present application, the embodiment of the present application provides a method for predicting the embodiment of the present application based on action recognition. The method can collect human skeleton key point data and comprehensively obtain human action information. The data quality is improved through preprocessing operation, and the model robustness is enhanced. The skeleton features are extracted through skeleton coding, the action behavior is effectively reflected, and the calculation complexity is low. Based on the prediction model including the Kron layer, the Kernel Matrix layer and the multi-layer perception layer, the Kron layer can capture high-order nonlinear relationships, the Kernel Matrix layer can assist in processing, and the multi-layer perception layer can output classification results. The future action trajectory is accurately identified, the precise action prediction for the embodiment of the present application is provided, the accuracy, smoothness and naturalness of the interaction are improved, the demand for computing resources is reduced, and the operation efficiency is improved.

[0081] Secondly, the device for predicting the embodiment of the present application based on action recognition of the embodiment of the present application is described with reference to the accompanying drawings.

[0082] Figure 5 The device for predicting the embodiment of the present application based on action recognition of the embodiment of the present application is a block schematic diagram.

[0083] As shown in Figure 5 The device for predicting the embodiment of the present application based on action recognition includes a data acquisition module 100, a data preprocessing module 200, a feature extraction module 300 and an identification module 400.

[0084] Specifically, the data acquisition module 100 is configured to collect three-dimensional coordinates of a plurality of skeleton key points of a human body and time frames corresponding to coordinates of each skeleton joint, so as to generate initial skeleton data.

[0085] The data preprocessing module 200 is configured to preprocess the skeleton data to obtain final skeleton data satisfying a preset processing condition.

[0086] The feature extraction module 300 is configured to perform skeleton coding on the final skeleton data to extract at least one skeleton feature.

[0087] The identification module 400 is configured to train a prediction model based on the at least one skeleton feature, wherein the prediction model includes a Kron layer, a Kernel Matrix layer and a multi-layer perception layer, so as to identify a future action trajectory.

[0088] Optionally, in one embodiment of this application, the data preprocessing module 200 includes: a conversion unit, an enhancement unit, and a normalization unit.

[0089] The transformation unit is used to convert the initial skeleton data from the camera coordinate system to the human coordinate system to obtain the first skeleton data.

[0090] An enhancement unit is used to enhance the first bone data and perform clipping and / or flipping operations to obtain the second bone data.

[0091] The normalization unit is used to normalize the second skeleton data to obtain the final skeleton data.

[0092] Optionally, in one embodiment of this application, the feature extraction module 300 includes: mapping the x, y, and z coordinates to the three RGB channels respectively, converting the temporal data into spatial data, so as to convert the final skeletal data into an RGB color image.

[0093] Optionally, in one embodiment of this application, the identification module 400 includes: a first calculation unit, a second calculation unit, and a third calculation unit.

[0094] The first computational unit is used to perform multi-order Kronecker product operations on the input matrix based on the Kronecker product layer to form an N-order input matrix, and to design the processing of X after the N-order Kronecker product. N Organize the code to ignore higher-order duplicates.

[0095] The second computational unit is used to define N kernel functions based on the Kernel Matrix layer, so as to obtain the corresponding order output by using the input matrices of each order.

[0096] The third computing unit is used to concatenate the output matrix through a multilayer perceptron layer and output it through a purely linear multilayer perceptron to obtain the future motion trajectory.

[0097] Optionally, in one embodiment of this application, after training the prediction model, the method further includes: evaluating the actual performance of the prediction model using a preset test dataset until a preset condition is met, and then forming the final prediction model.

[0098] It should be noted that the foregoing explanation of the method embodiment for predicting embodied intelligent human-computer interaction system based on action recognition also applies to the apparatus for predicting embodied intelligent human-computer interaction system based on action recognition in this embodiment, and will not be repeated here.

[0099] The device for predicting the embodied intelligent human-computer interaction system based on action recognition provided by the embodiment of the present application can comprehensively obtain human action information by collecting human skeleton key point data, improve data quality through preprocessing operation, enhance model robustness, effectively reflect action behavior through skeleton coding to extract skeleton features, and has low calculation complexity. Based on the prediction model containing a Kronecker product layer, a Kernel Matrix layer and a multilayer perception layer, the Kronecker product layer can capture high-order nonlinear relationships, the Kernel Matrix layer can assist in processing, and the multilayer perception layer can output classification results, accurately identify future action trajectories, provide accurate action prediction for embodied intelligent human-computer interaction, improve the accuracy, fluency and naturalness of interaction, reduce the demand for computing resources, and improve the running efficiency.

[0100] Figure 6 The structure schematic diagram of the electronic device provided by the embodiment of the present application is provided. The electronic device can include:

[0101] The memory 601, the processor 602 and the computer program stored in the memory 601 and executable on the processor 602.

[0102] The processor 602 implements the method for predicting the embodied intelligent human-computer interaction system based on action recognition provided in the above embodiment when executing the program.

[0103] Further, the electronic device further includes:

[0104] The communication interface 603 is used for communication between the memory 601 and the processor 602.

[0105] The memory 601 is used for storing the computer program executable on the processor 602.

[0106] The memory 601 can include a high-speed RAM memory, and can also include a non-volatile memory, for example, at least one disk memory.

[0107] If the memory 601, the processor 602 and the communication interface 603 are implemented independently, the communication interface 603, the memory 601 and the processor 602 can be connected with each other through a bus and complete communication between each other. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For convenience of representation, Figure 6 Only one thick line is used to represent the bus in the figure, but it does not mean that there is only one bus or only one type of bus.

[0108] Optionally, in a specific implementation, if the memory 601, the processor 602 and the communication interface 603 are integrated on a chip, the memory 601, the processor 602 and the communication interface 603 can complete communication between each other through an internal interface.

[0109] The processor 602 can be a Central Processing Unit (CPU), or an Application Specific Integrated Circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present application.

[0110] The embodiments of the present application also provide a computer readable storage medium, which stores a computer program, and the program is executed by a processor to implement the method for predicting a body-aware intelligent human-computer interaction system based on action recognition.

[0111] The embodiments of the present application also provide a computer program product, which includes a computer program, and the computer program is executed to implement the method for predicting a body-aware intelligent human-computer interaction system based on action recognition.

[0112] In the description of the application, reference to "one embodiment", "some embodiments", "an example", "a specific example", or "some examples" means that a particular feature, structure, material, or characteristic being described is included in at least one embodiment or example of the application. The appearances of the phrase in various places in the specification are not necessarily all referring to the same embodiment or example. Furthermore, the described specific features, structures, materials, or characteristics can be combined in any suitable manner in one or more embodiments or examples. In addition, the usage of "N" means at least two, for example, two, three or the like, unless explicitly stated otherwise.

[0113] Furthermore, the terms "first", "second", or the like, are used merely as a designation of certain elements or features of the application, and do not imply or connote relative importance or a specific order of precedence. Thus, features defined with "first", "second", etc. can include at least one of the features, either explicitly or implicitly.

[0114] Any process or method descriptions or blocks in flow charts or otherwise described herein represent embodiments which can be managed as one or more modules, segments, or portions of code which include one or more executable instructions for implementing specific logic functions or steps, and alternate implementations are possible. In some embodiments, the processes and methods described can be executably encoded on a machine- readable medium in a data signal embodied in an electromagnetic signal, a wireless signal, or a propagated signal.

[0115] The logic and / or steps represented in the flowcharts and / or described herein, for example, can be considered as a sequence of executable instructions stored in a computer readable medium, which can be executed by an instruction execution system, apparatus or device, such as a computer-based system, a processor-based system, or other system that can fetch the instructions from the instruction execution system, apparatus or device and execute the instructions, or a combination of the above. For the purposes of this specification, a "computer readable medium" can be any apparatus that can contain, store, communicate, propagate, or transport the program for use by or in connection with the instruction execution system, apparatus or device. The computer readable medium can be a computer readable storage medium or a computer readable signal medium. The computer readable storage medium can include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or a propagation medium. The computer readable signal medium can include, but is not limited to, a computer readable medium that facilitates transfer of the program from one place to another. A specific example of a computer readable medium is a non-transitory computer-readable storage medium. A specific example of a computer readable signal medium is a source or destination of the computer readable medium. Another specific example of a computer readable signal medium is a computer readable signal travelling through space. Thus, a computer readable medium can take many forms of hardware to carry out the program for use by or in connection with the instruction execution system, apparatus or device.

[0116] It should be understood that aspects of the application can be implemented in hardware, software, firmware or combinations thereof. In the above embodiments, the N steps or methods can be implemented in software or firmware stored in a memory and executed by a suitable instruction execution system. If implemented in hardware and in another embodiment, the hardware can be implemented using any or a combination of the following technologies, which are each well known in the art: a discrete logic circuit(s) having logic gates for implementing logic functions upon an application of data signals, an application specific integrated circuit having appropriate combinational logic gates, a programmable gate array(s) (PGA), a field programmable gate array (FPGA), etc.

[0117] Those of skill in the art would understand that the steps of the methods carried out above can be carried out wholly or partly by a program instructing relevant hardware, and the program can be stored in a computer readable storage medium, and when executed, includes one or a combination of the steps of the method embodiments.

[0118] In addition, each of the functional units in the various embodiments of the present application can be integrated in one processing module, or each of the units can be physically present separately, or two or more units can be integrated in one module. The integrated module can be realized in the form of hardware or in the form of a software functional module. When the integrated module is realized in the form of a software functional module and sold or used as an independent product, it can also be stored in a computer readable storage medium.

[0119] The storage medium mentioned above can be a read-only memory, a magnetic disk or an optical disk, etc. Although the embodiments of the present application have been shown and described above, it should be understood that the above embodiments are exemplary and should not be construed as limiting the present application, and those skilled in the art can make changes, modifications, replacements and variations to the above embodiments within the scope of the present application.

Claims

1. A method for embodied intelligent human-computer interaction system prediction based on action recognition, characterized in that, The method comprises the following steps: Collecting three-dimensional coordinates of a plurality of human skeleton key points and time frames corresponding to coordinates of each skeleton joint to generate initial skeleton data; Preprocessing the skeleton data to obtain final skeleton data satisfying a preset preprocessing condition; Skeleton coding the final skeleton data to extract at least one skeleton feature; Training a prediction model based on the at least one skeleton feature, wherein the prediction model comprises a Kronecker product layer, a Kernel Matrix layer and a multilayer perceptron layer to predict a future motion trajectory; The skeleton coding of the final skeleton data to extract at least one skeleton feature comprises: Mapping the coordinates of the skeleton joints to the RGB three channels respectively, converting the time series data into spatial data, and converting the final skeleton data into an RGB color image.

2. The method of claim 1, wherein, The preprocessing of the skeleton data to obtain the final skeleton data satisfying the preset processing condition comprises: Converting the initial skeleton data from a camera coordinate system to a human body coordinate system to obtain first skeleton data; Performing enhancement processing on the first skeleton data and performing cropping and / or flipping to obtain second skeleton data; Performing normalization processing on the second skeleton data to obtain the final skeleton data.

3. The method of claim 1, wherein, The prediction of the future motion trajectory comprises: Taking the at least one skeleton feature as an input matrix; performing multi-order Kronecker product operation on the input matrix based on the Kronecker product layer to form an N-order input matrix and arranging the N-order Kronecker product processed to ignore high-order repeated items. Defining N kernel functions based on the Kernel Matrix layer to obtain an output matrix of a corresponding order by using an input matrix of a corresponding order; Splicing the output matrix through the multilayer perceptron layer and outputting through a pure linear multilayer perceptron to obtain the future motion trajectory.

4. The method of claim 1, wherein, After training the prediction model, further comprising: Using a preset test data set to evaluate the actual performance of the prediction model until a preset condition is reached to form a final prediction model.

5. An apparatus for action recognition based embodied intelligent human-computer interaction system prediction, characterized in that, Comprise: A data acquisition module for collecting three-dimensional coordinates of a plurality of human skeleton key points and time frames corresponding to coordinates of each skeleton joint to generate initial skeleton data; A data preprocessing module for preprocessing the skeleton data to obtain final skeleton data satisfying a preset preprocessing condition; A feature extraction module for skeleton coding the final skeleton data to extract at least one skeleton feature; An identification module for training a prediction model based on the at least one skeleton feature, wherein the prediction model comprises a Kronecker product layer, a Kernel Matrix layer and a multilayer perceptron layer to predict a future motion trajectory; The skeleton coding of the final skeleton data to extract at least one skeleton feature comprises: Mapping the coordinates of the skeleton joints to the RGB three channels respectively, converting the time series data into spatial data, and converting the final skeleton data into an RGB color image.

6. The apparatus of claim 5, wherein, The data preprocessing module comprises: A conversion unit for converting the initial skeleton data from a camera coordinate system to a human body coordinate system to obtain first skeleton data; an enhancement unit configured to perform an enhancement process on the first skeleton data, and perform a clipping operation and / or a flipping to obtain second skeleton data; a normalization unit configured to perform a normalization process on the second skeleton data to obtain the final skeleton data.

7. An electronic device, comprising: comprising: a memory, a processor, and a computer program stored on the memory and executable on the processor, the processor executing the program to implement the method for predicting embodied intelligent human-machine interaction system based on action recognition according to any one of claims 1-4.

8. A computer-readable storage medium having stored thereon a computer program, characterized in that, The program is executed by the processor to implement the method for predicting embodied intelligent human-machine interaction system based on action recognition according to any one of claims 1-4.

9. A computer program product comprising a computer program, characterized in that, The computer program is executed to implement the method for predicting embodied intelligent human-machine interaction system based on action recognition according to any one of claims 1-4.

Images