Interactive teaching evaluation auxiliary system and device based on multi-modal fusion

Abstract

The invention belongs to the field of evaluation assistance, and particularly relates to an interactive teaching evaluation assistance system and device based on multi-modal fusion, and the system collects multi-dimensional information through a portrait module, and constructs a user training action portrait library in combination with multi-modal feature extraction and a knowledge graph algorithm; the strategy recommendation module generates an optimal guidance scheme through a correction scheme matching model according to the portrait library, the real-time information and the teaching guidance strategy library; the auxiliary adjustment module collects training action images in real time, and obtains a guided continuous correction action sequence and a standard score by using a spatial-temporal feature extraction model and an evaluation algorithm; the traceability feedback module is combined with the action standard knowledge base to obtain transverse and longitudinal spatial-temporal change chains of non-standard actions through a forward reasoning model, and feeds back the transverse and longitudinal spatial-temporal change chains to the instructor to adjust the scheme in real time until the actions reach the standard; according to the method, accurate evaluation and personalized scheme matching of multi-modal data fusion are realized, and scientificity and interactivity of evaluation assistance are effectively improved.

Description

TECHNICAL FIELD

[0001] The application belongs to the field of evaluation assistance, and particularly relates to an interactive teaching evaluation assistance system and device based on multi-modal fusion. BACKGROUND

[0002] Under the background of the in-depth promotion of educational informatization, the traditional physical education classroom teaching mode has obvious limitations: teachers rely on on-site observation and experience judgment, and it is difficult to quantitatively record and systematically analyze the student action standardization and training intensity, and individual differences are affected, and personalized teaching goals are difficult to implement, such as difficult to capture the subtle deviation of joint angle in track and throw, and difficult to adjust the load according to real-time physiological action in physical training; at the same time, the existing mode also has problems such as lack of multi-modal data fusion evaluation system, lack of personalized scheme matching mechanism, and insufficient action evaluation and tracing ability, which leads to one-sided description of student sports ability, difficulty in generating adaptive guidance scheme, lack of accurate basis for teacher correction, and restricts the improvement of physical education quality; therefore, the application provides an interactive teaching evaluation assistance system and device based on multi-modal fusion. SUMMARY

[0003] In view of the deficiencies of the prior art, the application provides an interactive teaching evaluation assistance system and device based on multi-modal fusion, which collects multi-dimensional information through an image module, combines multi-modal feature extraction and knowledge graph algorithm to construct a user training action image library; an optimal guidance scheme is generated by a strategy recommendation module according to the image library, real-time information and a teaching guidance strategy library through a correction scheme matching model; a continuous correction action sequence and a standard score after guidance are obtained by an auxiliary adjustment module by using a space-time feature extraction model and an evaluation algorithm to real-time collect training action images; a transverse and longitudinal space-time change chain of non-standard action is obtained by a tracing feedback module through a forward reasoning model in combination with an action standard knowledge base, and is fed back to the guidance teacher to adjust the scheme in real time until the action meets the standard; the application realizes precise evaluation and personalized scheme matching of multi-modal data fusion, and effectively improves the scientificity and interactivity of evaluation assistance.

[0004] To achieve the above object, the application provides the following technical scheme:

[0005] The interactive teaching evaluation assistance system based on multi-modal fusion comprises an image module, a strategy recommendation module, an auxiliary adjustment module and a tracing feedback module.

[0006] The image module is used for collecting user basic information, user historical training standard information, corresponding guidance scheme information and standard action information knowledge base of the student terminal, and obtaining a user training action image library in combination with a knowledge graph algorithm.

[0007] The strategy recommendation module is configured to obtain an optimal matching guidance scheme corresponding to the user by a correction scheme matching model according to the user training action portrait library, the real-time collected user basic information and the configured teaching guidance strategy library.

[0008] The auxiliary adjustment module is configured to assist in guidance according to the matching guidance scheme, and obtain a continuous correction action sequence after guidance and a corresponding correction standard evaluation score by a preset spatiotemporal feature extraction model and a comprehensive evaluation algorithm in combination with the training action information of the student end.

[0009] The traceability feedback module is configured to obtain an action spatiotemporal change chain of a non-standard action in a corresponding teaching scene by a forward reasoning model in combination with the standard action information knowledge base according to the continuous correction action sequence after guidance and the corresponding correction standard evaluation score, feed back the action spatiotemporal change chain of the non-standard action to an alarm device configured at the teacher end for real-time early warning, collect feedback opinions of the teacher end to adjust the matching guidance scheme in real time until the training action of the corresponding user meets the condition, and store the guidance scheme of the user training action meeting the condition to the standard action information knowledge base for management.

[0010] Specifically, the action spatiotemporal change chain of the non-standard action includes a horizontal non-standard action spatiotemporal change chain of different training actions constructed under the same guidance scheme in a current training period and a vertical non-standard action spatiotemporal change chain of the same training action constructed under the same guidance scheme in different training periods.

[0011] The portrait module includes a data collection unit, a first data analysis unit and a second data analysis unit.

[0012] The data collection unit is configured to collect user basic information, user historical motion evaluation action text and image information, corresponding guidance scheme information and a standard action information knowledge base of a configured student end, and classify and arrange the collected information according to users and collection time stamps and perform standardization preprocessing.

[0013] The first data analysis unit is configured to obtain user action evaluation information triplets and corresponding action spatiotemporal feature sets according to user historical motion evaluation action text information and image information in combination with an entity relationship extraction algorithm and an image recognition algorithm.

[0014] The second data analysis unit is configured to obtain action correlation relationships at continuous time points and probabilities of non-standard actions after a previous action being non-standard according to the standard action information knowledge base in combination with an associated analysis algorithm with built-in expert experience.

[0015] Specifically, the portrait module further includes a portrait unit.

[0016] An image unit is configured to construct a user training action image library by a graph algorithm according to user basic information, a user action evaluation information triple, a corresponding action spatiotemporal feature set, a standard action information knowledge base, a correlation between actions at continuous time points, and a probability that a next action is not standard under the condition that a previous action is not standard; the user training action image library includes a reference layer, a training layer, and an inference layer;

[0017] The reference layer is configured to store user basic information; the training layer is configured to store training action spatiotemporal feature information, a corresponding training evaluation score, and action guidance correction information in each guidance scheme in a user history; and the inference layer is configured to store a standard action information knowledge base, and construct a causal inference connection between continuous actions in combination with a correlation between actions at continuous time points and a probability that a next action is not standard under the condition that a previous action is not standard, to obtain a training action causal inference chain, which is simultaneously mapped to a training layer under a corresponding user to evaluate and inferively correct action information of the corresponding user.

[0018] Specifically, the auxiliary adjustment module includes a spatiotemporal feature extraction unit and an evaluation unit.

[0019] The spatiotemporal feature extraction unit is configured to obtain a spatiotemporal correlation feature space of an optimal matching guidance scheme for a user in combination with a preset spatiotemporal graph convolution model according to training action information corresponding to the optimal matching guidance scheme.

[0020] The evaluation unit is configured to obtain a training evaluation score of each action in the optimal matching guidance scheme by a comprehensive evaluation algorithm in combination with the training action causal inference chain according to the spatiotemporal correlation feature space of the optimal matching guidance scheme for the user.

[0021] Specifically, the traceability feedback module includes a reverse inference unit and a feedback adjustment unit.

[0022] The reverse inference unit is configured to perform forward causal inference on a nonstandard action in combination with a causal inference connection in the training action causal inference chain according to the training evaluation score of each action in the optimal matching guidance scheme and an action spatiotemporal change chain of a historical longitudinal nonstandard action, and obtain an action spatiotemporal change chain of a real-time transverse nonstandard action and an action spatiotemporal change chain of a real-time longitudinal nonstandard action according to the training evaluation score and an HSV color mapping space.

[0023] The feedback adjustment unit is configured to feed back the action space-time variation chain of the real-time transverse non-standard action, the action space-time variation chain of the real-time longitudinal non-standard action and the corresponding training evaluation score to the portrait unit for storage, and judge the comprehensive training evaluation score of the optimal matching guidance scheme and the preset evaluation threshold. If the comprehensive training evaluation score is greater than the preset evaluation threshold but any action does not meet the corresponding standard action information, the corresponding action information and the non-standard action trace information obtained by the transverse non-standard action are fed back to the corresponding teacher of the teacher end for guidance and correction. If the comprehensive training evaluation score is less than or equal to the preset evaluation threshold, the corresponding guidance scheme information, the training evaluation score and the trace result are fed back to the corresponding guidance teacher, and the optimal matching guidance scheme is adjusted in real time.

[0024] Specifically, the construction process of the correction scheme matching model includes:

[0025] Based on the training action space-time feature information and the corresponding training evaluation score in each guidance scheme in the user training action portrait library, the action space-time feature and the training evaluation score change trend of the same training action in different training periods are extracted by the non-standard action layer in the correction scheme matching model, and the user longitudinal non-standard action feature change space and the training evaluation score fluctuation coefficient are obtained.

[0026] Based on the user longitudinal non-standard action feature change space and the training evaluation score fluctuation coefficient, the associated action associated coverage adjustment degree of the same training period after the current non-standard action adjustment under the same training period is obtained in combination with the action space-time variation chain of the transverse non-standard action in the user training action portrait library.

[0027] Meanwhile, according to the training evaluation score fluctuation coefficient of the associated action under the corresponding associated coverage adjustment degree of different training periods after the current non-standard action adjustment, the longitudinal effect transfer rate of the same associated action in different training periods is obtained. Specifically, the construction process of the correction scheme matching model further includes:

[0028] Based on the training action in each training period, a transverse action node sequence is constructed, and a transverse association connection is constructed by using the associated action associated coverage adjustment degree of the same training period after the current non-standard action adjustment under the same training period.

[0029] Based on the training evaluation score fluctuation coefficient of the same training action of the user in different training periods and the longitudinal effect transfer rate of the same associated action in different training periods, a longitudinal association transfer connection is constructed.

[0030] Based on the constructed different training period horizontal action node, the horizontal association connection and the longitudinal association migration connection are combined with the topology space layer to obtain the user historical training action effect association migration space;

[0031] Based on the user historical training action effect association migration space, the teaching guidance strategy library is combined, the matching layer configured with the maximum adjustment evaluation effect function is matched for searching to generate a matching guidance scheme sequence and a corresponding matching loss function; the maximum adjustment evaluation effect function is a function constructed by taking the maximum value of the correction standard evaluation score corresponding to the non-standard action and the associated action, the probability of the next action being non-standard under the condition that the previous action is non-standard, and the effect migration rate combined with the linear fitting function after matching scheme training;

[0032] Based on the matching guidance scheme sequence and the corresponding matching loss function, the preset loss function threshold and the set model training period are combined for training to obtain a trained correction scheme matching model, and the optimal matching guidance scheme corresponding to the maximum adjustment evaluation effect is output.

[0033] Specifically, the construction process of the space-time feature extraction model includes:

[0034] The training action information and the corresponding standard action image information in the continuous time period in the different training periods of the user are obtained and enhanced and denoised, and the key skeleton node marking and the gravity node marking under different actions are performed according to the expert marking algorithm pre-trained based on the processed images and the skeleton node information; a first continuous action marking image sequence of the same training period and a second marking image sequence of the same training action corresponding to different training periods are obtained;

[0035] The first continuous action marking image sequence of the same training period and the second marking image sequence of the same training action corresponding to different training periods are simultaneously input into the OpenPose action estimation layer to obtain a continuous action node feature space of the same training period, a node feature space of the same training action corresponding to different training periods, and a node deviation feature space of the same training action corresponding to different training periods constructed based on the standard continuous action node feature space;

[0036] Based on the continuous action node feature space of the same training period, the connection edges between nodes are constructed based on the body structure to obtain the undirected node graph corresponding to each image;

[0037] Based on the priority of the corresponding node belonging to the limb action in each image at the continuous time point in the undirected node graph corresponding to each image, the continuous action connection is constructed.

[0038] Based on the undirected node graph corresponding to each image and the continuous action connection, the continuous action topology space is constructed through the topology space layer.

[0039] Specifically, the construction process of the spatio-temporal feature extraction model further includes:

[0040] Based on the continuous action topological space combining the node deviation feature space of the same training action corresponding to different training periods, the continuous action topological space containing the action deviation change trend of the same action in different training periods is obtained through a time trend extraction layer;

[0041] Meanwhile, the continuous action topological space is input into a limb curvature recognition layer for action stage division, and the stage feature sequence of each action in the continuous action is obtained; the stage feature sequence is constructed by the node space coordinates corresponding to the starting time, the node space coordinates corresponding to the turning time and the node space coordinates corresponding to the ending time of the corresponding action;

[0042] Based on the root node and the gravity node in each action corresponding node in the continuous action topological space, the regional node feature set in the undirected node graph corresponding to each image is obtained in combination with a preset node division strategy;

[0043] The continuous action topological space containing the action deviation change trend of the same action in different training periods, the stage feature sequence of each action in the continuous action and the regional node feature set are input into a feature fusion layer constructed by a convolution attention network, and a continuous action fusion topological space is obtained;

[0044] Based on the continuous action fusion topological space combining the standard continuous action node feature space, the continuous action node coordinate sequence and the node coordinate deviation loss are obtained through an output layer, and the deviation change trend and the trend confidence of the corresponding action are obtained; meanwhile, the continuous action fusion topological space combining the standard continuous action node feature space is input into a correlation causality layer, and the forward cumulative contribution path and the path confidence corresponding to the non-standard action are obtained;

[0045] The loss function constructed based on the node coordinate deviation loss, the trend confidence and the path confidence is trained in combination with a preset loss function threshold and a training period, and a trained spatio-temporal feature extraction model is obtained.

[0046] The interactive teaching evaluation auxiliary device based on multi-modal fusion includes a data acquisition component and a processor configured in the device, and an image module is integrated in the data acquisition component; a strategy recommendation module, an auxiliary adjustment module and a traceability feedback module are integrated in the processor;

[0047] The data acquisition component is used to acquire the user basic information, the user historical training compliance information, the corresponding guidance scheme information and the standard action information knowledge base configured at the student end, and to obtain a user training action image library in combination with the built-in multi-modal feature extraction algorithm and the knowledge graph algorithm;

[0048] The processor generates an optimal matching guidance scheme through a correction scheme matching model based on a user training action image library, real-time collected user basic information and a configured teaching guidance strategy library, and when the user trains according to the scheme, real-time training action images are collected, and a continuous correction action sequence after guidance and a corresponding correction standard evaluation score are obtained by combining a space-time feature extraction model and an action evaluation algorithm; according to the continuous correction action sequence after guidance and the corresponding correction standard evaluation score and the standard action information knowledge base, a transverse and longitudinal space-time change chain of non-standard actions is obtained through a forward reasoning model, and the action space-time change chain of the non-standard actions is fed back to the configured teacher end, and the matching guidance scheme is adjusted in real time according to the feedback opinions of the teacher end until the corresponding user training action meets the condition, and the guidance scheme of the user training action meeting the condition is stored in the standard action information knowledge base for management.

[0049] Compared with the prior art, the beneficial effects of the present application are:

[0050] The present application aims at the deficiencies of the prior art, and realizes the deep integration of user information and action features by constructing a user training action image library through multi-modal feature extraction and graph algorithm of the image module; the individualization and pertinence of the guidance scheme are improved by means of the correction scheme matching model combined with longitudinal and transverse correlation analysis; the accuracy of action evaluation is enhanced by accurately extracting action features and evaluation by using the space-time feature extraction model; the space-time change chain is obtained by forward reasoning of the traceability feedback module and dynamically adjusted to ensure that the action continues to meet the standard, and finally a closed loop from personalized scheme recommendation to accurate evaluation and dynamic optimization is realized, effectively improving the scientificity and effectiveness of training. BRIEF DESCRIPTION OF DRAWINGS

[0051] Figure 1 Figure 1 is a module diagram of the interactive teaching evaluation auxiliary system based on multi-modal fusion of the embodiment 1 of the present application;

[0052] Figure 2 Figure 2 is an architecture diagram of the space-time feature extraction model of the embodiment 1 of the present application. DETAILED DESCRIPTION

[0053] Embodiment 1

[0054] Please refer to Figure 1 An embodiment provided by the present application: an interactive teaching evaluation auxiliary system based on multi-modal fusion, comprising: an image module, a strategy recommendation module, an auxiliary adjustment module and a traceability feedback module;

[0055] The image module is used for collecting user basic information, user historical training standard information and corresponding guidance scheme information of the configured student end, and combining a knowledge graph algorithm to obtain a user training action image library;

[0056] The strategy recommendation module is configured to obtain an optimal matching guidance scheme corresponding to the user by a correction scheme matching model according to the user training action portrait library, the real-time collected user basic information and the configured teaching guidance strategy library.

[0057] The auxiliary adjustment module is configured to perform auxiliary guidance according to the matching guidance scheme, and obtain a continuous correction action sequence after guidance and a corresponding correction standard evaluation score by a preset spatiotemporal feature extraction model and a comprehensive evaluation algorithm in combination with the training action information of the student end.

[0058] The traceability feedback module is configured to obtain an action spatiotemporal change chain of a non-standard action in a corresponding teaching scene by a forward reasoning model in combination with the standard action information knowledge base according to the continuous correction action sequence after guidance and the corresponding correction standard evaluation score, feed back the action spatiotemporal change chain of the non-standard action to an alarm device configured at the teacher end for real-time early warning, collect feedback opinions of the teacher end to adjust the matching guidance scheme in real time until the training action of the corresponding user meets the condition, and store the guidance scheme of the user training action meeting the condition to the standard action information knowledge base for management. It needs to be further explained that the action spatiotemporal change chain of the non-standard action in the embodiment includes a horizontal non-standard action spatiotemporal change chain constructed by different training actions of the same guidance scheme in the current training period and a vertical non-standard action spatiotemporal change chain constructed by the same training action in the same guidance scheme in different training periods. The forward reasoning model in the embodiment is preferably a deep reasoning algorithm, and the user derives a continuous non-standard action path leading to the current non-standard action. It needs to be further explained that the student end and the teacher end in the embodiment are realized by a monitoring device configured by a person skilled in the art.

[0059] It needs to be further explained that the portrait module in the embodiment includes a data acquisition unit, a first data analysis unit, a second data analysis unit and a portrait unit.

[0060] The data acquisition unit is configured to acquire user basic information, user historical motion evaluation action text and image information, corresponding guidance scheme information and a standard action information knowledge base of a configured student end, and classify and arrange the acquired information according to users and acquisition time stamps and perform standardization preprocessing.

[0061] It needs to be further explained that the process of classification, arrangement and standardization preprocessing in the embodiment is as follows:

[0062] Based on the collected user basic information, historical motion evaluation action text information, guidance scheme information and standard action information knowledge base, grouping clustering is performed through the user unique identifier, and then classification is realized by ascending order arrangement according to the collection time stamp in each user group. Meanwhile, through data cleaning, samples with missing field proportion exceeding 30% are filtered out, abnormal values deviating from the mean value by 3 times of the standard deviation in the numerical information are removed, the date format is standardized to "YYYY-MM-DD HH:MM:SS", the text information is converted to UTF-8 code, the numerical data such as training evaluation scores are mapped to the 0-100 standardized interval, the evaluation text is segmented by using the jieba segmentation, meaningless words are removed by using the stop word table, professional terms are mapped to the unified terms in the action standard knowledge base, and the standardized information data set with user grouping, time sequence arrangement and unified format and clean data is obtained.

[0063] The first data analysis unit is configured to obtain user action evaluation information triplets and corresponding action spatio-temporal feature sets according to the user historical motion evaluation action text information and image information combined entity relationship extraction algorithm and image recognition algorithm.

[0064] It should be further explained that one specific way of implementing the entity relationship extraction algorithm and the image recognition algorithm in the embodiment is as follows: based on the user historical motion evaluation action text information, first, segmentation, part-of-speech tagging and named entity recognition are performed, then a pre-trained language model is used to extract deep semantic features of the text, a bidirectional LSTM and a CRF model are combined to construct an entity relationship extraction framework, the association relationship between entities is recognized, and user action evaluation information triplets are obtained; based on the user training action information, first, image denoising and scale normalization preprocessing are performed, then a target detection algorithm is used to locate the human body region, 18 key skeleton node coordinates are extracted by combining the action estimation model, and the spatio-temporal feature set of the skeleton node motion trajectory of the continuous frame image is encoded by a 3D convolutional neural network, and the corresponding action spatio-temporal feature set is obtained. It should be noted that the named entity recognition in the embodiment includes but is not limited to recognizing action names, evaluation indexes, score values and the like;

[0065] The second data analysis unit is configured to obtain the action association relationship at continuous time points and the probability of the next action being non-standard under the condition that the previous action is non-standard according to the standard action information knowledge base combined with the correlation analysis algorithm with built-in expert experience.

[0066] It should be further explained that one specific way of implementing the correlation analysis algorithm with built-in expert experience in the embodiment is as follows:

[0067] Based on the continuous action sequence data in the standard action information knowledge base and the built-in expert experience rule base, the expert experience is first converted into algorithmic constraints. For example, the influence weight range of lower limb action on core action is specified. Then, the improved association rule algorithm is used to perform frequent pattern mining on the historical action evaluation data to extract the association item set between continuous actions. The probability of non-standard action under the condition of non-standard previous action is calculated by combining the Bayesian probability formula to obtain the action association relationship at the continuous time point and the corresponding conditional probability. The role is to provide causal reasoning basis for action evaluation, support subsequent traceability analysis of non-standard action and targeted adjustment of guidance scheme; the expert experience rule base includes but is not limited to action dependency relationship, joint linkage priority, and connection timing threshold.

[0068] It needs to be further explained that the implementation process of the improved association rule algorithm in the embodiment includes:

[0069] Based on the expert experience preset weight of different action association pairs, for example, the basic weight of lower limb action and core action, and the weighting coefficient of joint linkage action are specified. The modified support is obtained by multiplying the original item set support (the number of item set occurrences / total transaction number) by the corresponding expert weight. The modified confidence is obtained by multiplying the original confidence (the probability of the occurrence of the next action when the previous action occurs) by the product of the expert weight and the importance coefficient of the item set in the training process (such as the basic action coefficient being higher than the auxiliary action). The minimum modified support and confidence threshold are set. The historical continuous action evaluation data is scanned transaction by transaction to filter out the item set that meets the two thresholds at the same time as the frequent item set. The modified strong association rule is obtained to realize the accurate quantitative mining of the association relationship between continuous actions.

[0070] The image unit is used to construct the user training action image library by graph algorithm according to the user basic information, the user action evaluation information triple, the corresponding action space-time feature set, the standard action information knowledge base, the action association relationship at the continuous time point, and the probability of non-standard next action under the condition of non-standard previous action; the user training action image library includes the benchmark layer, the training layer and the reasoning layer;

[0071] The benchmark layer is used to store the user basic information; the training layer is used to store the training action space-time feature information of the user in each guidance scheme, the corresponding training evaluation score and the action guidance correction information; the reasoning layer is used to store the standard action information knowledge base and construct the causal reasoning connection between continuous actions in combination with the action association relationship at the continuous time point and the probability of non-standard next action under the condition of non-standard previous action to obtain the training action causal reasoning chain and simultaneously map to the training layer under the corresponding user to evaluate and reason the action information of the corresponding user.

[0072] It needs to be further explained that the detailed process of constructing the user training action image library in this embodiment includes:

[0073] Based on the user basic information converted into a 12-dimensional feature vector, the historical training evaluation information triple code is encoded into an entity relationship vector, the action space-time feature space is extracted into a 300-dimensional bone trajectory sequence, the action standard knowledge base rule is converted into a 200-dimensional constraint feature, the continuous action causal correlation and conditional probability are constructed into a 100-dimensional correlation matrix, three types of nodes are defined first: user node integrates basic information and historical evaluation vector, action node integrates space-time feature and training evaluation score, standard node encapsulates knowledge base rule and correlation probability. The edge connection is constructed by calculating the training frequency weight of the user and the action (training times / total times), the causal probability weight between actions, and the deviation weight of the action and the standard (Euclidean distance of actual feature and standard feature). Graph convolution network is used to aggregate neighbor features layer by layer. The first layer aggregates the interaction features of users and actions, the second layer integrates the causal correlation between actions, and the third layer combines the standard rule constraint. At the same time, through the attention mechanism, high-frequency training actions and strong causal correlation pairs are given higher weights. After 5 rounds of iteration and update of node embedding, the user node features are finally mapped to the benchmark layer, the action node sequence and evaluation data are integrated into the training layer, and the standard node rule and reasoning logic are constructed into the reasoning layer. A user training action image library containing three layers of structure is obtained; The 12-dimensional feature vector includes but is not limited to user age, gender, and body movement structured data; The causal probability weight between actions is the probability value that the non-standard of the previous action leads to the non-standard of the next action;

[0074] It needs to be further explained that the auxiliary adjustment module in this embodiment includes a space-time feature extraction unit and an evaluation unit.

[0075] The space-time feature extraction unit is configured to obtain the space-time correlation feature space of the user's optimal matching guidance scheme by combining the preset space-time graph convolution model with the training action information corresponding to the optimal matching guidance scheme.

[0076] The evaluation unit is configured to obtain the training evaluation score of each action in the optimal matching guidance scheme by combining the space-time correlation feature space of the user's optimal matching guidance scheme with the training action causal reasoning chain through a comprehensive evaluation algorithm.

[0077] It needs to be further explained that the traceability feedback module in this embodiment includes a reverse reasoning unit and a feedback adjustment unit.

[0078] a reverse reasoning unit configured to perform forward causal reasoning on the non-standard action according to the training evaluation score of each action in the optimal matching guidance scheme, the action spatiotemporal variation chain of the historical longitudinal non-standard action, and the causal reasoning connection in the training action causal reasoning chain, and obtain the action spatiotemporal variation chain of the real-time transverse non-standard action and the action spatiotemporal variation chain of the real-time longitudinal non-standard action according to the training evaluation score and the color mapping space of HSV;

[0079] By way of example, the color mapping space of HSV in the embodiment is specifically: the embodiment sets 0 points to correspond to pure red H=0, 100 points to correspond to pure green H=120, and the intermediate score is gradient transition hue in 5-point intervals, S and V are fixed at 1 to maintain saturation and brightness, and specific implementation is specifically set by a person skilled in the art according to use habits. The transverse chain is processed first: the non-standard actions are sorted according to the action execution order in the same training period, an ID and a non-standard feature label are added to each action node, the node is filled with HSV color corresponding to the score, and the color transparency of the connection line is adjusted according to the action correlation probability intensity, for example, the higher the intensity, the lower the transparency, to form a transverse time sequence chain. The longitudinal chain is processed next: the same non-standard action is sorted according to the time stamp of different training periods, a period number and a score change value are added to each period node, the node is filled with HSV color corresponding to the score, and adjacent nodes are connected by a hue gradient line to form a longitudinal trend chain, to obtain the action spatiotemporal variation chain of the real-time transverse and longitudinal non-standard action with color marking and clear node attributes. The action is to convert abstract score data into intuitive color chain, clearly present the same period action correlation and different period action evolution, help the instructor quickly locate the key non-standard action and change trend, and improve the correction efficiency.

[0080] a feedback adjustment unit configured to feed back the action spatiotemporal variation chain of the real-time transverse non-standard action and the action spatiotemporal variation chain of the real-time longitudinal non-standard action and the corresponding training evaluation score to the portrait unit for storage, and judge according to the comprehensive training evaluation score of the optimal matching guidance scheme and the preset evaluation threshold. If the comprehensive training evaluation score is greater than the preset evaluation threshold but any action does not meet the corresponding standard action information, the corresponding action information and the non-standard action trace information obtained by the transverse non-standard action spatiotemporal variation chain are fed back to the instructor of the corresponding teacher terminal, the corresponding action is guided and corrected, and if the comprehensive training evaluation score is less than or equal to the preset evaluation threshold, the corresponding guidance scheme information and the training evaluation score and the trace result are fed back to the corresponding instructor, and the optimal matching guidance scheme is adjusted in real time.

[0081] It needs to be further explained that the acquisition process of the training evaluation score in the embodiment is: according to the spatio-temporal feature extraction unit of the auxiliary adjustment module, the spatio-temporal correlation feature space of the training action information of the optimal matching guidance scheme is extracted by using a preset spatio-temporal graph convolution model, the evaluation unit compares the spatio-temporal correlation feature space with the training action causal reasoning chain, and the training evaluation score of each action is calculated by using a comprehensive evaluation algorithm; the comprehensive training evaluation score of the optimal matching guidance scheme is obtained by weighting and summing the training evaluation scores of all actions in the scheme according to the action importance weight (the core action weight is higher than the auxiliary action); the feedback adjustment unit compares the comprehensive training evaluation score with the preset evaluation threshold, and realizes the adjustment judgment of the guidance scheme or the action.

[0082] This process builds a complete training evaluation closed loop from information collection to scheme optimization through multi-module cooperation, bringing multi-dimensional practical value; the portrait module integrates user basic information, historical evaluation and standard knowledge base, and constructs a user training action portrait library by using multi-modal feature extraction and knowledge graph algorithm, which structures and correlates scattered information, provides comprehensive feature support for subsequent scheme matching, and makes every recommendation rooted in the user's unique exercise history and body characteristics; the strategy recommendation module filters the scheme through the correction scheme matching model based on the portrait library combined with real-time information, avoids the blindness of general schemes, and makes the recommended guidance scheme accurately fit the user's current action and improvement needs; the auxiliary adjustment module collects action images in real time and converts them into spatio-temporal features, generates training evaluation scores combined with standard reasoning chains, and realizes dynamic monitoring of the training process, so that non-standard actions can be captured in time. The traceability feedback module generates an intuitive non-standard action spatio-temporal change chain through reverse reasoning, which not only helps the instructor quickly locate the problem source, but also updates the feedback information to the portrait library in real time, promoting continuous optimization of the scheme. This whole process design from information integration, accurate recommendation, real-time evaluation to dynamic adjustment not only improves the accuracy of action evaluation and the pertinence of scheme matching, but also continuously approaches the user's training goal through closed-loop iteration, effectively shortens the action correction period, and enhances the stability and sustainability of the training effect.

[0083] It needs to be further explained that the construction process of the correction scheme matching model in the embodiment includes:

[0084] Based on the training action spatio-temporal feature information and the corresponding training evaluation score of each guidance scheme in the user training action portrait library, the action spatio-temporal feature and the training evaluation score change trend of the same training action in different training periods are extracted by the non-standard action layer in the correction scheme matching model, and the user longitudinal non-standard action feature change space and the training evaluation score fluctuation coefficient are obtained.

[0085] It needs to be further explained that the implementation process of the training evaluation score fluctuation coefficient in the embodiment is: based on the training evaluation scores of the same training action in different training periods, the scores are sorted by training period first to form a score sequence, the abnormal values deviating from the average of adjacent scores by more than twice are removed from the sequence, and then the arithmetic mean and sample standard deviation of the remaining scores are calculated, and the ratio of the sample standard deviation to the arithmetic mean is taken as the training evaluation score fluctuation coefficient to obtain the score fluctuation degree quantization value of the action in different periods; its role is to reflect the stability of the same action in long-term training, and the greater the fluctuation coefficient, the more unstable the score change, which provides a quantitative basis for identifying the characteristics of persistent non-standard actions and subsequent matching correction schemes.

[0086] Based on the user longitudinal non-standard action feature change space and the training evaluation score fluctuation coefficient, the action space-time change chain of the transverse non-standard action in the user training action portrait library is combined to obtain the associated coverage adjustment degree of the current non-standard action after adjustment to the same training period under the same training period;

[0087] It needs to be further explained that the specific implementation process of the associated coverage adjustment degree in the embodiment includes:

[0088] Based on the adjustment scheme of the current non-standard action in the user longitudinal non-standard action feature change space, including but not limited to action amplitude correction value, force timing adjustment parameter, training evaluation score fluctuation coefficient, combined with the associated action list in the action space-time change chain of the transverse non-standard action which has causal association with the current action within the same training period, optionally, the associated actions with a probability greater than or equal to 0.4 are selected by the causal probability matrix of the inference layer, the non-standard score vector of each associated action before adjustment is extracted, the theoretical score vector of the associated action after adjustment is calculated by inputting the adjustment parameter of the current action into the physical motion simulation model, and the score improvement vector of each associated action is calculated. The training evaluation score fluctuation coefficient is used to weight the improvement vector of each associated action, specifically, the associated action with a larger fluctuation coefficient is given a higher weight in the spatial dimension, and the ratio of the weighted improvement vector length sum to the total number of associated actions is taken as the associated coverage adjustment degree of the current non-standard action after adjustment under the same training period; its role is to accurately quantify the multi-dimensional improvement effect of the current action adjustment on the associated action in the same period, and to provide a quantitative basis for developing action combination adjustment strategy.

[0089] At the same time, according to the training evaluation score fluctuation coefficients of the associated actions under the corresponding associated coverage adjustment degree of the current non-standard action after adjustment in different training periods, the longitudinal effect transfer rate of the same associated action in different training periods is obtained.

[0090] It needs to be further explained that the implementation process of the longitudinal effect transfer rate in the embodiment includes:

[0091] Based on the effect evaluation results of the associated action after the adjustment of the current non-standard action under different training periods (i.e., the improvement degree of the subsequent action with causal relationship after the improvement of the current action, here the fluctuation coefficient of the training evaluation scores before and after the improvement of the associated action with causal relationship is used to measure), the evaluation data of the same associated action in three consecutive training periods are screened out. In this embodiment, the ratio of the spatial deviation improvement score of the action in the n+1 period to the spatial deviation improvement score in the n period, the ratio of the time deviation improvement score, and the ratio of the strength deviation improvement score are extracted first, then the three ratios are weighted by assigning a time attenuation coefficient to the period interval, the weighted ratio sum is divided by 3 as the longitudinal effect transfer rate, and the effect continuity quantitative value of the same associated action in different periods is obtained. The role is to quantify the transfer ability of the adjustment effect in cross-period training, and the higher the transfer rate indicates that the effect is more stable and persistent, which provides a basis for designing a coherent cross-period correction scheme.

[0092] Based on the training action in each training period, a horizontal action node sequence is constructed, and the horizontal association connection is constructed by using the association coverage adjustment degree of the associated action in the same training period after the adjustment of the current non-standard action in the same training period.

[0093] Based on the training evaluation score fluctuation coefficient of the same training action of the user in different training periods and the longitudinal effect transfer rate of the same associated action in different training periods, a longitudinal association transfer connection is constructed.

[0094] Based on the constructed horizontal action nodes of different training periods, the horizontal association connection and the longitudinal association transfer connection are combined to construct a topological space layer, and the user historical training action effect association transfer space is obtained.

[0095] It should be further explained that the construction process of the user historical training action effect association transfer space in this embodiment includes:

[0096] Based on different training cycles, the lateral action nodes (including but not limited to action ID, training evaluation score, space-time feature vector), lateral association connection (directed edge with association coverage adjustment degree weight), and longitudinal association transfer connection (cross-cycle edge with effect transfer rate weight) are connected. First, each training cycle is regarded as an independent two-dimensional plane. The lateral action nodes are sequentially arranged in the plane according to the training execution order. The lateral association connection is connected with the associated nodes in the same plane by an arrow line with a weight value. Specifically, the weight value is directly marked on the line. Then, the same action node is connected vertically between adjacent cycle planes by the longitudinal association transfer connection. The effect transfer rate is marked on the line to form a multi-layer three-dimensional topological skeleton. Then, the local topological strength of each node is calculated. The local topological strength is the ratio of the weight sum of the node association edge to the number of connected nodes. The top 20% of the nodes with local topological strength are selected as key nodes. The manifold learning algorithm is used to map the multi-layer topological skeleton to a three-dimensional space, maintain the distance ratio between the key nodes, and scale down the non-key nodes in proportion. Finally, the nodes in the three-dimensional space are gridded by Delaunay triangulation. The grid cell density is positively correlated with the connection weight to obtain the user historical training action effect association transfer space. The role is to convert the abstract action association relationship into a quantifiable three-dimensional topological structure, intuitively present the lateral action association strength and longitudinal effect transfer rule, provide the association basis for the spatial structure level of the subsequent matching guidance scheme, and improve the accuracy of the scheme matching.

[0097] Based on the user historical training action effect association transfer space and the teaching guidance strategy library, the matching layer configured with the maximum adjustment evaluation effect function is used for matching search to generate a matching guidance scheme sequence and a corresponding matching loss function. It needs to be further explained that the maximum adjustment evaluation effect function in the embodiment is constructed by fitting the maximum value of the standard correction evaluation score corresponding to the non-standard action and the associated action, the probability of the next action being non-standard under the condition that the previous action is non-standard, and the effect transfer rate combined with the linear fitting function after training the matching scheme.

[0098] It needs to be further explained that the specific implementation process of the matching layer for matching search in the embodiment includes:

[0099] Based on the correction standard evaluation score of non-standard action and associated action, the probability of non-standard action of the next action under the condition of non-standard action of the previous action, and the effect transfer rate, the importance level of the three is preset by expert experience, a linear combination function containing the three variables is constructed, the function is fitted with historical matching data (the error between the function value and the actual improvement is minimized by adjusting the variable coefficient), and the form corresponding to the maximum value of the fitted function is taken as the maximum adjustment evaluation effect function; based on the topological nodes of the user's historical training action effect associated transfer space and the scheme characteristics in the teaching guidance strategy library, the matching layer first selects the schemes matching the action types of the key nodes of the topological space from the strategy library as the initial candidate set, calculates the function value of each candidate scheme, and retains the schemes with higher function values; the action sequence of the retained guidance schemes is optimized, the function value is recalculated, and the optimization is repeated for several times to generate a matching guidance scheme sequence; the deviation between the actual function value of the optimized scheme and the fitted function value is taken as the matching loss function; its role is to quantify the improvement potential of the scheme through the constructed evaluation function, combined with multiple rounds of optimization search to select the scheme sequence that can maximize the improvement effect of non-standard action, taking into account the influence of associated action and cross-cycle effect transfer, providing high-quality scheme samples for model training, and improving the effectiveness of the final recommended scheme. The scheme characteristics in the teaching guidance strategy library include but are not limited to action type, training parameter;

[0100] Based on the matching guidance scheme sequence and the corresponding matching loss function, combined with the preset loss function threshold and the set model training period, the training is carried out, the trained correction scheme matching model is obtained, and the optimal matching guidance scheme corresponding to the maximum adjustment evaluation effect is output.

[0101] It needs to be further explained that the correction scheme matching model in the embodiment is constructed through multiple links in a progressive manner, forming a scheme generation mechanism that accurately adapts to the training needs of users, bringing significant practical value; the non-standard action layer accurately captures the stability difference of the same action in different periods by extracting longitudinal non-standard action features and training evaluation score fluctuation coefficients, providing quantitative basis for identifying persistent action defects and avoiding misjudgment of accidental fluctuations. The calculation of the correlation coverage adjustment degree quantifies the improvement range of the current action adjustment on the associated action in the same period, ensuring that single-action correction can drive overall training effect improvement and avoid local adjustment and disconnection with associated actions. The longitudinal effect transfer rate focuses on the continuity of the effect across periods, measuring the improvement transmission ability of the same associated action in different periods to support the design of coherent long-term guidance schemes and prevent short-term effects from being difficult to sustain. The user historical training action effect correlation migration space converts abstract action correlation into a perceptible three-dimensional topological structure, intuitively presenting the horizontal correlation strength and longitudinal migration law, providing a concrete basis for scheme matching at the spatial structure level. The matching layer selects the scheme sequence that can maximize the improvement effect from the teaching guidance strategy library by constructing the maximum adjustment evaluation effect function and combining multiple rounds of optimization search, taking into account the influence of associated actions and cross-period migration, ensuring that the scheme is both suitable for current needs and has long-term effectiveness. The entire construction process forms a complete closed loop from problem identification, influence quantification to scheme optimization, and through the synergistic effect of each link, the optimal matching guidance scheme output can accurately target non-standard actions, taking into account short-term improvement and long-term stability, effectively improving the efficiency of action correction and the sustainability of training effect.

[0102] It needs to be further explained that, referring to Figure 2 , the construction process of the spatiotemporal feature extraction model in the embodiment includes:

[0103] Obtain training action information and corresponding standard action image information in consecutive time periods in different training periods of the user and perform enhancement and denoising processing, and according to the processed images, combine the pre-trained expert marking algorithm based on bone node information to perform key bone node marking and gravity node marking under different actions; obtain a first continuous action marking image sequence in the same training period and a second marking image sequence corresponding to the same training action in different training periods;

[0104] Input the first continuous action marking image sequence in the same training period and the second marking image sequence corresponding to the same training action in different training periods into the OpenPose action estimation layer at the same time, obtain the continuous action node feature space in the same training period and the node feature space corresponding to the same training action in different training periods, and construct the node deviation feature space corresponding to the same training action in different training periods from the node feature space of the standard continuous action;

[0105] It needs to be further explained that the implementation process of the OpenPose action estimation layer in the embodiment includes:

[0106] Based on the processed first continuous action label image sequence of the same training period and the second label image sequence corresponding to the same training action of different training periods, the OpenPose action estimation algorithm first uses a feature extraction network to perform layer-by-layer feature coding on the image to obtain a multi-scale feature map containing the human body contour and joint candidate region; then a key point detection algorithm is used to predict the heat map (reflecting the node existence probability) and the limb correlation vector field (reflecting the connection strength between nodes) of each skeletal node on the feature map; based on the heat map, candidate nodes with high confidence are selected, and the candidate nodes are grouped into limbs in combination with the correlation vector field to form a complete skeletal node set of a single frame image, including node coordinates and connection relationships; the skeletal node sets of consecutive frames of images are associated in time sequence, and a continuous action node sequence of the same training period is constructed by using a frame-to-frame node matching algorithm in combination with node displacement continuity and limb length constraints, the spatiotemporal variation features of the nodes in the sequence are extracted, including but not limited to displacement trajectory and speed variation, and a continuous action node feature space of the same training period is obtained; at the same time, the node features of the same training action of different training periods are compared with the standard continuous action node feature space frame by frame, the node coordinate deviation and connection relationship deviation are calculated, and the node deviation feature space of the same training action of different training periods is integrated; the role is to accurately extract the spatiotemporal features of the skeletal nodes of the training action and the deviation from the standard action, to provide fine-grained node-level feature support for subsequent construction of continuous action topology space and analysis of action deviation trend, and to improve the accuracy of action recognition and evaluation.

[0107] Based on the continuous action node feature space of the same training period, the connection edges between nodes are constructed based on the body structure, and the corresponding undirected node graph of each image is obtained;

[0108] Based on the priority of the corresponding node in each image in the implementation of the limb action at consecutive time points in the corresponding undirected node graph of each image, the continuous action connection is constructed; for example, the A action is a hand-raising action, and the B action is an arm-bending action, the A action is implemented before the B action, and the continuous action connection between the same elbow joint nodes of the A action and the B action is constructed based on the implementation time priority of the A action and the B action with respect to the elbow joint nodes;

[0109] Based on the corresponding undirected node graph of each image and the continuous action connection, the topology space layer is used to construct the continuous action topology space;

[0110] It needs to be further explained that the specific implementation process of the topology space layer in the embodiment includes:

[0111] Based on the same training cycle of the undirected node graph (including the coordinates of each bone node and the connection edge constructed according to the body structure) and the continuous action connection (including the limb belonging relationship and the action implementation priority), the node coordinates of the single frame undirected node graph are first converted into three-dimensional vectors and the connection edge is converted into an adjacency matrix representing the node association through the topological algorithm of the topological space layer; the adjacency matrices of the continuous time points are arranged in time sequence, the displacement deviation of the nodes and the change intensity of the connection edge between adjacent frames are calculated, and the connection edge with a change intensity lower than the set standard is selected as the stable topological skeleton; the nodes are weighted according to the limb priority (such as the core limb higher than the limbs), and the weight of the core node is dynamically adjusted according to the action amplitude; the weighted node vector and the stable topological skeleton are mapped to the three-dimensional space, the clustering method based on the node distance is used to classify the nodes with similar action characteristics into a class, and the most closely related node in each class is reserved as the topological vertex; the path length (accumulated according to the connection edge weight) between the topological vertices is calculated, the connection with the shortest path length is used as the topological edge, and the topological network of the single frame action is constructed; the topological networks of each frame are connected in time axis, the topological vertex matching (based on the coordinate similarity) between frames is performed to maintain the time continuity, and the continuously changing continuous action topological space is formed; the role is to convert the discrete action nodes and connections into a structured space-time topological structure, retain the key action characteristics and time association, provide a unified space framework for subsequent analysis of action stage division, deviation change trend, and improve the spatio-temporal coherence of action evaluation and the accuracy of feature extraction.

[0112] Based on the continuous action topological space and the node deviation feature space corresponding to the same training action of different training cycles, the time trend extraction layer is used to obtain the continuous action topological space containing the action deviation change trend of the same action in different training cycles.

[0113] It should be further explained that the implementation mode corresponding to the time trend extraction layer of the embodiment is:

[0114] Based on the node coordinates of each frame of the continuous action topological space, the connection strength, and the node deviation features of the same action corresponding to different training periods, including but not limited to the node coordinate deviation, the connection angle deviation, and the time sequence deviation, through the time trend extraction layer configured time sequence convolution network algorithm, first, the topological space features and the deviation features are combined into a three-dimensional input sequence in the order of training period time, specifically, each time step contains 18 bone node coordinates, 12 limb connection strength values, and 6 deviation indicators, and the sequence is standardized; a time sequence convolution network containing 5 layers of causal convolution is constructed, the first two layers use small dilation rate (1 and 2 respectively) to extract short-term deviation fluctuation features of adjacent training frames (such as instantaneous change of node deviation within a single period), the middle two layers use medium dilation rate (4 and 8) to capture medium-term trend features within the same training period (such as deviation accumulation change of continuous 5 frames), and the last layer uses large dilation rate (16) to associate features of different training periods, integrates the outputs of each layer through residual connection, and captures long-term deviation change trend across periods (such as deviation improvement or deterioration trajectory of continuous 3 periods); the actual change curve of the historical deviation is used as the supervision label, the mean absolute error of the trend prediction value of the network output and the actual label is calculated, the convolution kernel parameters are updated through back propagation, and after the training is completed, the continuous action topological space sequence is input into the network, the deviation trend features of each node in different periods are output, including but not limited to trend slope, fluctuation amplitude, and stable interval, and these trend features are embedded into the node attributes of the original topological space, to obtain a continuous action topological space containing the action deviation change trend of the same action in different training periods; the role is to accurately capture the short-term fluctuation and long-term evolution law of the action deviation in the time dimension, to provide feature support at the time sequence level for identifying persistent action defects and evaluating the stability of the training effect, and to improve the accuracy of action deviation trend prediction.

[0115] Meanwhile, the continuous action topological space is input into the limb curvature recognition layer to divide the action stage, and the stage feature sequence of each action in the continuous action is obtained; the stage feature sequence is constructed by the node space coordinates corresponding to the starting time of the corresponding action, the node space coordinates corresponding to the turning time, and the node space coordinates corresponding to the ending time;

[0116] It should be further explained that the specific implementation process of the limb curvature recognition layer of the embodiment includes:

[0117] Based on the sequence of the bone node coordinates of each limb in the continuous action topological space, exemplarily, the shoulder-elbow-wrist node of the upper limb and the hip-knee-ankle node of the lower limb, by first performing smoothing processing on the node sequence of each limb, optionally, filtering high-frequency noise by using a sliding window average method, and setting the window size to 5 frames, then calculating the vector between adjacent nodes, exemplarily, the elbow node minus the shoulder node to obtain the shoulder-elbow vector, and the wrist node minus the elbow node to obtain the elbow-wrist vector, the cosine value of the included angle of the adjacent vectors is obtained by the vector dot product formula, and the angle value is converted as the limb joint angle; the joint angle sequence of the continuous frames is subjected to cubic spline curve fitting, and the curvature value of the fitting curve is calculated, specifically: curvature = angle change rate / limb length, the limb length is the sum of the Euclidean distance of the node sequence of the limb, and the real-time curvature change curve of each limb in the continuous action is obtained; the curvature threshold is set, specifically, the starting point threshold is the time when the curvature starts from 0 and continuously rises to 5%, the turning point threshold is the time when the absolute value of the curvature reaches the peak value, and the end point threshold is the time when the curvature falls back to within 10% of the initial value, by scanning the curvature change curve, the frame that first exceeds the starting point threshold is marked as the starting time, the frame corresponding to the curvature peak value is marked as the turning time, and the frame that first falls below the end point threshold is marked as the end time; the node space coordinates corresponding to the three time points are extracted, combined in time sequence to form a stage feature sequence, and the action stage is divided according to the time interval of the key frame; exemplarily, the first stage is from the starting time to the turning time, and the second stage is from the turning time to the end time, if there are multiple turning times, the stages are subdivided according to the number of peaks; the role is to quantify the curvature change of the limb deformation, accurately locate the key stage node of the action, provide fine-grained time sequence feature support for subsequent analysis of the action standard degree of each stage and identification of stage defects, and improve the objectivity and accuracy of the action stage division.

[0118] Based on the root node and the gravity node in each action corresponding node in the continuous action topological space, and a preset node division strategy, a region node feature set in the undirected node graph corresponding to each image is obtained.

[0119] It should be further explained that the implementation process of the preset node division strategy in the embodiment includes:

[0120] Based on the root node coordinates of each image in the continuous action topology space, for example, the hip joint node, the center of gravity node coordinates, the preset space-time reference point of all joint nodes and other joint node coordinates, the average value of the coordinates of all joint nodes in the continuous 5 frames of images is calculated first, and the average value is taken as the stable coordinates of the skeleton center of gravity; the root node (in this embodiment, the joint node with the most connected nodes in the topology space, such as the hip joint, is selected as the root node) is determined, the directly adjacent nodes of the root node are extracted, for example, the waist, left thigh and right thigh nodes; the straight line distance from the root node to the skeleton center of gravity is calculated, and the straight line distance from each adjacent node to the skeleton center of gravity is calculated respectively; the distances from the adjacent nodes to the center of gravity are compared: the adjacent nodes with a distance less than the straight line distance from the root node to the skeleton center of gravity are classified into the centripetal group, the adjacent nodes with a distance greater than the straight line distance from the root node to the skeleton center of gravity are classified into the centrifugal group, and the root node itself is separately classified into subset 0; a three-dimensional partition mask is generated for each node, wherein the three dimensions correspond to the three subsets respectively, the dimension value of the subset to which the node belongs is set to 1, and the other two dimension values are set to 0; the spatial coordinates of each node, the connection strength with the adjacent nodes and the deviation features in the topology space are extracted, these features are associated and combined with the corresponding three-dimensional partition mask according to the node ID, the regional feature item of a single node is formed, the regional feature items of all nodes are integrated, and the regional node feature set in the undirected node graph corresponding to each image is obtained; the function is to partition the nodes according to the relationship between the spatial position and the center of gravity, and clearly distinguish the functional attributes of the nodes in different regions. In this embodiment, the centripetal group nodes mainly affect the action stability, and the centrifugal group nodes mainly affect the action amplitude, which provides structured partition feature support for subsequent analysis of the action standard degree of each regional node and identification of regional action defects, and improves the regional pertinence of action evaluation.

[0121] The continuous action topology space containing the action deviation change trend of the same action in different training periods, the stage feature sequence of each action in the continuous action and the regional node feature set are input into the feature fusion layer constructed by the convolution attention network to obtain a continuous action fusion topology space.

[0122] It should be further explained that the implementation process of the feature fusion layer in this embodiment includes:

[0123] Based on the continuous action topology space including the same action in different training cycle action deviation change trend, including but not limited to node coordinates, deviation trend slope, fluctuation amplitude, stage feature sequence of each action in the continuous action, including but not limited to node coordinates, time stamp at the start, turning end time, and regional node feature set, including but not limited to node partition mask, spatial coordinates, connection strength, three types of features are first converted into a unified dimension tensor through a feature fusion layer: the topology space feature is expanded into a three-dimensional tensor according to the time frame, the stage feature sequence is embedded into the feature channel of the corresponding frame according to the time stamp, and the regional node feature set is spliced into a spatial partition tensor through the partition mask and the node feature; a convolutional attention network is constructed, the first layer uses a 1*1 convolution kernel to reduce the dimension of the topology space tensor to retain key deviation trend features, and a 3*3 convolution is performed on the regional partition tensor to extract local spatial correlation features; the second layer introduces a channel attention mechanism to calculate the importance weight of each feature channel, in this embodiment, the deviation trend feature is assigned a weight of 0.4, the stage feature is assigned a weight of 0.3, and the regional feature is assigned a weight of 0.3, the weighted fusion of the features after convolution is performed through global average pooling and sigmoid activation; the third layer uses a self-attention mechanism to capture the time sequence correlation across frames, for example, the stage feature connection strength of adjacent frames, an attention map is generated by calculating the inter-frame feature similarity matrix to highlight the key frame features; finally, the weighted fused features are combined with the original topology space structure, each feature item is associated according to the node ID, and a three-dimensional topology structure including multi-dimensional feature correlation is reconstructed to obtain a continuous action fusion topology space; the role is to integrate the spatial topology, time sequence stage and regional partition features through the convolutional attention network, highlight the correlation weight of the key features, provide more comprehensive fusion feature support for subsequent action evaluation, and improve the accuracy and robustness of the action standard degree judgment.

[0124] Based on the continuous action fusion topology space combined with the standard continuous action node feature space, the output layer is used to obtain the continuous action node coordinate sequence and the node coordinate deviation loss, and the deviation change trend and the trend confidence of the corresponding action; at the same time, the continuous action fusion topology space combined with the standard continuous action node feature space is input into the association causal layer to obtain the forward cumulative contribution path and the path confidence corresponding to the non-standard action; in this embodiment, the output layer is preferably realized by a support vector machine algorithm;

[0125] The loss function constructed based on the node coordinate deviation loss, the trend confidence and the path confidence is trained in combination with the preset loss function threshold and the training cycle to obtain a trained spatio-temporal feature extraction model.

[0126] It needs to be further explained that the specific implementation mode of the output layer in this embodiment is:

[0127] Based on the fusion topology space of continuous action (including but not limited to node coordinates, deviation trend, stage characteristics, and region attributes) and the standard continuous action node feature space (including but not limited to standard coordinates, time sequence template, and space constraint), the node features of the fusion topology space are flattened into a one-dimensional vector through the support vector machine algorithm configured by the output layer, and the standard continuous action node feature space is processed synchronously as a reference vector; a multi-output support vector regression machine is constructed, an independent regression model is trained for each node, a radial basis function is used as the kernel function, a regularization parameter is optimized through cross-validation, a mapping relationship between the current fusion topology space node features and the standard coordinates is fitted with the standard coordinates as the target value, and a continuous action node coordinate sequence is output; the Euclidean distance between the predicted coordinates and the standard coordinates is calculated as the node coordinate deviation loss, the 95% confidence interval of the loss is estimated through Bootstrap resampling based on the deviation distribution of the training samples, the rate of change of the deviation loss in the time dimension is taken as the deviation change trend of the action, and the stability of the trend is taken as the trend confidence; the role is to accurately predict the action node coordinates and quantify the deviation through the nonlinear mapping capability of the support vector machine, and to provide coordinate-level guidance for action correction.

[0128] It needs to be further explained that the specific implementation mode of the association causal layer in the embodiment is:

[0129] Based on the difference features of the fusion topology space of continuous action and the standard continuous action node feature space, a causal inference algorithm configured by the association causal layer is used to first construct an action node causal graph, wherein the nodes are action components, the edges are causal relationships, and the weights are conditional probabilities; the conditional dependence relationship between the action components is counted from the historical training data, for example, the probability that the abnormal knee joint angle leads to hip joint compensation; the current difference features are taken as intervention variables, the causal effect of each difference on the subsequent action is calculated through counterfactual reasoning, for example, the influence degree of the current frame shoulder joint position deviation on the next frame elbow joint trajectory, and the causal effect values of each node are accumulated to form a forward cumulative contribution path; the path confidence is evaluated by combining the prior causal knowledge and the current data likelihood through the Bayesian network, the posterior probability of each causal relationship on the path is calculated, and the product of the posterior probabilities of all relationships on the path is taken as the path confidence; the role is to locate the root and propagation path of the non-standard action through causal inference, and to provide a theoretical basis for formulating a targeted correction strategy.

[0130] Embodiment 2

[0131] Another embodiment provided by the application is an interactive teaching evaluation auxiliary device based on multi-modal fusion, which comprises a data acquisition component and a processor arranged in the device, and an imaging module is integrated in the data acquisition component; a strategy recommendation module, an auxiliary adjustment module, and a traceability feedback module are integrated in the processor.

[0132] The data collection component is used to collect the user basic information, user historical training compliance information, corresponding guidance scheme information and standard action information knowledge base of the student terminal, and obtain the user training action image library by combining the built-in multi-modal feature extraction algorithm and knowledge graph algorithm;

[0133] The processor generates an optimal matching guidance scheme through a correction scheme matching model based on the user training action image library, the real-time collected user basic information and the configured teaching guidance strategy library, and when the user trains according to the scheme, real-time training action images are collected, and continuous correction action sequences after guidance and corresponding correction standard evaluation scores are obtained by combining a time-space feature extraction model and an action evaluation algorithm; according to the continuous correction action sequences after guidance and the corresponding correction standard evaluation scores and the standard action information knowledge base, a transverse and longitudinal time-space change chain of non-standard actions is obtained through a forward reasoning model, and the action time-space change chain of the non-standard actions is fed back to the configured teacher terminal, while the teacher terminal feedback is collected to adjust the matching guidance scheme in real time until the corresponding user training action meets the conditions, and the guidance scheme of the user training action meeting the conditions is stored in the standard action information knowledge base for management.

[0134] The embodiments of the present application are described above in combination with the drawings, but the present application is not limited to the above specific embodiments, and the above specific embodiments are only illustrative but not limiting, and those skilled in the art can make changes, modifications, replacements and variations to the above embodiments without departing from the purpose of the present application and the scope protected by the claims, which are all within the protection of the present application.

[0135] If the technical solution of the present disclosure involves personal information, the product applying the technical solution of the present disclosure has been explicitly informed of the personal information processing rules before processing the personal information, and has obtained the personal independent consent. If the technical solution of the present disclosure involves sensitive personal information, the product applying the technical solution of the present disclosure has obtained the personal independent consent before processing the sensitive personal information, and at the same time meets the requirement of "explicit consent". For example, at the personal information collection device such as camera, a clear and prominent sign is set to inform that the personal information collection range has been entered, and the personal information will be collected. If the individual voluntarily enters the collection range, it is considered to agree to collect the personal information; or on the device for processing personal information, the personal information processing rules are informed by using obvious signs / information, and the personal authorization is obtained by pop-up information or asking the individual to upload the personal information by himself / herself; wherein, the personal information processing rules can include personal information processor, personal information processing purpose, processing method and personal information type, etc.

Claims

1. An interactive teaching evaluation auxiliary system based on multimodal fusion, characterized by: include: Profiling module, strategy recommendation module, auxiliary adjustment module, and traceability feedback module; The portrait module is used to collect basic information of users who configure the student terminal, user historical training standard information, and corresponding guidance program information and standard action information knowledge base, and combine it with the knowledge graph algorithm to obtain a user training action portrait library; The strategy recommendation module is used to obtain the optimal matching guidance plan for the corresponding user through the correction plan matching model based on the user training action portrait library combined with the real-time collected user basic information and the configured teaching guidance strategy library; The auxiliary adjustment module is used to provide auxiliary guidance according to the matching guidance plan, and collect the training movement information of the student end in real time, combining it with the preset spatiotemporal feature extraction model and comprehensive evaluation algorithm to obtain the continuous movement correction space after guidance and the corresponding correction standard evaluation score; The traceability feedback module is used to obtain the spatiotemporal change chain of non-standard actions in the corresponding teaching scenario based on the continuous corrective action sequence after guidance and the corresponding correction standard evaluation score in combination with the standard action information knowledge base through a forward reasoning model; and feed back the spatiotemporal change chain of non-standard actions to the alarm device configured on the teacher side for real-time warning, and collect feedback from the teacher side to make real-time adjustments to the matching guidance plan until the corresponding user training action meets the conditions, and at the same time store the guidance plan of the user training action that meets the conditions in the standard action information knowledge base for management.

2. The interactive teaching evaluation auxiliary system based on multimodal fusion according to claim 1 is characterized in that: The spatiotemporal change chain of the non-standard movements includes a spatiotemporal change chain of horizontal non-standard movements constructed by different training movements in the same guidance scheme in the current training cycle and a spatiotemporal change chain of vertical non-standard movements constructed by the same training movement in the same guidance scheme in different training cycles; the portrait module includes a data acquisition unit, a first data analysis unit and a second data analysis unit; The data collection unit is used to collect basic information of users configuring the student terminal, text and image information of users' historical sports evaluation movements, and corresponding guidance program information and standard movement information knowledge base, and classify and arrange the collected information according to users and collection timestamps and perform standardization preprocessing; The first data analysis unit is used to obtain a user action evaluation information triple and a corresponding action spatiotemporal feature set based on the user's historical motion evaluation action text information and image information in combination with an entity relationship extraction algorithm and an image recognition algorithm; The second data analysis unit is used to obtain the action association relationship at consecutive time points and the probability of a subsequent action being non-standard under the condition that the previous action is non-standard based on the standard action information knowledge base combined with the association analysis algorithm with built-in expert experience.

3. The interactive teaching evaluation auxiliary system based on multimodal fusion according to claim 2 is characterized in that: The portrait module also includes a portrait unit; The profiling unit is used to construct a user training action profile library through a graph algorithm based on the user basic information, the user action evaluation information triples and the corresponding action spatiotemporal feature set, the standard action information knowledge base, the action association relationship at consecutive time points, and the probability of the next action being non-standard under the condition that the previous action is non-standard; the user training action profile library includes a reference layer, a training layer and an inference layer; The base layer is used to store basic user information; The training layer is used to store the spatiotemporal feature information of the training actions in each guidance plan in the user's history, as well as the corresponding training evaluation scores and action guidance correction information; the reasoning layer is used to store the standard action information knowledge base and combine the action association relationship at consecutive time points and the probability of the subsequent action being non-standard under the condition that the previous action is non-standard to construct a causal reasoning connection between consecutive actions, obtain the causal reasoning chain of training actions, and at the same time map it to the training layer under the corresponding user to evaluate and reason and correct the action information of the corresponding user.

4. The interactive teaching evaluation auxiliary system based on multimodal fusion according to claim 3 is characterized in that: The auxiliary adjustment module includes a spatiotemporal feature extraction unit and an evaluation unit; The spatiotemporal feature extraction unit is used to obtain the associated spatiotemporal feature space of the user's optimal matching guidance solution based on the training action information corresponding to the optimal matching guidance solution and the preset spatiotemporal graph convolution model; The evaluation unit is used to obtain a training evaluation score for each action in the optimal matching guidance scheme through a comprehensive evaluation algorithm based on the spatiotemporal correlation feature space of the user's optimal matching guidance scheme and the causal reasoning chain of the training action.

5. The interactive teaching evaluation auxiliary system based on multimodal fusion according to claim 4 is characterized in that: The traceability feedback module includes a reverse reasoning unit and a feedback adjustment unit; The reverse reasoning unit is used to perform forward causal reasoning of non-standard actions based on the training evaluation score of each action in the optimal matching guidance scheme, the action spatiotemporal change chain of historical longitudinal non-standard actions, and the causal reasoning connection in the training action causal reasoning chain, and obtain the action spatiotemporal change chain of real-time horizontal non-standard actions and the action spatiotemporal change chain of real-time longitudinal non-standard actions based on the training evaluation score and the HSV color mapping space; The feedback adjustment unit is used to feed back the real-time spatiotemporal change chain of non-standard horizontal movements and the real-time spatiotemporal change chain of non-standard vertical movements and the corresponding training evaluation scores to the portrait unit for storage, and at the same time make a judgment based on the comprehensive training evaluation score of the optimal matching guidance plan and the preset evaluation threshold. If the comprehensive training evaluation score is greater than the preset evaluation threshold but there is any movement that does not meet the corresponding standard movement information, the corresponding movement information and the non-standard movement traceability information currently obtained through the spatiotemporal change chain of non-standard horizontal movements are fed back to the instructor on the corresponding teacher side to guide and correct the corresponding movement. If the comprehensive training evaluation score is less than or equal to the preset evaluation threshold, the corresponding guidance plan information, training evaluation score and traceability result are fed back to the corresponding instructor to make real-time adjustments to the optimal matching guidance plan.

6. The interactive teaching evaluation auxiliary system based on multimodal fusion according to claim 5 is characterized in that: The process of constructing the correction scheme matching model includes: Based on the spatiotemporal feature information of the training movements and the corresponding training evaluation scores in each historical guidance plan in the user training movement portrait library, the spatiotemporal features and training evaluation score change trends of the same training movement in different training cycles are extracted through the non-standard movement layer in the correction plan matching model to obtain the longitudinal non-standard movement feature change space and training evaluation score fluctuation coefficient of the user; Based on the user's longitudinal non-standard movement feature change space and training evaluation score fluctuation coefficient combined with the movement spatiotemporal change chain of the horizontal non-standard movement in the user's training movement portrait library, the degree of correlation coverage adjustment of the associated movements in the same training cycle after the current non-standard movement is adjusted is obtained; At the same time, according to the fluctuation coefficient of the training evaluation score of the associated action under the corresponding degree of associated coverage adjustment in different training cycles after the current non-standard action is adjusted in different training cycles, the longitudinal effect migration rate of the same associated action in different training cycles is obtained.

7. The interactive teaching evaluation auxiliary system based on multimodal fusion according to claim 6, characterized in that: The process of constructing the correction scheme matching model also includes: Based on the training actions in each training cycle, a horizontal action node sequence is constructed, and the horizontal association connection is constructed using the association coverage adjustment degree of the associated actions in the same training cycle after the current non-standard action is adjusted; Based on the fluctuation coefficient of the training evaluation score of the same training action in different training cycles and the longitudinal effect migration rate of the same associated action in different training cycles, a longitudinal association transfer connection is constructed; Based on the horizontal action nodes of different training cycles, the horizontal association connection and the vertical association transfer connection are combined with the topological space layer to obtain the association transfer space of the user's historical training action effect; Based on the user's historical training action effect associated migration space combined with the teaching guidance strategy library, a matching search is performed through a matching layer configured with a maximum adjustment evaluation effect function to generate a matching guidance scheme sequence and a corresponding matching loss function; the maximum adjustment evaluation effect function is a function constructed by fitting a linear fitting function with the correction standard evaluation score of the corresponding non-standard action and the associated action after matching scheme training, the probability of the subsequent action being non-standard under the condition of the previous action being non-standard, and the effect migration rate, and taking the maximum value; Based on the matching guidance scheme sequence and the corresponding matching loss function, combined with the preset loss function threshold and the set model training cycle, training is performed to obtain the trained correction scheme matching model, and the optimal matching guidance scheme corresponding to the maximum adjustment evaluation effect is output.

8. The interactive teaching evaluation auxiliary system based on multimodal fusion according to claim 6, characterized in that: The construction process of the spatiotemporal feature extraction model includes: The training action information and corresponding standard action image information of the user in continuous time periods in different training cycles are obtained and enhanced and denoised. Based on the processed images and the expert labeling algorithm pre-trained with the skeleton node information, key skeleton nodes and center of gravity nodes under different actions are labeled; a first continuous action labeled image sequence of the same training cycle and a second labeled image sequence corresponding to the same training action in different training cycles are obtained; The first continuous action labeled image sequence of the same training cycle and the second labeled image sequence corresponding to the same training action in different training cycles are simultaneously input into the OpenPose action estimation layer to obtain the continuous action node feature space of the same training cycle and the node deviation feature space corresponding to the same training action in different training cycles constructed by combining the node feature space corresponding to the same training action in different training cycles with the standard continuous action node feature space; Based on the continuous action node feature space of the same training cycle and combined with the body structure, the connection edges between each node are constructed to obtain the undirected node graph corresponding to each image; Based on the priority of the limb actions realized by the corresponding nodes in each image at consecutive time points in the undirected node graph corresponding to each image, a continuous action connection is constructed; Based on the undirected node graph and continuous action connections corresponding to each image, a continuous action topological space is constructed through the topological space layer.

9. The interactive teaching evaluation auxiliary system based on multimodal fusion according to claim 8, characterized in that: The construction process of the spatiotemporal feature extraction model also includes: Based on the continuous action topological space and the node deviation feature space corresponding to the same training action in different training cycles, the continuous action topological space containing the action deviation change trend of the same action in different training cycles is obtained through the time trend extraction layer; At the same time, the continuous action topological space is input into the limb curvature recognition layer to divide the action into stages, and a stage feature sequence of each action in the continuous action is obtained; the stage feature sequence is constructed by the node space coordinates corresponding to the starting moment of the corresponding action, the node space coordinates corresponding to the turning moment, and the node space coordinates corresponding to the ending moment; Based on the root node and the center node of each action corresponding node in the continuous action topological space and a preset node partitioning strategy, a regional node feature set in the undirected node graph corresponding to each image is obtained; The continuous action topological space containing the trend of action deviation changes of the same action in different training cycles, the stage feature sequence of each action in the continuous action, and the regional node feature set are input into the feature fusion layer constructed by the convolutional attention network to obtain the continuous action fusion topological space; Based on the continuous action fusion topological space combined with the standard continuous action node feature space, the continuous action node coordinate sequence and node coordinate deviation loss and the corresponding action deviation change trend and trend confidence are obtained through the output layer; at the same time, the continuous action fusion topological space combined with the standard continuous action node feature space is input into the association causal layer to obtain the forward cumulative contribution path and path confidence corresponding to the non-standard action; The loss function constructed based on node coordinate deviation loss, trend confidence and path confidence is combined with the preset loss function threshold and training cycle for training to obtain a trained spatiotemporal feature extraction model.

10. An interactive teaching evaluation auxiliary device based on multimodal fusion, which is implemented based on the interactive teaching evaluation auxiliary system based on multimodal fusion according to any one of claims 1 to 9, characterized in that: include: A data acquisition component and a processor are configured in the device, wherein the data acquisition component is integrated with a portrait module; The processor is integrated with a strategy recommendation module, an auxiliary adjustment module and a traceability feedback module; The data collection component is used to collect basic information of users configuring the student terminal, user historical training standard information and corresponding guidance program information and standard action information knowledge base, and combine the built-in multimodal feature extraction algorithm and knowledge graph algorithm to obtain the user training action portrait library; The processor generates an optimal matching guidance scheme through a correction scheme matching model based on a user training action portrait library, real-time collected user basic information, and a configured teaching guidance strategy library. When the user trains according to this scheme, the processor collects training action images in real time, combines the spatiotemporal feature extraction model with the action evaluation algorithm to obtain a continuous correction action sequence after guidance and a corresponding correction standard evaluation score; based on the continuous correction action sequence after guidance and the corresponding correction standard evaluation score and the standard action information knowledge base, the processor obtains the horizontal and vertical spatiotemporal change chains of non-standard actions through a forward reasoning model, and feeds back the spatiotemporal change chains of non-standard actions to the configured teacher end, and at the same time collects feedback from the teacher end to adjust the matching guidance scheme in real time until the corresponding user training action meets the conditions, and stores the guidance scheme of the user training action that meets the conditions in the standard action information knowledge base for management.

Images