Action evaluation method and device based on large model, equipment and medium

Abstract

The invention relates to an action evaluation method and device based on a large model, equipment and a medium. The method comprises the following steps: acquiring action image data and multi-source sensing data of an action evaluation object; the multi-source sensing data are acquired by a plurality of sensors at the position of the action evaluation object; according to a preset environment meta-knowledge base and the multi-source sensing data, obtaining environment influence parameters corresponding to the multi-source sensing data; on the basis of a generative adversarial network, correcting the action image data through the environmental influence parameters to obtain corrected action image data; and obtaining an action evaluation result by adopting a support vector machine classifier according to the corrected action image data. The method can identify the dynamic change of environmental parameters, and improves the success rate of action evaluation.

Description

Technical Field

[0001] The present invention belongs to the field of motion assessment technology, and in particular relates to a motion assessment method, device, equipment and medium based on a large model. Background Art

[0002] With the development of motion assessment technology, intelligent motion assessment technology based on large models has emerged. This technology can quickly and efficiently give motion assessment results based on the motion pictures provided by the user. It has been initially applied in sports training, medical rehabilitation and other fields.

[0003] The current large-model-based intelligent motion assessment technology, with deep learning as its core technical route, can extract spatiotemporal features from animated image sequences through end-to-end multimodal data processing, and combine with the expert knowledge base to quantitatively score the standardization, fluency and biomechanical rationality of the movements.

[0004] However, the current intelligent action evaluation system based on large models is limited by the static assumption of the environment. In open scenarios (such as robot interaction and autonomous driving), the dynamic changes of environmental parameters will cause the performance of the evaluation model to degrade and reduce the success rate of action evaluation. Summary of the Invention

[0005] Based on this, it is necessary to provide a large-model-based motion evaluation method, device, equipment and medium that can identify dynamic changes in environmental parameters and improve the success rate of motion evaluation in response to the above technical problems.

[0006] In a first aspect, the present application provides an action evaluation method based on a large model, comprising:

[0007] Acquiring motion image data and multi-source sensor data of the motion assessment object; the multi-source sensor data is collected from the location of the motion assessment object;

[0008] According to the preset environmental meta-knowledge base and multi-source sensor data, the environmental impact parameters corresponding to the multi-source sensor data are obtained;

[0009] Based on the generative adversarial network, the action image data is corrected by the environmental impact parameters to obtain the corrected action image data;

[0010] According to the corrected action image data, the support vector machine classifier is used to obtain the action evaluation results.

[0011] Furthermore, based on the preset environmental meta-knowledge base and the multi-source sensor data, environmental impact parameters corresponding to the multi-source sensor data are obtained, including:

[0012] Using principal component analysis, the dominant features in multi-source sensor data are extracted to obtain dominant feature data and sensor feature vectors.

[0013] According to the dominant feature data, multi-source sensor data and sensor feature vector, the multi-source sensor data is integrated and heterogeneous to obtain preliminary fusion feature data;

[0014] According to the preset environmental meta-knowledge base, the preliminary fused feature data are physically constrained and quantified to obtain the environmental impact parameters.

[0015] Furthermore, principal component analysis is used to extract the dominant features from the multi-source sensor data, and the dominant feature data and sensor feature vectors are obtained, including:

[0016] Performing standardized calculation on multi-source sensor data to obtain standardized sensor data;

[0017] Calculate the covariance of the standardized sensor data to obtain the sensor covariance matrix;

[0018] According to the sensor covariance matrix, the eigenvector of the sensor covariance matrix is ​​calculated to obtain the sensor eigenvector;

[0019] Use the following formula to select the standardized sensor data corresponding to the sensor feature vector that meets the preset cumulative contribution rate to form the dominant feature data:

[0020]

[0021] Among them, K is the rank of the data corresponding to the preset cumulative contribution rate in the standardized sensor data, min is the function of selecting the minimum value, t is the rank of the data in the standardized sensor data, λ i is the i-th sensor feature vector, d is the dimension of multi-source sensor data, and N is the cumulative contribution rate.

[0022] Furthermore, based on the dominant feature data, multi-source sensor data and sensor feature vectors, the multi-source sensor data is heterogeneously integrated to obtain preliminary fused feature data, including:

[0023] The contribution weight of the dominant feature data is calculated based on the sensing feature vector using the following formula:

[0024]

[0025] Among them, w n is the contribution weight of the nth dominant feature data, λ n is the nth sensor feature vector, K is the rank of the data corresponding to the preset cumulative contribution rate in the standardized sensor data, and λ j is the jth sensing feature vector;

[0026] The following formula is used to calculate the preliminary fusion feature data based on the contribution weight, dominant feature data, and multi-source sensor data:

[0027]

[0028] Among them, f i is the preliminary fusion feature data, K is the rank of the data corresponding to the preset cumulative contribution rate in the standardized sensor data, and w j is the contribution weight of the jth dominant feature data, f j is the jth dominant feature data, α is the retention coefficient, x j is the jth multi-source sensor data.

[0029] Furthermore, based on the preset environmental meta-knowledge base, the preliminary fused feature data is physically constrained and quantified to obtain environmental impact parameters, including:

[0030] According to the preliminary fusion feature data, match the environment meta-knowledge base to obtain the corresponding physical constraint set;

[0031] Combine the preliminary fused feature data with the corresponding physical constraint set to form a feature selection matrix;

[0032] Use the following formula to get the feature influence probability based on the feature selection matrix and multilayer perceptron:

[0033] p i =σ(W2·ReLU(W1f i +b1)+b2)

[0034] Among them, p i is the feature influence probability of the i-th feature, σ is the activation function, W2 is the output layer matrix, W1 is the hidden layer matrix, f i is the i-th feature subset in the feature selection matrix, b1 is the hidden layer bias term, and b2 is the output layer bias term;

[0035] The environmental impact parameter is obtained by combining the characteristic impact probabilities using the following formula:

[0036]

[0037] Among them, p f is the environmental impact parameter, m is the number of feature subsets contained in the feature selection matrix, w j is the weight of the meta-knowledge base of the jth feature, p j is the feature influence probability of the j-th feature.

[0038] Furthermore, based on the generative adversarial network, the action image data is corrected by the environmental impact parameter to obtain the corrected action image data, including:

[0039] Normalizing the environmental impact parameters to obtain normalized environmental impact parameters;

[0040] Use the following formula to encode the normalized environmental impact parameter to obtain the environmental impact vector:

[0041] h P =σ(W p P n +b p )

[0042] Among them, h P is the environmental influence vector, σ is the activation function, W p is the influence weight, P n is the normalized environmental impact parameter, b p is the influencing bias term;

[0043] Extract multi-scale features of action image data to obtain multi-layer action feature maps;

[0044] According to the dimension of the multi-layer motion feature map, the normalized environmental impact parameter is expanded to obtain the environmental impact parameter of the same dimension;

[0045] Use the following formula to connect the same-dimensional environmental impact parameters with the action feature map to obtain the hierarchical action feature map;

[0046] f l =Concat(W l +h s )

[0047] Among them, f l is the l-th layered action feature map, Concat is the connection function, W l is the l-th layer action feature map, h s is the environmental impact parameter of the same dimension;

[0048] The layered action feature map is deconvolved to obtain the corrected action image data.

[0049] Furthermore, based on the corrected action image data, a support vector machine classifier is used to obtain action evaluation results, including:

[0050] The Z-score algorithm is used to standardize the corrected action image data to obtain standard action feature data;

[0051] Use the radial basis kernel function to map the standard motion feature data to a high-dimensional space to obtain high-dimensional motion feature data:

[0052] Use the following formula to classify high-dimensional action feature data and obtain the action probability output:

[0053]

[0054] Among them, P(y=1|f) is the action probability output of the action category, A is the fitting parameter, N is the dimension of the high-dimensional action feature data, α i is the i-th Lagrange multiplier, y i is the action probability output of the i-th high-dimensional action feature data, K(X i ,X) is the i-th high-dimensional action feature data, B is the bias term;

[0055] Use the following formula to get the action evaluation result based on the action probability output:

[0056]

[0057] in, is the action evaluation result, Select the maximum value function of K, P k (y=1|f) is the k-th action probability output.

[0058] In a second aspect, the present application further provides a large-model-based action evaluation device, comprising:

[0059] A data acquisition module, used to acquire motion image data and multi-source sensor data;

[0060] A parameter determination module is used to obtain environmental impact parameters corresponding to the multi-source sensor data based on a preset environmental meta-knowledge base and the multi-source sensor data;

[0061] An image correction module is used to correct the action image data using environmental impact parameters based on a generative adversarial network to obtain corrected action image data;

[0062] The result evaluation module is used to obtain the action evaluation result by using the support vector machine classifier based on the corrected action image data.

[0063] In a third aspect, the present application further provides a computer device comprising a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, it implements any large model-based action evaluation method described in the first aspect of the present application.

[0064] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements any of the large model-based action evaluation methods described in the first aspect of the present application.

[0065] The above-mentioned large-scale model-based action evaluation method, device, equipment and medium obtain action image data and multi-source sensor data of the action evaluation object; the multi-source sensor data is collected from the location of the action evaluation object; based on a preset environmental meta-knowledge base and multi-source sensor data, environmental impact parameters corresponding to the multi-source sensor data are obtained; based on a generative adversarial network, the action image data is corrected by the environmental impact parameters to obtain corrected action image data; based on the corrected action image data, a support vector machine classifier is used to obtain an action evaluation result, thereby combining the environmental parameter changes of the environment where the action evaluation object is located with the action image data to obtain a corrected image, and performing action evaluation based on the corrected image that is more consistent with the actual action, thereby improving the success rate of action evaluation. BRIEF DESCRIPTION OF THE DRAWINGS

[0066] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the following briefly introduces the drawings required for use in the embodiments or related technical descriptions. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0067] Figure 1 A flowchart of a large model-based action evaluation method provided as an exemplary embodiment of the present application;

[0068] Figure 2 A structural block diagram of an action evaluation device based on a large model provided as an exemplary embodiment of the present application;

[0069] Figure 3 A structural block diagram of a computer device for a large model-based action evaluation method provided by an exemplary embodiment of the present invention. DETAILED DESCRIPTION

[0070] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0071] like Figure 1 As shown, a method for action evaluation based on a large model is provided. This embodiment uses the method applied to a terminal as an example for illustration. It is understandable that the method can also be applied to a server, or to a system including a terminal and a server, and implemented through interaction between the terminal and the server. In this embodiment, the method includes the following S101 to S104, wherein:

[0072] S101, obtaining motion image data and multi-source sensor data of a motion evaluation object; the multi-source sensor data is collected from the location of the motion evaluation object.

[0073] Specifically, the terminal obtains motion image data of the motion assessment object and multi-source sensor data collected from the location of the motion assessment object. Illustratively, the motion image data can be continuous RGB image data captured by a camera, and the resolution can be set based on the user's accuracy in assessing the motion and the device being used. Optionally, the multi-source sensor data can be data from multiple sensors, including but not limited to a speedometer, gyroscope, magnetometer, and barometer.

[0074] S102 : Obtaining environmental impact parameters corresponding to the multi-source sensor data based on a preset environmental meta-knowledge base and the multi-source sensor data.

[0075] Specifically, based on multi-source sensor data, the primary features of the multi-source sensor data that have a major impact are extracted. Based on the primary features and their proportion in the multi-source sensor data, the multi-source sensor data is weighted and fused to obtain fused sensor data. This fused sensor data is then mapped to a pre-defined environmental meta-knowledge base to obtain environmental impact parameters. Illustratively, principal component analysis is used to extract the primary features from the multi-source sensor data. Optionally, the environmental meta-knowledge base can be configured based on the physical classification of environmental parameters that influence the action assessment site.

[0076] S103 , based on a generative adversarial network, correcting the action image data using the environmental impact parameter to obtain corrected action image data.

[0077] Specifically, the environmental impact parameters are converted into conditional vectors. Using a generative adversarial network, the conditional vectors are added to the action image to denoise and enhance it, resulting in corrected action image data with reduced environmental impact. Schematically, the generative adversarial network consists of two neural networks: a generator and a discriminator. The generator can generate training data containing the input parameters and transmit it to the discriminator. The discriminator receives the real action image and the training data generated by the generator as input. Adversarial training is used to optimize the generator and discriminator to improve the authenticity and environmental consistency of the corrected image. The original image is corrected according to the environmental conditions, resulting in corrected image data that meets the environmental conditions.

[0078] S104: Using a support vector machine classifier based on the corrected action image data, obtain an action evaluation result.

[0079] Specifically, a support vector machine classifier is used to classify the corrected action image data to obtain an action evaluation result. Schematically, the support vector machine classifier can be configured based on the user's requirements for the type of action evaluation result. It can be a multi-classifier to determine the posture type of the action, or a binary classifier to determine the correctness of the action.

[0080] This embodiment provides a large-model-based action evaluation method, which obtains action image data of an action evaluation object and multi-source sensor data of its location, maps the multi-source sensor data to a preset environmental meta-knowledge base to obtain environmental impact parameters, adds the environmental impact parameters to the action image data based on a generative adversarial network, and obtains corrected image data with reduced environmental impact. A support vector machine classifier is used to perform action evaluation on the corrected image data to obtain an evaluation result, thereby combining changes in environmental parameters of the environment in which the action evaluation object is located with the action image data to obtain a corrected image, and performing action evaluation based on the corrected image that is more consistent with the actual action, thereby improving the success rate of action evaluation.

[0081] In one embodiment, obtaining environmental impact parameters corresponding to the multi-source sensor data based on a preset environmental meta-knowledge base and the multi-source sensor data includes:

[0082] S201 , using principal component analysis to extract dominant features from multi-source sensor data, and obtaining dominant feature data and sensor feature vectors.

[0083] Specifically, the multi-source sensor data is standardized to obtain standardized multi-source sensor data. Principal component analysis is then used to perform eigenvalue decomposition on the standardized multi-source sensor data. Eigenvectors corresponding to eigenvalues ​​that meet preset requirements are used as principal components to form dominant feature data. These eigenvalues ​​that meet the preset requirements are then combined into sensor feature vectors. For example, the preset requirement may be the contribution ratio of the feature to the multi-source sensor data. This contribution ratio can be set based on the user's selection of environmental impact parameters.

[0084] S202 , based on the dominant feature data, the multi-source sensor data and the sensor feature vector, the multi-source sensor data is heterogeneously integrated to obtain preliminary fused feature data.

[0085] Specifically, according to the sensing feature vector, the dominant feature data and multi-source sensing data are weightedly fused and normalized to obtain preliminary fused feature data.

[0086] S203 , based on a preset environmental meta-knowledge base, physically constrain and quantify the preliminary fused feature data to obtain environmental impact parameters.

[0087] Specifically, based on the features contained in the preliminary fused feature data, the environmental meta-knowledge base is mapped to obtain the physical laws corresponding to the features. The preliminary fused feature data is further constrained and quantified based on the physical laws to obtain environmental impact parameters that are more in line with the actual environmental impact.

[0088] This embodiment extracts the dominant features of multi-source sensor data through principal component analysis to obtain dominant feature data, thereby achieving data dimensionality reduction and key information retention; integrates the dominant feature data and fully utilizes the information of different sensors through weighted fusion and normalization processing; physically constrains and quantifies the preliminary fused features based on a preset environmental meta-knowledge base, generates environmental impact parameters that conform to actual environmental impacts, realizes the quantification of environmental changes, obtains environmental impact parameters, and improves the accuracy and robustness of environmental perception.

[0089] In one embodiment, principal component analysis is used to extract dominant features from multi-source sensor data to obtain dominant feature data and sensor feature vectors, including:

[0090] S301 , performing standardized calculation on multi-source sensor data to obtain standardized sensor data.

[0091] Specifically, the multi-source sensor data is normalized to eliminate the dimensional differences between different sensors and obtain the normalized sensor data. Schematically, the multi-source sensor data can be normalized using a Z-score normalization algorithm.

[0092] S302 , calculating the covariance of the standardized sensor data to obtain a sensor covariance matrix.

[0093] Specifically, the covariance matrix of the standardized sensor data is calculated to capture the correlation between different sensor data and obtain the sensor covariance matrix.

[0094] S303 , calculating the eigenvector of the sensor covariance matrix according to the sensor covariance matrix to obtain the sensor eigenvector.

[0095] Specifically, the sensor covariance matrix is ​​subjected to eigenvalue decomposition to obtain a sensor eigenvector, which schematically represents the main pattern or trend in the sensor data.

[0096] S304: Using the following formula, select the standardized sensor data corresponding to the sensor feature vector that meets the preset cumulative contribution rate to form the dominant feature data:

[0097]

[0098] Among them, K is the rank of the data corresponding to the preset cumulative contribution rate in the standardized sensor data, min is the function of selecting the minimum value, t is the rank of the data in the standardized sensor data, λ i is the i-th sensor feature vector, d is the dimension of multi-source sensor data, and N is the cumulative contribution rate.

[0099] Specifically, according to the preset cumulative contribution rate threshold, eigenvectors that can explain the main data variation are selected, and these eigenvectors are used to reduce the dimension of the standardized sensor data to obtain the dominant characteristic data. Schematically, the rank K of the data corresponding to the preset cumulative contribution rate in the standardized sensor data refers to the data in the first K positions of the standardized sensor data, and the contribution rate to the change of the standardized sensor data reaches the preset cumulative contribution rate, and the data in the first K positions constitute the dominant characteristic data. Optionally, the minimum function min is selected to select the minimum rank K that meets the requirements. Schematically, the rank t of the data in the standardized sensor data is the rank of the data corresponding to the preset cumulative contribution rate in the standardized sensor data, which is used to find the rank of the data corresponding to the preset cumulative contribution rate. Optionally, the i-th sensor eigenvector λ i is the feature vector of the i-th data in the standardized sensor data. Schematically, the multi-source sensor data dimension d is the dimension of the standardized sensor data, which is determined by the type of input sensor data. The cumulative contribution rate N is the contribution rate of the standardized sensor data to the variance of the standardized sensor data. This cumulative contribution rate can be set based on the user's preference for the type of environmental impact.

[0100] This embodiment eliminates the dimensional differences of multi-source sensor data and captures the correlation between data through standardization processing and covariance matrix calculation. The eigenvectors obtained by eigenvalue decomposition represent the main patterns of the data, and the eigenvectors selected according to the preset cumulative contribution rate are further extracted to obtain the dominant feature data, thereby achieving data dimensionality reduction and key feature extraction, improving the efficiency and effectiveness of data processing, and retaining the main feature data.

[0101] In one embodiment, based on the dominant feature data, the multi-source sensor data, and the sensor feature vector, the multi-source sensor data is heterogeneously integrated to obtain preliminary fused feature data, including:

[0102] S401, using the following formula, calculate the contribution weight of the dominant feature data based on the sensing feature vector:

[0103]

[0104] Among them, w n is the contribution weight of the nth dominant feature data, λ n is the nth sensor feature vector, K is the rank of the data corresponding to the preset cumulative contribution rate in the standardized sensor data, and λj is the jth sensing feature vector.

[0105] Specifically, the contribution weight of each dominant feature data is calculated based on the sensing feature vector. Schematically, the contribution weight reflects its contribution to the data change. Optionally, the contribution weight w of the nth dominant feature data is n is the contribution weight of the nth data in the dominant feature data. Schematically, the nth sensing feature vector λ n is the sensor feature vector calculated in S303. Optionally, the rank K of the data corresponding to the preset cumulative contribution rate in the standardized sensor data is the rank calculated in S304, which is also the dimension of the dominant feature data. j Used to calculate the sum of sensor feature vectors.

[0106] S402, using the following formula, calculate preliminary fusion feature data based on contribution weights, dominant feature data, and multi-source sensor data:

[0107]

[0108] Among them, f i is the preliminary fusion feature data, K is the rank of the data corresponding to the preset cumulative contribution rate in the standardized sensor data, and w j is the contribution weight of the jth dominant feature data, f j is the jth dominant feature data, α is the retention coefficient, x j is the jth multi-source sensor data.

[0109] Specifically, according to the contribution weight, the dominant feature data and the original multi-source sensor data, the importance of different features and the information of the original data are comprehensively considered to calculate the preliminary fusion feature data. i It is a combination of the importance of different features obtained by principal component analysis and the information of the original multi-source sensor data. Optionally, the rank K of the data corresponding to the preset cumulative contribution rate in the standardized sensor data is the rank calculated in S304 and is also the dimension of the dominant feature data. Schematically, the contribution weight w of the jth dominant feature data is j is calculated in S402. Optionally, the jth dominant feature data f j is the jth data in the dominant feature data obtained in S304, and is used to fuse each data in the dominant feature data according to the weighted calculation formula to obtain preliminary fused feature data. Schematically, the retention coefficient α is used to balance the contribution of the dominant feature data and the original data, and can be set according to the user's sensor data processing requirements. Optionally, the jth multi-source sensor data x j It is the multi-source sensor data acquired by S101, which is the data originally obtained from the sensor.

[0110] This embodiment calculates the contribution weight of the dominant feature data and combines it with the original multi-source sensor data to achieve preliminary fusion of multi-source data, obtaining preliminary fused feature data, taking into account the contribution degree of different features to data changes, and retaining the information of the original data.

[0111] In one embodiment, the preliminary fused feature data is physically constrained and quantified based on a preset environmental meta-knowledge base to obtain environmental impact parameters, including:

[0112] S501, according to the preliminary fusion feature data, match the environment meta-knowledge base to obtain the corresponding physical constraint set.

[0113] Specifically, the preliminary fused feature data is matched against a pre-set environmental meta-knowledge base to obtain the physical laws and constraints associated with the feature data. Illustratively, this environmental meta-knowledge base includes, but is not limited to, classical mechanics, electromagnetism, thermal physics, and fluid mechanics. Optionally, a data item in this environmental meta-knowledge base can include a physics classification label and a corresponding physics constraint formula.

[0114] S502: Combine the preliminary fusion feature data and the corresponding physical constraint set to form a feature selection matrix.

[0115] Specifically, the preliminary fused feature data is integrated with the matched physical constraint set to construct a feature selection matrix. Illustratively, the physical constraint set consists of physical constraint formulas corresponding to the physical classification labels in the environmental meta-knowledge base that match the preliminary fused feature data. Alternatively, the feature selection matrix can be {preliminary fused feature data, physical constraint set}.

[0116] S503: Using the following formula, based on the feature selection matrix and the multilayer perceptron, the feature influence probability is obtained:

[0117] p i =σ(W2·ReLU(W1f i +b1)+b2)

[0118] Among them, p i is the feature influence probability of the i-th feature, σ is the activation function, W2 is the output layer matrix, W1 is the hidden layer matrix, f i is the i-th feature subset in the feature selection matrix, b1 is the hidden layer bias term, and b2 is the output layer bias term.

[0119] Specifically, the feature selection matrix is ​​processed using a multi-layer perceptron (MLP) to calculate the probability of each feature affecting the environment. Schematically, the feature influence probability p of the i-th feature is iis the influence probability of the i-th feature in the preliminary fusion feature data, which can be the probability of causing noise to the acquired action image data. Optionally, the activation function σ can be a Sigmoid function. Schematically, the output layer matrix W2 is the weight matrix of the output layer of the multilayer perceptron, which can be obtained through training. Optionally, the hidden layer matrix W1 is the weight matrix of the hidden layer of the multilayer perceptron, which can be obtained through training. Schematically, the i-th feature subset f in the feature selection matrix i The feature data of the feature and the corresponding physical constraint set are included. Optionally, the hidden layer bias term b1 is the bias term of the hidden layer of the multilayer perceptron, which can be obtained through training. Schematically, the output layer bias term b2 is the bias term of the output layer of the multilayer perceptron, which can be obtained through training.

[0120] In step S504, the following formula is used to combine the characteristic impact probabilities to obtain the environmental impact parameter:

[0121]

[0122] Among them, p f is the environmental impact parameter, m is the number of feature subsets contained in the feature selection matrix, w j is the weight of the meta-knowledge base of the jth feature, p j is the feature influence probability of the j-th feature.

[0123] Specifically, based on the feature influence probability and the weight of the physics label in the meta-knowledge base, the environmental influence parameter is calculated to describe the degree of influence of the environment on the action image. f is the influence parameter of all features in the initial fusion feature data. Optionally, the number of feature subsets m included in the feature selection matrix is ​​the number of features included in the initial fusion feature data, and is the dimension of the initial fusion feature data. Schematically, the weight w of the meta-knowledge base of the jth feature is j It is preset by the meta-knowledge base and is the weight of the physics classification label corresponding to the jth feature in the preliminary fusion feature data. It can be set according to the user's needs for environment fitting. Optionally, the feature influence probability p of the jth feature j It is calculated in S503.

[0124] This example extracts relevant physical constraints and constructs a feature selection matrix by matching the environmental meta-knowledge base and preliminarily fusing feature data. A multi-layer perceptron (MLP) is then used to calculate the probability of each feature's impact on the environment. By weighted summing the feature impact probabilities, an environmental impact parameter is derived that integrates the influence of different features. This approach combines the correlation between features and physical laws, capturing the complex relationships between features through a neural network model, thereby providing a more accurate and comprehensive quantitative description of environmental impact.

[0125] In one embodiment, based on a generative adversarial network, the action image data is corrected using environmental impact parameters to obtain the corrected action image data, including:

[0126] S601: Normalize the environmental impact parameters to obtain normalized environmental impact parameters.

[0127] Specifically, the environmental impact parameter is normalized to standardize its value range to obtain a normalized environmental impact parameter. Schematically, a maximum-minimum normalization method can be used to scale the environmental impact parameter to a range of [0, 1] to obtain the normalized environmental impact parameter.

[0128] S602: Use the following formula to encode the normalized environmental impact parameter to obtain an environmental impact vector.

[0129] h P =σ(W p P n +b p )

[0130] Among them, h P is the environmental influence vector, σ is the activation function, W p is the influence weight, P n is the normalized environmental impact parameter, b p is the influencing bias term.

[0131] Specifically, the normalized environmental impact parameters are encoded into a vector form to obtain an environmental impact vector. Schematically, the environmental impact vector h P The normalized environmental impact parameter is encoded by the formula, and the environmental impact vector is used for subsequent action feature fusion and extraction. Optionally, the activation function σ is used to map the result of the linear transformation to a specific range, which can be a Sigmoid function. Schematically, the impact weight W p It is used to perform linear transformation on the normalized environmental impact parameter, which can be obtained through training or set according to the user's coding requirements for the environmental parameter. n is the normalized environmental impact parameter obtained in S601. Schematically, the impact bias term b p The result used to adjust the linear transformation can be obtained through training or set according to the user's encoding requirements for the environment parameters.

[0132] S603: extract multi-scale features of the action image data to obtain a multi-layer action feature map.

[0133] Specifically, the action image data is layered and multi-scale features are extracted from it to capture action information at different scales, resulting in a multi-layer action feature map. Schematically, a convolutional neural network can be used to perform multi-layer convolution operations, gradually reducing the resolution of the action image data and increasing the number of channels in the feature map to obtain a multi-layer action feature map.

[0134] S604: Expand the normalized environmental impact parameters according to the dimensions of the multi-layer motion feature graph to obtain environmental impact parameters of the same dimension.

[0135] Specifically, the normalized environmental impact parameter is expanded based on the dimension of the multi-layer action feature map to have the same dimension as the multi-layer action feature map, thereby obtaining the same-dimensional environmental impact parameter. In principle, the dimension of the multi-layer action feature map can be set according to the user's requirements for action target recognition.

[0136] S605: Use the following formula to connect the same-dimensional environmental impact parameters with the motion feature map to obtain a layered motion feature map;

[0137] f l =Concat(W l +h s )

[0138] Among them, f l is the l-th layered action feature map, Concat is the connection function, W l is the l-th layer action feature map, h s It is the environmental impact parameter of the same dimension.

[0139] Specifically, the expanded same-dimensional environmental impact parameters are connected to the feature maps of each layer in the multi-layer action feature map to obtain a hierarchical action feature map. l The motion feature graph of the first layer in the multi-layer motion feature graph obtained in S603 is connected with the same-dimensional environmental impact parameter obtained in S604 to obtain the motion feature graph. Optionally, the connection function Concat is a connection operation for splicing the same-dimensional environmental impact parameter with the motion feature graph. Schematically, the motion feature graph of the first layer W l is the motion feature map of the first layer in the multi-layer motion feature map obtained in S603. Optionally, the same-dimensional environmental impact parameter h s is the same-dimensional environmental impact parameter obtained in S604.

[0140] S606: Perform a deconvolution operation on the layered action feature map to obtain corrected action image data.

[0141] Specifically, deconvolution is performed on the layered action feature maps, gradually restoring the image's resolution and number of channels to produce corrected action image data. Schematically, a deconvolution neural network can be used to perform multi-layer deconvolution operations, gradually increasing the image's resolution and reducing the number of channels in the feature maps to produce corrected action image data. Optionally, this corrected action image data can be used for adversarial training of the discriminator, calculating the adversarial reconstruction loss to ensure geometric consistency between the corrected image and the true action.

[0142] This embodiment adopts a generative adversarial network. The generator that has undergone adversarial training obtains normalized parameter encoding by normalizing environmental impact parameters, converts the normalized parameter encoding into an environmental impact vector, and uses multi-layer convolution to extract multi-scale features of the action image, captures details at different spatial resolutions, and obtains a multi-layer action feature map. The environmental impact parameters are matched with the feature map dimensions through expansion operations to obtain environmental impact parameters of the same dimension. The environmental impact parameters of the same dimension are layered connected with the feature map to obtain a layered action feature map; the image resolution is gradually restored through deconvolution operations to obtain a corrected action image; the discriminator that has undergone adversarial training determines that the corrected action image meets geometric consistency, and outputs corrected action image data, thereby realizing dynamic association between environmental parameters and image features and improving the robustness and accuracy of the action image.

[0143] In one embodiment, based on the corrected motion image data, a support vector machine classifier is used to obtain a motion assessment result, including:

[0144] S701, using the Z-score algorithm to perform standardization processing on the corrected action image data to obtain standard action feature data.

[0145] Specifically, the Z-score algorithm is used to standardize the corrected action image data, eliminate the dimensional differences of the data, and obtain standard action feature data that conforms to the standard normal distribution.

[0146] S702 , using a radial basis kernel function to map the standard motion feature data to a high-dimensional space to obtain high-dimensional motion feature data.

[0147] Specifically, the radial basis kernel function is used to map the normalized image data into a high-dimensional space, enhancing the data's expressiveness and discriminability, and generating high-dimensional motion feature data. In principle, the radial basis kernel function can effectively capture the local structural information of the data, improving the generalization ability of the classifier.

[0148] S703: Use the following formula to classify the high-dimensional action feature data and obtain the action probability output:

[0149]

[0150] Among them, P(y=1|f) is the action probability output of the action category, A is the fitting parameter, N is the dimension of the high-dimensional action feature data, α i is the i-th Lagrange multiplier, y i is the action probability output of the i-th high-dimensional action feature data, K(X i ,X) is the i-th high-dimensional action feature data, and B is the bias term.

[0151] Specifically, a support vector machine is used to classify high-dimensional action feature data, and the probability output of each action category is calculated to obtain the action probability output. Schematically, the action probability output P(y=1|f) of the action category is the probability output of any action category, and the number of action categories can be set according to the user's action evaluation requirements. Optionally, the fitting parameter A is used to adjust the sensitivity of the probability output, which can be set according to the user's requirements for action category identification. Schematically, the dimension N of the high-dimensional action feature data is the dimension of the high-dimensional action feature data obtained in S702. Optionally, the i-th Lagrange multiplier α i It is used to indicate the importance of the i-th support vector in constructing the classification hyperplane. Schematically, the action probability output y of the i-th high-dimensional action feature data is i is the classification label of the i-th high-dimensional action feature data, which is used to indicate the category tendency to which the feature data belongs, and can be obtained by classification labeling by a classifier. Optionally, the i-th high-dimensional action feature data K(X i , X) is the high-dimensional action feature data obtained in S702. Schematically, the bias term B is used to adjust the position of the decision boundary, which can be obtained through training or set according to the user's requirements for identifying the action type.

[0152] S704: Use the following formula to obtain the action evaluation result based on the action probability output:

[0153]

[0154] in, is the action evaluation result, Select the maximum value function of K, P k (y=1|f) is the k-th action probability output.

[0155] Specifically, based on the action probability output, the maximization strategy is used to select the most likely action category as the evaluation result to obtain the action evaluation result. Represents the final action evaluation result, that is, the action category judged by the classifier. Optionally, select the maximum value function of K It is used to select the category K with the largest corresponding probability among all possible action categories. Schematically, the k-th action probability output Pk (y=1|f) is the action probability output obtained in S703.

[0156] This embodiment standardizes the corrected action image data to obtain standard action feature data, uses a support vector machine to perform multi-classification on the standard action feature data, calculates the probability output of each action category, and adopts a maximization strategy to select the most likely action category to obtain an action evaluation result, thereby improving the accuracy and robustness of the classification and achieving efficient and accurate evaluation of the action image data.

[0157] In the above-mentioned large-model-based action evaluation method, the action image data of the action evaluation object and the multi-source sensor data of the location are obtained, the multi-source sensor data are mapped to a preset environmental meta-knowledge base to obtain environmental impact parameters, and based on a generative adversarial network, the environmental impact parameters are added to the action image data to obtain corrected image data with reduced environmental impact. A support vector machine classifier is used to perform action evaluation on the corrected image data to obtain an evaluation result, thereby combining the environmental parameter changes of the environment where the action evaluation object is located with the action image data to obtain a corrected image, and performing action evaluation based on the corrected image that is more in line with the actual action, thereby improving the success rate of action evaluation.

[0158] It should be understood that, although the various steps in the flowcharts involved in the various embodiments described above are displayed in sequence according to the instructions of the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and these steps can be executed in other orders. Moreover, at least a portion of the steps in the flowcharts involved in the various embodiments described above can include multiple steps or multiple stages, and these steps or stages are not necessarily executed and completed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a portion of steps or stages in other steps.

[0159] Based on the same inventive concept, the present application also provides a large-model-based action assessment device for implementing the large-model-based action assessment method mentioned above. The implementation solution provided by this device is similar to the implementation solution described in the above method. Therefore, the specific limitations of one or more large-model-based action assessment device embodiments provided below can be found in the above-mentioned limitations of the large-model-based action assessment method, and will not be repeated here.

[0160] In an exemplary embodiment, Figure 2 As shown, a large model-based action evaluation device 200 is provided, comprising:

[0161] Data acquisition module 201, used to acquire motion image data and multi-source sensor data;

[0162] The parameter determination module 202 is used to obtain the environmental impact parameters corresponding to the multi-source sensor data based on the preset environmental meta-knowledge base and the multi-source sensor data;

[0163] An image correction module 203 is configured to correct the action image data using environmental impact parameters based on a generative adversarial network to obtain corrected action image data;

[0164] The result evaluation module 204 is used to obtain the action evaluation result by using a support vector machine classifier based on the corrected action image data.

[0165] Furthermore, the parameter determination module also includes:

[0166] A principal component analysis unit is used to extract dominant features from multi-source sensor data using a principal component analysis method to obtain dominant feature data and sensor feature vectors;

[0167] A preliminary fusion unit is used to integrate the heterogeneous multi-source sensor data according to the dominant feature data, the multi-source sensor data and the sensor feature vector to obtain preliminary fusion feature data;

[0168] The environmental parameter determination unit is used to physically constrain and quantify the preliminary fused feature data based on a preset environmental meta-knowledge base to obtain environmental impact parameters.

[0169] Furthermore, the principal component analysis unit is also used to:

[0170] Performing standardized calculation on multi-source sensor data to obtain standardized sensor data;

[0171] Calculate the covariance of the standardized sensor data to obtain the sensor covariance matrix;

[0172] According to the sensor covariance matrix, the eigenvector of the sensor covariance matrix is ​​calculated to obtain the sensor eigenvector;

[0173] Use the following formula to select the standardized sensor data corresponding to the sensor feature vector that meets the preset cumulative contribution rate to form the dominant feature data:

[0174]

[0175] Among them, K is the rank of the data corresponding to the preset cumulative contribution rate in the standardized sensor data, min is the function of selecting the minimum value, t is the rank of the data in the standardized sensor data, λ i is the i-th sensor feature vector, d is the dimension of multi-source sensor data, and n is the cumulative contribution rate.

[0176] Furthermore, the preliminary fusion unit is also used to:

[0177] The contribution weight of the dominant feature data is calculated based on the sensing feature vector using the following formula:

[0178]

[0179] Among them, w i is the contribution weight of the nth dominant feature data, λ n is the nth sensor feature vector, K is the rank of the data corresponding to the preset cumulative contribution rate in the standardized sensor data, and λ j is the jth sensing feature vector;

[0180] The following formula is used to calculate the preliminary fusion feature data based on the contribution weight, dominant feature data, and multi-source sensor data:

[0181]

[0182] Among them, f i is the preliminary fusion feature data, K is the rank of the data corresponding to the preset cumulative contribution rate in the standardized sensor data, and w j is the contribution weight of the jth dominant feature data, f j is the jth dominant feature data, α is the retention coefficient, x j is the jth multi-source sensor data.

[0183] Furthermore, the environmental parameter determination unit is further configured to:

[0184] According to the preliminary fusion feature data, match the environment meta-knowledge base to obtain the corresponding physical constraint set;

[0185] Combine the preliminary fused feature data with the corresponding physical constraint set to form a feature selection matrix;

[0186] Use the following formula to get the feature influence probability based on the feature selection matrix and multilayer perceptron:

[0187] p i =σ(W2·ReLU(W1f i +b1)+b2)

[0188] Among them, p i is the feature influence probability of the i-th feature, σ is the activation function, W2 is the output layer matrix, W1 is the hidden layer matrix, f i is the i-th feature subset in the feature selection matrix, b1 is the hidden layer bias term, and b2 is the output layer bias term;

[0189] The environmental impact parameter is obtained by combining the characteristic impact probabilities using the following formula:

[0190]

[0191] Among them, p f is the environmental impact parameter, m is the number of feature subsets contained in the feature selection matrix, w j is the weight of the meta-knowledge base of the jth feature, p j is the feature influence probability of the j-th feature.

[0192] Furthermore, the image correction module is also used to:

[0193] Normalizing the environmental impact parameters to obtain normalized environmental impact parameters;

[0194] Use the following formula to encode the normalized environmental impact parameter to obtain the environmental impact vector:

[0195] h P =σ(W p P n +b p )

[0196] Among them, h P is the environmental influence vector, σ is the activation function, W p is the influence weight, P n is the normalized environmental impact parameter, b p is the influencing bias term;

[0197] Extract multi-scale features of action image data to obtain multi-layer action feature maps;

[0198] According to the dimension of the multi-layer motion feature map, the normalized environmental impact parameter is expanded to obtain the environmental impact parameter of the same dimension;

[0199] Use the following formula to connect the same-dimensional environmental impact parameters with the action feature map to obtain the hierarchical action feature map;

[0200] f l =Concat(W l +h s )

[0201] Among them, f l is the l-th layered action feature map, Concat is the connection function, W l is the l-th layer action feature map, h s is the environmental impact parameter of the same dimension;

[0202] The layered action feature map is deconvolved to obtain the corrected action image data.

[0203] Furthermore, the result evaluation module is also used to:

[0204] The Z-score algorithm is used to standardize the corrected action image data to obtain standard action feature data;

[0205] Use the radial basis kernel function to map the standard motion feature data to a high-dimensional space to obtain high-dimensional motion feature data:

[0206] Use the following formula to classify high-dimensional action feature data and obtain the action probability output:

[0207]

[0208] Among them, P(y=1|f) is the action probability output of the action category, A is the fitting parameter, N is the dimension of the high-dimensional action feature data, α i is the i-th Lagrange multiplier, y i is the action probability output of the i-th high-dimensional action feature data, K(X i ,X) is the i-th high-dimensional action feature data, B is the bias term;

[0209] Use the following formula to get the action evaluation result based on the action probability output:

[0210]

[0211] in, is the action evaluation result, Select the maximum value function of K, P k (y=1|f) is the k-th action probability output.

[0212] In one embodiment, Figure 3 A computer device is provided, comprising:

[0213] At least one processor 301, and a memory 302 communicatively connected to the at least one processor 301: the memory stores application code that can be executed by the at least one processor, and the application code is executed by the at least one processor to enable the at least one processor to perform the large model-based action evaluation method as described above.

[0214] The computer device may further include: sensor 303.

[0215] The processor 301 , the memory 302 and the sensor 303 may be connected via a bus or other means, with the bus being used as an example in the figure.

[0216] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0217] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to the partial description of the method embodiments. The device embodiments described above are merely illustrative, wherein the components described as separate parts may or may not be physically separated, and the parts displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the disclosed solution. A person of ordinary skill in the art can understand and implement it without expending creative work.

[0218] The above-described embodiments merely represent several implementation methods of the embodiments of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the patent application. It should be noted that a person skilled in the art may make various modifications and improvements without departing from the concept of the embodiments of the present application, and these modifications and improvements fall within the scope of protection of the embodiments of the present application.

Claims

1. A large model-based action evaluation method, characterized in that: The method comprises: Acquiring motion image data and multi-source sensor data of a motion evaluation object; the multi-source sensor data is collected by multiple sensors at the location of the motion evaluation object; Obtaining environmental impact parameters corresponding to the multi-source sensor data based on a preset environmental meta-knowledge base and the multi-source sensor data; Based on a generative adversarial network, the action image data is corrected by the environmental impact parameter to obtain corrected action image data; A support vector machine classifier is used according to the corrected action image data to obtain an action evaluation result.

2. The method according to claim 1, characterized in that The obtaining, based on a preset environmental meta-knowledge base and the multi-source sensor data, environmental impact parameters corresponding to the multi-source sensor data includes: Extracting dominant features from the multi-source sensor data using principal component analysis to obtain dominant feature data and sensor feature vectors; According to the dominant feature data, the multi-source sensor data and the sensor feature vector, the multi-source sensor data is heterogeneously integrated to obtain preliminary fused feature data; According to the preset environmental meta-knowledge base, the preliminary fused feature data is physically constrained and quantified to obtain the environmental impact parameters.

3. The method according to claim 2, characterized in that The method of extracting the dominant features from the multi-source sensor data by using the principal component analysis method to obtain the dominant feature data and the sensor feature vector includes: Performing standardized calculation on the multi-source sensor data to obtain standardized sensor data; Calculating the covariance of the standardized sensor data to obtain a sensor covariance matrix; Calculating the eigenvector of the sensor covariance matrix according to the sensor covariance matrix to obtain the sensor eigenvector; The following formula is used to select the standardized sensor data corresponding to the sensor feature vector that meets the preset cumulative contribution rate to form the dominant feature data: Among them, K is the rank of the data corresponding to the preset cumulative contribution rate in the standardized sensor data, min is the function of selecting the minimum value, t is the rank of the data in the standardized sensor data, λ i is the i-th sensor feature vector, d is the dimension of multi-source sensor data, and N is the cumulative contribution rate.

4. The method according to claim 2, characterized in that The step of heterogeneously integrating the multi-source sensor data according to the dominant feature data, the multi-source sensor data, and the sensor feature vector to obtain preliminary fused feature data includes: The contribution weight of the dominant feature data is calculated based on the sensing feature vector using the following formula: Among them, w n is the contribution weight of the nth dominant feature data, λ n is the nth sensor feature vector, K is the rank of the data corresponding to the preset cumulative contribution rate in the standardized sensor data, and λ j is the jth sensing feature vector; The following formula is used to calculate the preliminary fusion feature data based on the contribution weight, the dominant feature data, and the multi-source sensor data: Among them, f i is the preliminary fusion feature data, K is the rank of the data corresponding to the preset cumulative contribution rate in the standardized sensor data, and w j is the contribution weight of the jth dominant feature data, f j is the jth dominant feature data, α is the retention coefficient, x j is the jth multi-source sensor data.

5. The method according to claim 2, characterized in that The step of physically constraining and quantifying the preliminary fused feature data according to the preset environmental meta-knowledge base to obtain the environmental impact parameters includes: According to the preliminary fusion feature data, matching the environmental meta-knowledge base to obtain a corresponding physical constraint set; Combining the preliminary fused feature data with the corresponding physical constraint set to form a feature selection matrix; The following formula is used to obtain the feature influence probability based on the feature selection matrix and multilayer perceptron: p i =σ(W2·ReLU(W1f i +b1)+b2) Among them, p i is the feature influence probability of the i-th feature, σ is the activation function, W2 is the output layer matrix, W1 is the hidden layer matrix, f i is the i-th feature subset in the feature selection matrix, b1 is the hidden layer bias term, and b2 is the output layer bias term; The environmental impact parameter is obtained by combining the characteristic impact probabilities using the following formula: Among them, p f is the environmental impact parameter, m is the number of feature subsets contained in the feature selection matrix, w j is the weight of the meta-knowledge base of the jth feature, p j is the feature influence probability of the j-th feature.

6. The method according to claim 1, characterized in that The method of correcting the action image data by using the environmental impact parameter based on a generative adversarial network to obtain the corrected action image data includes: Normalizing the environmental impact parameters to obtain normalized environmental impact parameters; The normalized environmental impact parameter is encoded using the following formula to obtain an environmental impact vector: h P =σ(W p P n +b p ) Among them, h P is the environmental influence vector, σ is the activation function, W p is the influence weight, P n is the normalized environmental impact parameter, b p is the influencing bias term; Extracting multi-scale features of the action image data to obtain a multi-layer action feature map; Expanding the normalized environmental impact parameter according to the dimension of the multi-layer motion feature graph to obtain an environmental impact parameter of the same dimension; Use the following formula to connect the same-dimensional environmental impact parameter with the action feature map to obtain a layered action feature map; f l =Concat(W l +h s ) Among them, f l is the l-th layered action feature map, Concat is the connection function, W l is the l-th layer action feature map, h s is the environmental impact parameter of the same dimension; The layered action feature map is subjected to a deconvolution operation to obtain the corrected action image data.

7. The method according to claim 1, characterized in that The step of obtaining the action evaluation result by using a support vector machine classifier based on the corrected action image data includes: Using a Z-score algorithm, the corrected action image data is standardized to obtain standard action feature data; The standard motion feature data is mapped to a high-dimensional space using a radial basis kernel function to obtain high-dimensional motion feature data: Use the following formula to classify the high-dimensional action feature data and obtain the action probability output: Among them, P(y=1|f) is the action probability output of the action category, A is the fitting parameter, N is the dimension of the high-dimensional action feature data, α i is the i-th Lagrange multiplier, y i is the action probability output of the i-th high-dimensional action feature data, K(X i ,X) is the i-th high-dimensional action feature data, B is the bias term; The action evaluation result is obtained according to the action probability output using the following formula: in, is the action evaluation result, Select the maximum value function of K, P k (y=1|f) is the k-th action probability output.

8. A large-model-based motion assessment device, characterized in that: The device comprises: A data acquisition module, used to acquire motion image data and multi-source sensor data; a parameter determination module, configured to obtain environmental impact parameters corresponding to the multi-source sensor data based on a preset environmental meta-knowledge base and the multi-source sensor data; An image correction module, configured to correct the action image data using the environmental impact parameter based on a generative adversarial network to obtain corrected action image data; The result evaluation module is used to obtain the action evaluation result by using a support vector machine classifier based on the corrected action image data.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images