Data processing method and system based on virtual simulation practical training simulation system

Through virtual simulation training simulation systems and deep learning technology, the exercise ability level of the elderly is automatically evaluated, solving the safety and accuracy problems existing in traditional evaluation methods, and achieving more efficient and safe movement status monitoring.

CN118942691AInactive Publication Date: 2024-11-12BEIJING APE TECH CO LTD

Patent Information

Application Number
CN202410932709.9
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-12
Publication Date
2024-11-12
Estimated Expiration
Not applicable · inactive patent

AI Technical Summary

Technical Problem

Exercise status assessments in older adults may be at risk of injury, and traditional assessments require professional equipment and manual supervision, which are costly and limited in accuracy, especially when resources are limited.

Method used

Using a virtual simulation training simulation system, the histogram of the movement status monitoring video of the elderly is calculated by obtaining the directional optical flow histogram, and using deep learning technology to perform feature extraction and semantic analysis, and automatically evaluate the motor ability level based on multi-scale fusion features, combining with real Athletic ability level label for training.

Benefits of technology

Improve the accuracy and safety of evaluation, reduce the dependence on professional equipment and manual supervision, provide more accurate motion dynamic feature analysis, and reduce evaluation costs.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118942691A_ABST
    Figure CN118942691A_ABST
Patent Text Reader

Abstract

The invention discloses a data processing method and system based on a virtual simulation practical training simulation system, and relates to the field of data intelligent processing, and the method comprises the steps: obtaining an old people motion state monitoring video, calculating a direction optical flow histogram of the old people motion state monitoring video, and obtaining the direction optical flow histogram of the old people motion state monitoring video; and carrying out feature extraction and semantic analysis on the direction optical flow histogram by adopting an image analysis and processing technology based on deep learning, so as to automatically evaluate the athletic ability level of the old people according to the shallow features and deep semantic multi-scale fusion features of the direction optical flow histogram. Training is carried out based on the exercise ability evaluation result and the real exercise ability grade label of the elderly object. Through the mode, the data intelligent processing of the virtual simulation practical training simulation system can be realized, so that the dynamic characteristics of the movement of the old people can be more accurately captured, accurate data support is provided for evaluation, and the accuracy and safety of the evaluation are improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of intelligent data processing, and more specifically, to a data processing method and system based on a virtual simulation training system. Background Art

[0002] As the global population ages, there is a growing concern about the health of the elderly, especially their exercise and mobility. Monitoring and evaluating the exercise status and ability of the elderly is essential to maintaining and improving their quality of life. This process can detect potential health problems in a timely manner so that appropriate preventive or intervention measures can be taken to help the elderly enjoy a healthier and more active life in their later years.

[0003] However, actual exercise testing may put older adults at risk of injury, especially when assessing high-intensity or complex movements, and traditional assessments may require specialized equipment and manual supervision, which is costly in resource-limited settings and may also limit the accuracy of the assessment.

[0004] Therefore, a data processing method based on a virtual simulation training system is desired. Summary of the invention

[0005] In order to solve the above technical problems, the present application is proposed. The embodiment of the present application provides a data processing method and system based on a virtual simulation training system, which obtains a monitoring video of the movement status of the elderly, calculates the directional optical flow histogram of the monitoring video of the movement status of the elderly, and uses image analysis and processing technology based on deep learning to perform feature extraction and semantic analysis on the directional optical flow histogram, so as to automatically evaluate the elderly's athletic ability level based on the shallow features and deep semantic multi-scale fusion features of the directional optical flow histogram, and conduct training based on the athletic ability evaluation results and the actual athletic ability level label of the elderly object. In this way, intelligent data processing of the virtual simulation training system can be achieved, so as to more accurately capture the dynamic characteristics of the elderly's movements, provide accurate data support for the evaluation, and improve the accuracy and safety of the evaluation.

[0006] According to one aspect of the present application, a data processing method based on a virtual simulation training system is provided, which includes:

[0007] Obtain monitoring videos of the elderly’s exercise status;

[0008] Calculating the directional optical flow histogram of the elderly person's motion status monitoring video;

[0009] Using the motion state feature extractor based on the deep neural network to perform motion state multi-scale feature extraction on the directional optical flow histogram to obtain a motion feature shallow representation map and a motion feature semantic representation map;

[0010] Inputting the motion feature semantic representation map into a gated screening adaptive attention module to obtain a significant motion feature semantic representation map;

[0011] Inputting the shallow representation map of motion features and the semantic representation map of significant motion features into an information loss-compensation network to obtain a multi-scale fusion representation map of the motion state of the elderly as a multi-scale fusion representation feature of the motion state of the elderly;

[0012] Passing the multi-scale fusion representation features of the elderly's movement state through a movement ability evaluator to obtain a movement ability evaluation result;

[0013] The cross entropy loss function value between the motor ability assessment result and the actual motor ability level label of the elderly subject is used as the loss function value to train the motion state feature extractor based on the deep neural network, the gated screening adaptive attention module, the information loss-compensation network and the motor ability evaluator.

[0014] In the above-mentioned data processing method based on the virtual simulation training system, the motion state feature extractor based on the deep neural network is a motion state feature extractor based on the hollow pyramid network.

[0015] In the above-mentioned data processing method based on the virtual simulation training system, the motion feature semantic representation map is input into the gated screening adaptive attention module to obtain a significant motion feature semantic representation map, including: performing pooling processing on each feature matrix along the channel dimension of the motion feature semantic representation map in multiple ways to obtain a global maximum motion feature semantic pooling feature vector, a global mean motion feature semantic pooling feature vector and a global random value motion feature semantic pooling feature vector; multiplying the global maximum motion feature semantic pooling feature vector, the global mean motion feature semantic pooling feature vector and the global random value motion feature semantic pooling feature vector with the corresponding adjustment parameters by position respectively to obtain a weighted global maximum motion feature semantic pooling feature vector, a weighted global mean motion feature semantic pooling feature vector and a weighted global random value motion feature semantic pooling feature vector; multiplying the weighted global maximum motion feature semantic pooling feature vector, the global mean motion feature semantic pooling feature vector and the global random value motion feature semantic pooling feature vector by position respectively. The feature vector, the weighted global mean motion feature semantic pooling feature vector and the weighted global random value motion feature semantic pooling feature vector are added by position to obtain a global motion feature semantic representation vector; the global motion feature semantic representation vector is subjected to channel-by-channel semantic feature interaction and feature activation based on a fully connected layer to obtain a channel motion feature semantic feature vector; in response to each position feature value in the channel motion feature semantic feature vector being greater than or equal to a predetermined threshold, the position feature value takes the original value, otherwise it is zero, to obtain a gated filtered channel motion feature semantic feature vector; the gated filtered channel motion feature semantic feature vector is normalized to obtain a gated filtered channel motion feature semantic weight feature vector; and, the gated filtered channel motion feature semantic weight feature vector is used as a weight feature vector, and its channel-by-channel product with the motion feature semantic representation map is calculated to obtain the significant motion feature semantic representation map.

[0016] In the above-mentioned data processing method based on the virtual simulation training system, each feature matrix of the motion feature semantic representation graph along the channel dimension is pooled in multiple ways to obtain a global maximum motion feature semantic pooling feature vector, a global mean motion feature semantic pooling feature vector and a global random value motion feature semantic pooling feature vector, including: each feature matrix of the motion feature semantic representation graph along the channel dimension is pooled based on the maximum value, based on the average value and based on the random value to obtain the global maximum motion feature semantic pooling feature vector, the global mean motion feature semantic pooling feature vector and the global random value motion feature semantic pooling feature vector.

[0017] In the above-mentioned data processing method based on the virtual simulation training system, the global motion feature semantic representation vector is subjected to channel-by-channel semantic feature interaction and feature activation based on the fully connected layer to obtain the channel motion feature semantic feature vector, including: calculating the matrix multiplication of the global motion feature semantic representation vector and the weight matrix, and then adding the vector to the bias vector to obtain the bias-adjusted global motion feature semantic representation vector; and activating the bias-adjusted global motion feature semantic representation vector through a sigmoid activation function to obtain the channel motion feature semantic feature vector.

[0018] In the above-mentioned data processing method based on the virtual simulation training system, the shallow representation map of motion features and the semantic representation map of significant motion features are input into the information loss-compensation network to obtain a multi-scale fusion representation map of the motion state of the elderly as a multi-scale fusion representation feature of the motion state of the elderly, including: calculating the compensation loss information between the shallow representation map of motion features and the semantic representation map of significant motion features to obtain a motion state information loss compensation feature map; and compensating the shallow representation map of motion features based on the motion state information loss compensation feature map to obtain the multi-scale fusion representation map of the motion state of the elderly.

[0019] In the above-mentioned data processing method based on the virtual simulation training system, the compensation loss information between the shallow representation map of the motion feature and the semantic representation map of the significant motion feature is calculated to obtain the motion state information loss compensation feature map, including: downsampling the shallow representation map of the motion feature to obtain the downsampled shallow representation map of the motion feature; subtracting the feature map obtained by point convolution of the downsampled shallow representation map of the motion feature from the semantic representation map of the significant motion feature by position to obtain the motion state difference feature map; inputting the motion state difference feature map into the tanh function to obtain the activated motion state difference feature map; and upsampling the activated motion state difference feature map to obtain the motion state information loss compensation feature map.

[0020] In the above-mentioned data processing method based on the virtual simulation training system, the shallow representation map of motion features is compensated based on the motion state information loss compensation feature map to obtain the multi-scale fusion representation map of the motion state of the elderly, including: after calculating the positional product of the shallow representation map of motion features and the first trainable hyperparameter, the obtained weighted motion feature shallow representation map and the motion state information loss compensation feature map are multiplied by position to obtain a first compensated motion state feature map; the positional product of the significant motion feature semantic representation map and the second trainable hyperparameter is calculated to obtain a second compensated motion state feature map; and the first compensated motion state feature map and the second compensated motion state feature map are added by position to obtain the multi-scale fusion representation map of the motion state of the elderly.

[0021] In the above-mentioned data processing method based on the virtual simulation training system, the multi-scale fusion representation features of the elderly's movement state are passed through a movement ability evaluator to obtain a movement ability evaluation result, including: passing the multi-scale fusion representation graph of the elderly's movement state through a classifier-based movement ability evaluator to obtain a movement ability evaluation result, and the movement ability evaluation result is used to represent the movement ability level label.

[0022] According to another aspect of the present application, a data processing system based on a virtual simulation training system is provided, comprising:

[0023] The motion status monitoring video acquisition module is used to obtain the motion status monitoring video of the elderly;

[0024] A directional optical flow histogram calculation module, used to calculate the directional optical flow histogram of the elderly motion status monitoring video;

[0025] A motion state multi-scale feature extraction module, used to use the motion state feature extractor based on the deep neural network to perform motion state multi-scale feature extraction on the directional optical flow histogram to obtain a motion feature shallow representation map and a motion feature semantic representation map;

[0026] A significant motion feature extraction module, used for inputting the motion feature semantic representation map into a gated screening adaptive attention module to obtain a significant motion feature semantic representation map;

[0027] A multi-scale feature fusion module, used for inputting the shallow representation map of motion features and the semantic representation map of significant motion features into an information loss-compensation network to obtain a multi-scale fusion representation map of the motion state of the elderly as a multi-scale fusion representation feature of the motion state of the elderly;

[0028] A motor ability assessment result generation module, used for passing the multi-scale fusion representation features of the elderly's motor state through a motor ability evaluator to obtain a motor ability assessment result;

[0029] A training module is used to train the motion state feature extractor based on the deep neural network, the gated screening adaptive attention module, the information loss-compensation network and the motion ability evaluator by using the cross entropy loss function value between the motion ability assessment result and the actual motion ability level label of the elderly subject as the loss function value.

[0030] The present application has at least the following technical effects: Compared with the prior art, the data processing method and system based on the virtual simulation training system provided by the present application obtains the elderly's motion status monitoring video, calculates the directional optical flow histogram of the elderly's motion status monitoring video, and uses the image analysis and processing technology based on deep learning to perform feature extraction and semantic analysis on the directional optical flow histogram, so as to automatically evaluate the elderly's sports ability level based on the shallow features and deep semantic multi-scale fusion features of the directional optical flow histogram, and conduct training based on the sports ability evaluation results and the actual sports ability level label of the elderly object. In this way, the intelligent processing of the data of the virtual simulation training system can be realized, so as to more accurately capture the dynamic characteristics of the elderly's movement, provide accurate data support for the evaluation, and improve the accuracy and safety of the evaluation. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] By describing the embodiments of the present application in more detail in conjunction with the accompanying drawings, the above and other purposes, features and advantages of the present application will become more apparent. The accompanying drawings are used to provide a further understanding of the embodiments of the present application and constitute a part of the specification. Together with the embodiments of the present application, they are used to explain the present application and do not constitute a limitation of the present application. In the accompanying drawings, the same reference numerals generally represent the same components or steps.

[0032] Figure 1 It is a flowchart of a data processing method based on a virtual simulation training system according to an embodiment of the present application.

[0033] Figure 2 It is a schematic diagram of the architecture of a data processing method based on a virtual simulation training system according to an embodiment of the present application.

[0034] Figure 3 The present invention provides a flowchart of inputting the motion feature semantic representation graph into a gated screening adaptive attention module to obtain a significant motion feature semantic representation graph in a data processing method based on a virtual simulation training system according to an embodiment of the present application.

[0035] Figure 4The present invention is a flow chart of inputting the shallow representation map of motion features and the semantic representation map of significant motion features into an information loss-compensation network in a data processing method based on a virtual simulation training system according to an embodiment of the present application to obtain a multi-scale fusion representation map of the motion state of the elderly as a multi-scale fusion representation feature of the motion state of the elderly.

[0036] Figure 5 It is a block diagram of a data processing system based on a virtual simulation training system according to an embodiment of the present application. DETAILED DESCRIPTION

[0037] The embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments described herein, but rather these embodiments are provided for a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.

[0038] It should be understood that the various steps described in the method embodiments of the present disclosure may be performed in different orders and / or in parallel. In addition, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present disclosure is not limited in this respect.

[0039] In the description of the embodiments of the present disclosure, the term "including" and similar terms should be understood as open inclusion, that is, "including but not limited to". The term "based on" should be understood as "based at least in part on". The term "one embodiment" or "the embodiment" should be understood as "at least one embodiment". The terms "first", "second", etc. may refer to different or the same objects. Other explicit and implicit definitions may also be included below.

[0040] It should be noted that the modifications of "one" and "plurality" mentioned in the present disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, it should be understood as "one or more".

[0041] As the global population ages, there is a growing concern about the health of the elderly, especially their exercise and mobility. Monitoring and evaluating the exercise status and ability of the elderly is essential to maintaining and improving their quality of life. This process can detect potential health problems in a timely manner so that appropriate preventive or intervention measures can be taken to help the elderly enjoy a healthier and more active life in their later years.

[0042] However, actual exercise testing may put older adults at risk of injury, especially when assessing high-intensity or complex movements, and traditional assessments may require specialized equipment and manual supervision, which is costly in resource-limited settings and may also limit the accuracy of the assessment.

[0043] It should be understood that the virtual simulation training system is a high-tech platform that provides an immersive training and evaluation space through a computer-generated three-dimensional environment. In the assessment of the movement status of the elderly, this system can simulate various daily activities and sports scenes, allowing the elderly to practice and evaluate in a safe and risk-free virtual environment. Such a system can not only help the elderly improve their motor skills, but also detect potential health problems in a timely manner.

[0044] Based on this, the present application proposes a data processing method based on a virtual simulation training system, which obtains a monitoring video of the elderly's movement status, calculates the directional optical flow histogram of the elderly's movement status monitoring video, and uses image analysis and processing technology based on deep learning to extract features and semantic analysis on the directional optical flow histogram, so as to automatically evaluate the elderly's sports ability level based on the shallow features and deep semantic multi-scale fusion features of the directional optical flow histogram, and conducts training based on the sports ability evaluation results and the actual sports ability level label of the elderly object. In this way, intelligent data processing of the virtual simulation training system can be achieved, so as to more accurately capture the dynamic characteristics of the elderly's movement, provide accurate data support for the evaluation, and improve the accuracy and safety of the evaluation.

[0045] Figure 1 It is a flowchart of a data processing method based on a virtual simulation training system according to an embodiment of the present application. Figure 2 FIG. 1 is a schematic diagram of the architecture of a data processing method based on a virtual simulation training system according to an embodiment of the present application. Figure 1 and Figure 2As shown, according to the data processing method based on the virtual simulation training system of the embodiment of the present application, it includes: S110, obtaining the elderly motion state monitoring video; S120, calculating the directional optical flow histogram of the elderly motion state monitoring video; S130, using the motion state feature extractor based on the deep neural network to perform motion state multi-scale feature extraction on the directional optical flow histogram to obtain a motion feature shallow representation map and a motion feature semantic representation map; S140, inputting the motion feature semantic representation map into the gated screening adaptive attention module to obtain a significant motion feature semantic representation map; S150, inputting the motion feature shallow representation map and the motion feature semantic representation map into the gated screening adaptive attention module to obtain a significant motion feature semantic representation map; S150, inputting the motion feature shallow representation map and the motion feature semantic representation map into the gated screening adaptive attention module to obtain a significant motion feature semantic representation map; S151 The semantic representation map of significant motion features is input into the information loss-compensation network to obtain a multi-scale fusion representation map of the motion state of the elderly as a multi-scale fusion representation feature of the motion state of the elderly; S160, the multi-scale fusion representation feature of the motion state of the elderly is passed through a motion ability evaluator to obtain a motion ability evaluation result; and, S170, the cross entropy loss function value between the motion ability evaluation result and the actual motion ability level label of the elderly object is used as the loss function value to train the motion state feature extractor based on the deep neural network, the gated screening adaptive attention module, the information loss-compensation network and the motion ability evaluator.

[0046] In step S110, a monitoring video of the movement status of the elderly is obtained. It should be understood that the monitoring video of the movement status of the elderly refers to video data of the elderly performing physical activities recorded by a camera or other visual capture device. These videos can capture the movement characteristics of the elderly such as movements, speed, balance, coordination, etc. when performing daily activities or specific sports. Based on this, in the technical solution of the present application, the monitoring video of the movement status of the elderly is obtained, and it is analyzed and processed, which can be used to evaluate the elderly's sports ability level and understand their sports performance and potential sports risks.

[0047] In step S120, the directional optical flow histogram of the elderly motion state monitoring video is calculated. Accordingly, considering that in order to be able to more accurately capture and pay attention to the elderly motion state, so as to evaluate the elderly's motor ability, in the technical solution of the present application, the directional optical flow histogram of the elderly motion state monitoring video is calculated. It is worth mentioning that the directional optical flow histogram is a method for representing the motion mode in the image in the field of computer vision, which is obtained by calculating the pixel motion (i.e., optical flow) between two or more frames of images. In particular, the optical flow describes the changes in space and time of each point in the image, which can be used to estimate the speed and direction of movement of the object. That is, by calculating the directional optical flow histogram, a quantitative description of the elderly's motion mode can be provided, and the elderly's motor behavior, such as walking, running, sitting and standing, can be identified and analyzed, and the directional optical flow histogram can reflect the elderly's motor coordination and balance ability, which can provide valuable information for the monitoring and analysis of the elderly's motion state, and help to evaluate their motor ability.

[0048] In step S130, the motion state feature extractor based on the deep neural network is used to extract the motion state multi-scale features of the directional optical flow histogram to obtain a motion feature shallow representation map and a motion feature semantic representation map. In particular, the motion state feature extractor based on the deep neural network is a motion state feature extractor based on the hollow pyramid network. It should be understood that the directional optical flow histogram contains motion feature information of different scales of the motion state of the elderly. Specifically, the directionality of the optical flow can be used as a shallow feature, such as the optical flow pattern generated by the periodic movement of the legs when the elderly walk; and the deep features such as the sudden stop or change of the motion pattern of the elderly may reveal the potential reasons behind this behavior change, such as fatigue and hesitation. Based on this, in order to capture and mine the feature information of different motion scales implied in the directional optical flow histogram, so as to facilitate the accurate evaluation of the motor ability of the elderly, in the technical solution of the present application, the directional optical flow histogram is passed through the motion state feature extractor based on the hollow pyramid network to obtain a motion feature shallow representation map and a motion feature semantic representation map. That is, the atrous pyramid network can capture features of different scales simultaneously, which is very important for understanding the details and overall patterns of the movement state of the elderly. It can capture more abstract and advanced movement information to comprehensively and meticulously display the movement state of the elderly.

[0049] In step S140, the motion feature semantic representation graph is input into the gated screening adaptive attention module to obtain a significant motion feature semantic representation graph. Accordingly, considering that the motion feature correlations displayed in different channels in the motion feature semantic representation graph are different, that is, the motion features expressed in some channels are important for the evaluation of the motor ability level of the elderly, while some channels are not so relevant. Therefore, in order to further enhance the motion feature semantic representation information, and focus on the channel motion semantic information that is most important for the current task, and reduce the interference of irrelevant information, in the technical solution of the present application, the motion feature semantic representation graph is input into the gated screening adaptive attention module to obtain a significant motion feature semantic representation graph. In detail, the gated screening adaptive attention module first performs different ways of pooling and weighted fusion on the motion feature semantic representation graph to extract higher-level motion feature representations in the channels in the motion feature semantic representation graph, and gives different weights according to the importance of the features, strengthens useful features, and suppresses unimportant or redundant features to obtain a global motion feature representation. Then, the global motion feature representation is subjected to channel-by-channel semantic feature interaction and feature activation to enhance the correlation between motion features and improve the model's ability to understand complex motion scenes. Then, the activated features are gated and normalized to effectively select the motion features with important influence and remove noise or unimportant information. Finally, the normalized features are used as weight vectors to weight the motion feature semantic representation map channel by channel, which can better capture the significant motion features between features, such as abnormal behaviors or key actions in the elderly's movements, which is particularly important for understanding complex motion patterns.

[0050] Figure 3 The present invention is a flowchart of inputting the motion feature semantic representation graph into the gated screening adaptive attention module to obtain a significant motion feature semantic representation graph in the data processing method based on the virtual simulation training system according to the embodiment of the present application. Specifically, in the embodiment of the present application, Figure 3As shown, the motion feature semantic representation map is input into the gated screening adaptive attention module to obtain a significant motion feature semantic representation map, including: S210, each feature matrix of the motion feature semantic representation map along the channel dimension is pooled in multiple ways to obtain a global maximum motion feature semantic pooling feature vector, a global mean motion feature semantic pooling feature vector and a global random value motion feature semantic pooling feature vector; S220, the global maximum motion feature semantic pooling feature vector, the global mean motion feature semantic pooling feature vector and the global random value motion feature semantic pooling feature vector are respectively multiplied by position with the corresponding adjustment parameters to obtain a weighted global maximum motion feature semantic pooling feature vector, a weighted global mean motion feature semantic pooling feature vector and a weighted global random value motion feature semantic pooling feature vector; S230, the weighted global maximum motion feature semantic pooling feature vector, the weighted global mean The value motion feature semantic pooling feature vector and the weighted global random value motion feature semantic pooling feature vector are added by position to obtain a global motion feature semantic representation vector; S240, the global motion feature semantic representation vector is subjected to channel-by-channel semantic feature interaction and feature activation based on the fully connected layer to obtain a channel motion feature semantic feature vector; S250, in response to each position feature value in the channel motion feature semantic feature vector being greater than or equal to a predetermined threshold, the position feature value takes the original value, otherwise it is zero, to obtain a gated filtered channel motion feature semantic feature vector; S260, the gated filtered channel motion feature semantic feature vector is normalized to obtain a gated filtered channel motion feature semantic weight feature vector; and, S270, the gated filtered channel motion feature semantic weight feature vector is used as a weight feature vector, and its channel-by-channel product with the motion feature semantic representation map is calculated to obtain the significant motion feature semantic representation map.

[0051] More specifically, in an embodiment of the present application, each feature matrix of the motion feature semantic representation graph along the channel dimension is pooled in multiple ways to obtain a global maximum motion feature semantic pooling feature vector, a global mean motion feature semantic pooling feature vector and a global random value motion feature semantic pooling feature vector, including: performing maximum-based global pooling processing, average-based global pooling processing and random-value-based global pooling processing on each feature matrix of the motion feature semantic representation graph along the channel dimension to obtain the global maximum motion feature semantic pooling feature vector, the global mean motion feature semantic pooling feature vector and the global random value motion feature semantic pooling feature vector.

[0052] More specifically, in an embodiment of the present application, the global motion feature semantic representation vector is subjected to channel-by-channel semantic feature interaction and feature activation based on a fully connected layer to obtain a channel motion feature semantic feature vector, including: calculating the matrix multiplication of the global motion feature semantic representation vector and a weight matrix, and then adding the matrix to the bias vector to obtain a bias-adjusted global motion feature semantic representation vector; and activating the bias-adjusted global motion feature semantic representation vector through a sigmoid activation function to obtain the channel motion feature semantic feature vector.

[0053] In an embodiment of the present application, specifically, the motion feature semantic representation graph is input into a gated screening adaptive attention module to obtain a significant motion feature semantic representation graph, including: using the gated screening adaptive attention module to perform feature saliency on the motion feature semantic representation graph using the following saliency formula to obtain the significant motion feature semantic representation graph; wherein the saliency formula is:

[0054] V c =α·AvgPool(F)+β·MaxPool(F)+γ·StoPool(F)

[0055] A=σ(W*V c +B)

[0056] A " =mask(A)

[0057]

[0058] F′=A′⊙F

[0059] Wherein, F represents the semantic representation graph of the motion feature, AvgPool(·), MaxPool(·) and StoPool(·) are global pooling processing based on average value, global pooling processing based on maximum value and global pooling processing based on random value for each feature matrix along the channel dimension of the feature graph, α, β and γ are the corresponding adjustment parameters, V c represents the global motion feature semantic representation vector, W is the weight matrix, B is the bias vector, σ(·) represents the activation process, A represents the channel motion feature semantic feature vector, mask(·) represents the masking process, A " represents the semantic feature vector of the motion feature of the gated screening channel, n represents the number of feature values ​​in the semantic feature vector of the motion feature of the gated screening channel, and Respectively represent the feature values ​​of the i-th and k-th positions in the semantic feature vector of the motion feature of the gated screening channel, A &′ is the eigenvalue of the i-th position in the semantic weight feature vector of the motion feature of the gated filtering channel, A′ is the semantic weight feature vector of the motion feature of the gated filtering channel, F′ represents the semantic representation map of the significant motion feature, and ⊙ represents the weighted multiplication along the channel dimension.

[0060] In step S150, the shallow representation map of the motion feature and the semantic representation map of the significant motion feature are input into the information loss-compensation network to obtain a multi-scale fusion representation map of the motion state of the elderly as a multi-scale fusion representation feature of the motion state of the elderly. It should be understood that considering that the shallow representation map of the motion feature usually contains basic visual information such as local texture, edge and color of the image, such as a preliminary estimate of the direction and speed of motion, but lacks the ability to express complex motion patterns. The semantic representation map of the significant motion feature contains higher-level semantic features, such as human posture and motion behavior patterns, which can highlight important motion areas or events in the video, such as key motion joints or abnormal behaviors, but is not accurate enough in details, which easily leads to the loss of some intermediate-level motion information. Based on this, in order to effectively compensate for the feature information loss between the shallow representation map of the motion features and the semantic representation map of the significant motion features, so that the fused feature map can obtain a more comprehensive representation of the motion state, in the technical solution of the present application, the shallow representation map of the motion features and the semantic representation map of the significant motion features are input into the information loss-compensation network to obtain a multi-scale fusion representation map of the motion state of the elderly. In this way, the information loss-compensation network can attempt to restore or compensate for the lost motion feature information in the shallow representation map of the motion features and the semantic representation map of the significant motion features, and fuse motion features of different scales. It can simultaneously consider motion features of different scales and improve the recognition ability of motions of different speeds and ranges, so as to more accurately evaluate the motion state of the elderly, identify movement disorders or health risks, and provide a more comprehensive analysis of the elderly's motor ability.

[0061] Figure 4 The present invention is a flowchart of inputting the shallow representation graph of motion features and the semantic representation graph of significant motion features into the information loss-compensation network in the data processing method based on the virtual simulation training system according to the embodiment of the present application to obtain the multi-scale fusion representation graph of the motion state of the elderly as the multi-scale fusion representation feature of the motion state of the elderly. More specifically, in the embodiment of the present application, Figure 4As shown, the motion feature shallow representation map and the significant motion feature semantic representation map are input into the information loss-compensation network to obtain a multi-scale fusion representation map of the motion state of the elderly as a multi-scale fusion representation feature of the motion state of the elderly, including: S310, calculating the compensation loss information between the motion feature shallow representation map and the significant motion feature semantic representation map to obtain a motion state information loss compensation feature map; and, S320, compensating the motion feature shallow representation map based on the motion state information loss compensation feature map to obtain the multi-scale fusion representation map of the motion state of the elderly.

[0062] More specifically, in an embodiment of the present application, the compensation loss information between the motion feature shallow representation map and the significant motion feature semantic representation map is calculated to obtain a motion state information loss compensation feature map, including: downsampling the motion feature shallow representation map to obtain a downsampled motion feature shallow representation map; subtracting the feature map obtained by point convolution of the downsampled motion feature shallow representation map from the significant motion feature semantic representation map by position to obtain a motion state difference feature map; inputting the motion state difference feature map into a tanh function to obtain an activated motion state difference feature map; and upsampling the activated motion state difference feature map to obtain the motion state information loss compensation feature map.

[0063] More specifically, in an embodiment of the present application, the shallow representation map of motion features is compensated based on the motion state information loss compensation feature map to obtain the multi-scale fusion representation map of the motion state of the elderly, including: after calculating the positional product of the shallow representation map of motion features and a first trainable hyperparameter, the obtained weighted motion feature shallow representation map is point-wise multiplied with the motion state information loss compensation feature map to obtain a first compensated motion state feature map; the positional product of the semantic representation map of the salient motion features and a second trainable hyperparameter is calculated to obtain a second compensated motion state feature map; and, the first compensated motion state feature map and the second compensated motion state feature map are positionally added to obtain the multi-scale fusion representation map of the motion state of the elderly.

[0064] In an embodiment of the present application, specifically, the motion feature shallow representation map and the significant motion feature semantic representation map are input into an information loss-compensation network to obtain a multi-scale fusion representation map of the motion state of the elderly, including: using the information loss-compensation network, based on the semantic features of the significant motion feature semantic representation map, the motion feature shallow representation map is feature compensated by the following compensation formula to obtain the multi-scale fusion representation map of the motion state of the elderly; wherein the compensation formula is:

[0065] Loss = UpSampling((tanh(Conv1×1 (DownSampling(F + ))-F2)))

[0066] F s =εF + *Loss+θF <

[0067] Among them, F + represents the shallow representation of the motion feature, F < represents the semantic representation graph of the significant motion feature, DownSampling(·) represents the downsampling process, Conv 1×1 (·) represents point convolution processing, tanh(·) represents tanh function, UpSanmpling(·) represents upsampling processing, Loss represents the motion state information loss compensation feature map, ε and θ are the first trainable hyperparameter and the second trainable hyperparameter respectively, * is the point multiplication operation, F s It is a multi-scale fusion representation diagram of the movement state of the elderly.

[0068] In step S160, the multi-scale fusion representation feature of the elderly motion state is passed through a motor ability evaluator to obtain a motor ability evaluation result. Specifically, in an embodiment of the present application, the multi-scale fusion representation feature of the elderly motion state is passed through a motor ability evaluator to obtain a motor ability evaluation result, including: the multi-scale fusion representation graph of the elderly motion state is passed through a motor ability evaluator based on a classifier to obtain a motor ability evaluation result, and the motor ability evaluation result is used to represent a motor ability grade label. That is, the multi-scale fusion representation feature of the elderly motion state obtained after the information loss compensation using the shallow representation graph of the motion feature and the semantic representation graph of the significant motion feature is classified and processed, so as to automatically evaluate the motor ability grade of the elderly, and train based on the motor ability evaluation result and the real motor ability grade label of the elderly object. In this way, it is possible to realize the intelligent processing of data of the virtual simulation training simulation system, so as to more accurately capture the dynamic characteristics of the elderly movement, provide accurate data support for evaluation, and improve the accuracy and safety of evaluation.

[0069] In step S170, the cross entropy loss function value between the motor ability assessment result and the actual motor ability level label of the elderly subject is used as the loss function value to train the motion state feature extractor based on the deep neural network, the gated screening adaptive attention module, the information loss-compensation network and the motor ability evaluator.

[0070] It should be understood that here, when the cross-depth and cross-scale image semantic feature representations of the directional optical flow histogram of the motion feature shallow representation map and the motion feature semantic representation map are fused through the information loss-compensation network, the cross-depth and cross-scale distribution differences of the image semantics will further cause information compensation differences, thereby causing the image semantic distribution knowledge offset of the multi-scale fusion representation map of the elderly's motion state relative to the source image semantic distribution, thereby causing uncertainty in its image semantic feature distribution relative to the class probability understanding of the classifier, reducing the speed of classification training and the accuracy of the classification results.

[0071] Based on this, in a preferred embodiment, the multi-scale fusion representation graph of the elderly's motion state is passed through a classifier-based motion ability evaluator to obtain a motion ability evaluation result, including: determining n level probability values ​​corresponding to n motion ability level labels respectively obtained by the multi-scale fusion representation graph of the elderly's motion state through the classifier-based motion ability evaluator; calculating the second norm of the probability distribution vector composed of the n level probability values, and dividing the second norm by the square root of n to obtain a level classification probability value, and subtracting the level classification probability value from one to obtain a level classification probability difference; respectively calculating a power function of each eigenvalue of the multi-scale fusion representation graph of the elderly's motion state with the level classification probability value as an exponent and a power function with the level classification probability difference as an exponent to obtain a multi-scale fusion class graph of the elderly's motion state and the elderly. A multi-scale fusion class difference map of human motion state; performing point multiplication of the graded classification probability value and the multi-scale fusion class difference map of the motion state of the elderly to obtain a first multi-scale fusion intermediate class map of the motion state of the elderly, and performing point multiplication of the graded classification probability difference value and the multi-scale fusion class map of the motion state of the elderly to obtain a second multi-scale fusion intermediate class map of the motion state of the elderly; after performing point multiplication of the first multi-scale fusion intermediate class map of the motion state of the elderly with the multi-scale fusion intermediate class map of the motion state of the elderly, further performing point addition with the dot product result of the graded classification probability value and the multi-scale fusion class map of the motion state of the elderly to obtain an optimized multi-scale fusion representation map of the motion state of the elderly; and, passing the optimized multi-scale fusion representation map of the motion state of the elderly through the classifier-based motion ability evaluator to obtain the motion ability evaluation result.

[0072] That is, when the multi-scale fusion representation graph of the elderly's motion state is classified by the classifier-based motion ability evaluator, in order to achieve unsupervised domain adaptation from the feature distribution domain of the multi-scale fusion representation graph of the elderly's motion state to the probability distribution domain of the classification probability, the class probability joint representation of each motion ability level label obtained by the classifier-based motion ability evaluator of the multi-scale fusion representation graph of the elderly's motion state is used as a domain agent, and the probability distribution of the multi-scale fusion representation graph of the elderly's motion state is moved and averaged by interacting with the power function class distribution as an exponent, and the knowledge transfer from the unlabeled features to be classified to the labeled probability distribution is realized by superimposing the feature domain probability distribution knowledge of the multi-scale fusion representation graph of the elderly's motion state itself, thereby promoting the classification operation of the multi-scale fusion representation graph of the elderly's motion state through the classifier-based motion ability evaluator, that is, improving the speed of classification training of the classifier-based motion ability evaluator and the accuracy of classification results. In this way, the data intelligent processing of the virtual simulation training system can be realized, so as to more accurately capture the dynamic characteristics of the elderly's motion, provide accurate data support for the evaluation, and improve the accuracy and safety of the evaluation.

[0073] In summary, the data processing method based on the virtual simulation training system according to the embodiment of the present application is explained, which obtains the elderly's motion status monitoring video, calculates the directional optical flow histogram of the elderly's motion status monitoring video, and uses deep learning-based image analysis and processing technology to perform feature extraction and semantic analysis on the directional optical flow histogram, so as to automatically evaluate the elderly's athletic ability level based on the shallow features and deep semantic multi-scale fusion features of the directional optical flow histogram, and conducts training based on the athletic ability evaluation results and the elderly object's real athletic ability level label. In this way, the intelligent processing of the data of the virtual simulation training system can be realized, so as to more accurately capture the dynamic characteristics of the elderly's movement, provide accurate data support for the evaluation, and improve the accuracy and safety of the evaluation.

[0074] Figure 5 FIG. 1 is a block diagram of a data processing system based on a virtual simulation training system according to an embodiment of the present application. Figure 5As shown, according to the data processing system 100 based on the virtual simulation training system of the embodiment of the present application, it includes: a motion state monitoring video acquisition module 110, which is used to obtain the motion state monitoring video of the elderly; a directional optical flow histogram calculation module 120, which is used to calculate the directional optical flow histogram of the motion state monitoring video of the elderly; a motion state multi-scale feature extraction module 130, which is used to use the motion state feature extractor based on the deep neural network to perform motion state multi-scale feature extraction on the directional optical flow histogram to obtain a motion feature shallow representation map and a motion feature semantic representation map; a significant motion feature extraction module 140, which is used to input the motion feature semantic representation map into the gated screening adaptive attention module to obtain a significant motion feature semantic representation map; a multi-scale feature fusion module 140; a plurality of multi-scale feature extraction modules 150, which are used to extract the motion feature semantic representation map from the directional optical flow histogram using the motion state feature extractor based on the deep neural network to obtain a motion feature shallow representation map and a motion feature semantic representation map; a plurality of multi-scale feature extraction modules 150, which are used to extract the motion feature semantic representation map from the directional optical flow histogram ... Module 150 is used to input the shallow representation map of motion features and the semantic representation map of significant motion features into an information loss-compensation network to obtain a multi-scale fusion representation map of the motion state of the elderly as a multi-scale fusion representation feature of the motion state of the elderly; a motion ability assessment result generation module 160 is used to pass the multi-scale fusion representation feature of the motion state of the elderly through a motion ability evaluator to obtain a motion ability assessment result; and a training module 170 is used to train the motion state feature extractor based on a deep neural network, the gated screening adaptive attention module, the information loss-compensation network and the motion ability evaluator using the cross entropy loss function value between the motion ability assessment result and the actual motion ability level label of the elderly object as the loss function value.

[0075] Here, those skilled in the art can understand that the specific operations of each step in the data processing system based on the virtual simulation training system have been referred to above. Figures 1 to 4 It has been introduced in detail in the description of the data processing method based on the virtual simulation training system, and therefore, its repeated description will be omitted.

[0076] As described above, the data processing system 100 based on the virtual simulation training system according to the embodiment of the present disclosure can be implemented in various wireless terminals, such as a server with a data processing algorithm based on the virtual simulation training system. In one possible implementation, the data processing system 100 based on the virtual simulation training system according to the embodiment of the present disclosure can be integrated into the wireless terminal as a software module and / or a hardware module. For example, the data processing system 100 based on the virtual simulation training system can be a software module in the operating system of the wireless terminal, or it can be an application developed for the wireless terminal; of course, the data processing system 100 based on the virtual simulation training system can also be one of the many hardware modules of the wireless terminal.

[0077] Alternatively, in another example, the data processing system 100 based on the virtual simulation training system and the wireless terminal may also be separate devices, and the data processing system 100 based on the virtual simulation training system may be connected to the wireless terminal via a wired and / or wireless network and transmit interactive information in accordance with an agreed data format.

[0078] The foregoing is merely an example of the principles of the present disclosure, and various modifications may be made by those skilled in the art without departing from the scope of the present disclosure. The above embodiments are presented for the purpose of illustration and not limitation. The present disclosure may also take many forms other than those explicitly described herein. Therefore, it is emphasized that the present disclosure is not limited to the methods, systems and devices explicitly disclosed, but is intended to include variations and modifications within the spirit and scope of the appended claims.

Claims

1. A data processing method based on a virtual simulation training system, characterized in that: include: Obtain monitoring videos of the elderly’s exercise status; Calculating the directional optical flow histogram of the elderly person's motion status monitoring video; Using the motion state feature extractor based on the deep neural network to perform motion state multi-scale feature extraction on the directional optical flow histogram to obtain a motion feature shallow representation map and a motion feature semantic representation map; Inputting the motion feature semantic representation map into a gated screening adaptive attention module to obtain a significant motion feature semantic representation map; Inputting the shallow representation map of motion features and the semantic representation map of significant motion features into an information loss-compensation network to obtain a multi-scale fusion representation map of the motion state of the elderly as a multi-scale fusion representation feature of the motion state of the elderly; Passing the multi-scale fusion representation features of the elderly's movement state through a movement ability evaluator to obtain a movement ability evaluation result; The cross entropy loss function value between the motor ability assessment result and the actual motor ability level label of the elderly subject is used as the loss function value to train the motion state feature extractor based on the deep neural network, the gated screening adaptive attention module, the information loss-compensation network and the motor ability evaluator.

2. The data processing method based on the virtual simulation training system according to claim 1 is characterized in that: The motion state feature extractor based on deep neural network is a motion state feature extractor based on hollow pyramid network.

3. The data processing method based on the virtual simulation training system according to claim 2 is characterized in that: Inputting the motion feature semantic representation graph into a gated screening adaptive attention module to obtain a significant motion feature semantic representation graph, including: Performing pooling processing in multiple ways on each feature matrix of the motion feature semantic representation graph along the channel dimension to obtain a global maximum motion feature semantic pooling feature vector, a global mean motion feature semantic pooling feature vector and a global random value motion feature semantic pooling feature vector; The global maximum motion feature semantic pooling feature vector, the global mean motion feature semantic pooling feature vector and the global random value motion feature semantic pooling feature vector are respectively multiplied by position with corresponding adjustment parameters to obtain a weighted global maximum motion feature semantic pooling feature vector, a weighted global mean motion feature semantic pooling feature vector and a weighted global random value motion feature semantic pooling feature vector; The weighted global maximum motion feature semantic pooling feature vector, the weighted global mean motion feature semantic pooling feature vector and the weighted global random value motion feature semantic pooling feature vector are added by position to obtain a global motion feature semantic representation vector; Performing channel-by-channel semantic feature interaction and feature activation based on a fully connected layer on the global motion feature semantic representation vector to obtain a channel motion feature semantic feature vector; In response to each position feature value in the channel motion feature semantic feature vector being greater than or equal to a predetermined threshold, the position feature value takes an original value, otherwise it is zero, so as to obtain a gated filtered channel motion feature semantic feature vector; Normalizing the gated screening channel motion feature semantic feature vector to obtain a gated screening channel motion feature semantic weight feature vector; The gated screening channel motion feature semantic weight feature vector is used as a weight feature vector, and its channel-by-channel product with the motion feature semantic representation map is calculated to obtain the significant motion feature semantic representation map.

4. The data processing method based on the virtual simulation training system according to claim 3 is characterized in that: The various feature matrices of the motion feature semantic representation graph along the channel dimension are respectively pooled in multiple ways to obtain a global maximum motion feature semantic pooling feature vector, a global mean motion feature semantic pooling feature vector and a global random value motion feature semantic pooling feature vector, including: the various feature matrices of the motion feature semantic representation graph along the channel dimension are respectively pooled based on the maximum value, based on the average value and based on the random value to obtain the global maximum motion feature semantic pooling feature vector, the global mean motion feature semantic pooling feature vector and the global random value motion feature semantic pooling feature vector.

5. The data processing method based on the virtual simulation training system according to claim 4 is characterized in that: The global motion feature semantic representation vector is subjected to channel-by-channel semantic feature interaction and feature activation based on a fully connected layer to obtain a channel motion feature semantic feature vector, including: Calculating the matrix multiplication of the global motion feature semantic representation vector and the weight matrix and then adding the vector to the bias vector to obtain the bias-adjusted global motion feature semantic representation vector; The bias-adjusted global motion feature semantic representation vector is activated by a sigmoid activation function to obtain the channel motion feature semantic feature vector.

6. The data processing method based on the virtual simulation training system according to claim 5 is characterized in that: Inputting the shallow representation map of motion features and the semantic representation map of significant motion features into an information loss-compensation network to obtain a multi-scale fusion representation map of the motion state of the elderly as a multi-scale fusion representation feature of the motion state of the elderly, including: Calculating compensation loss information between the shallow representation map of the motion feature and the semantic representation map of the significant motion feature to obtain a motion state information loss compensation feature map; The motion feature shallow representation map is compensated based on the motion state information loss compensation feature map to obtain the multi-scale fusion representation map of the motion state of the elderly.

7. The data processing method based on the virtual simulation training system according to claim 6 is characterized in that: Calculating compensation loss information between the shallow representation map of the motion feature and the semantic representation map of the significant motion feature to obtain a motion state information loss compensation feature map, including: Downsampling the shallow representation map of motion features to obtain a downsampled shallow representation map of motion features; Subtracting the feature map obtained by point convolution of the downsampled motion feature shallow representation map from the significant motion feature semantic representation map by position to obtain a motion state difference feature map; Inputting the motion state difference characteristic map into a tanh function to obtain an activated motion state difference characteristic map; The activation motion state difference feature map is up-sampled to obtain the motion state information loss compensation feature map.

8. The data processing method based on the virtual simulation training system according to claim 7 is characterized in that: The motion feature shallow representation map is compensated based on the motion state information loss compensation feature map to obtain the multi-scale fusion representation map of the motion state of the elderly, including: After calculating the positional product of the shallow representation map of the motion feature and the first trainable hyperparameter, multiplying the obtained shallow representation map of the weighted motion feature by the motion state information loss compensation feature map by position points to obtain a first compensated motion state feature map; Calculating the positional product of the significant motion feature semantic representation map and a second trainable hyperparameter to obtain a second compensated motion state feature map; The first compensating motion state feature map and the second compensating motion state feature map are added according to position to obtain the multi-scale fusion representation map of the motion state of the elderly.

9. The data processing method based on the virtual simulation training system according to claim 8 is characterized in that: The multi-scale fusion representation features of the elderly's movement state are passed through a movement ability evaluator to obtain a movement ability assessment result, including: passing the multi-scale fusion representation graph of the elderly's movement state through a classifier-based movement ability evaluator to obtain a movement ability assessment result, and the movement ability assessment result is used to represent a movement ability level label.

10. A data processing system based on a virtual simulation training system, characterized in that: include: The motion status monitoring video acquisition module is used to obtain the motion status monitoring video of the elderly; A directional optical flow histogram calculation module, used to calculate the directional optical flow histogram of the elderly motion status monitoring video; A motion state multi-scale feature extraction module, used to use the motion state feature extractor based on the deep neural network to perform motion state multi-scale feature extraction on the directional optical flow histogram to obtain a motion feature shallow representation map and a motion feature semantic representation map; A significant motion feature extraction module, used for inputting the motion feature semantic representation map into a gated screening adaptive attention module to obtain a significant motion feature semantic representation map; A multi-scale feature fusion module, used for inputting the shallow representation map of motion features and the semantic representation map of significant motion features into an information loss-compensation network to obtain a multi-scale fusion representation map of the motion state of the elderly as a multi-scale fusion representation feature of the motion state of the elderly; A motor ability assessment result generation module, used for passing the multi-scale fusion representation features of the elderly's motor state through a motor ability evaluator to obtain a motor ability assessment result; A training module is used to train the motion state feature extractor based on the deep neural network, the gated screening adaptive attention module, the information loss-compensation network and the motion ability evaluator by using the cross entropy loss function value between the motion ability assessment result and the actual motion ability level label of the elderly subject as the loss function value.

Citation Information

Patent Citations

  • Online defect detection system and method for carbon fiber sucker rod

    CN118429335A

  • Combustible gas intelligent detection system and method based on big data

    CN118482874A

  • Method for efficiently and statically fermenting cattle bed padding

    CN118805693A

  • Production method and equipment of energy-saving building material dilute stone concrete wallboard

    CN118809767A

  • Early warning analysis method and system for financial data and readable storage medium

    CN118822722A

Cited By

  • Traffic shunting simulation method and device for rail transit transportation vehicle

    CN122239515A