Electronic data processing system based on deep learning

Abstract

The invention discloses an electronic data processing system based on deep learning, and relates to the technical field of deep learning, and the system comprises a basic metabolism analysis module which is configured to obtain the basic energy consumption of a to-be-tested user based on the basic information of the to-be-tested user when the to-be-tested user is in a ready state, and create a user body digital twinborn model; the feature extraction module is configured to shoot fitness actions of a to-be-detected user based on a high-definition camera to obtain shot video information, and performs data processing and multi-dimensional feature extraction to obtain key video frame information; the action category analysis module is configured to effectively capture local modes of fitness actions of a to-be-tested user, analyze an action complete period to output action preliminary judgment information, perform dynamic weighting processing to output action category judgment information, perform residual error connection operation and regularization operation, and perform action classification analysis on the to-be-tested user. And judging probability distribution of different action categories of the fitness actions to obtain action category probability distribution information. The method has the effect of improving the electronic data processing efficiency based on deep learning.

Description

TECHNICAL FIELD

[0001] The application relates to the technical field of deep learning, in particular to an electronic data processing system based on deep learning. BACKGROUND

[0002] In the existing electronic data processing system based on deep learning in the fitness field, the core of fitness action recognition and counting is to rely on human posture estimation technology, which has been upgraded from 2D to 3D at present, and the human key points in the image are extracted through a convolutional neural network, and the action type is judged based on the angle change of the key points, but it is easy to be affected by the difference in viewing angle, resulting in low accuracy of fitness action recognition and counting, and further resulting in low efficiency of electronic data processing based on deep learning, which needs to be improved. SUMMARY

[0003] In order to improve the efficiency of electronic data processing in the fitness field, the application provides an electronic data processing system based on deep learning.

[0004] The electronic data processing system based on deep learning provided by the application adopts the following technical scheme:

[0005] An electronic data processing system based on deep learning comprises:

[0006] A basal metabolism analysis module is configured to, when a to-be-tested user is ready, acquire basal information of the to-be-tested user, judge the basal metabolism of the to-be-tested user to obtain basal energy consumption of the to-be-tested user based on the basal information of the to-be-tested user, and create a user body digital twin model based on the basal information of the to-be-tested user;

[0007] A feature extraction module is configured to acquire shooting video information by shooting the fitness action of the to-be-tested user based on a high-definition camera, perform data processing operation and multi-dimensional feature extraction on the shooting video information to obtain key video frame information, and adjust the user body digital twin model;

[0008] An action category analysis module is configured to effectively capture the local mode of the fitness action of the to-be-tested user to obtain user local spatiotemporal feature information, analyze the complete cycle of the action to output action preliminary judgment information, dynamically weight process to output action category judgment information, perform residual connection operation and regularization operation, judge the probability distribution of different action categories in the fitness action to obtain action category probability distribution information;

[0009] A force line direction judgment module is configured to judge whether the force line direction of the fitness action of the to-be-tested user is correct based on the user local spatiotemporal feature information, and record force line direction abnormal information if the force line direction is incorrect;

[0010] an action counting module, configured to create a plurality of action state machines, and count each action category of the user to obtain an action quantity of each action category based on each action state machine;

[0011] a heat calculation module, configured to obtain a movement energy consumption of the user based on the action category judgment information, the action completion time length and the additional weight, and obtain a total energy consumption of the user based on the movement energy consumption and the basal energy consumption of the user;

[0012] an interaction recording module, configured to visually display the user body digital twin model, the key video frame information, the action category judgment information, the action quantity and the total energy consumption in real time.

[0013] Preferably, the basal information of the user to be detected is obtained and cached, and the basal information of the user to be detected includes age information, gender information, weight information, height information, body dimension information and muscle information.

[0014] It is judged whether the environment and the state of the user to be detected are suitable, and if suitable, a user ready signal is output, and if not suitable, the environment and the state of the user to be detected are adjusted until the user ready signal is output when the environment and the state of the user to be detected are suitable.

[0015] When the user ready signal is received, the basal metabolism of the user to be detected is preliminarily detected based on the age information, the gender information, the weight information and the height information of the user to be detected, and preliminary user basal metabolism information is obtained.

[0016] Based on the muscle information of the user to be detected, the influence of the muscle mass of the user to be detected on the basal metabolism of the user to be detected is judged to obtain a first basal metabolism compensation value.

[0017] Based on the body dimension information of the user to be detected, the body surface area of the user to be detected is judged, and based on the body surface area of the user to be detected, the influence of the body surface area on the basal metabolism of the user to be detected is judged to obtain a second basal metabolism compensation value.

[0018] The first basal metabolism compensation value and the second basal metabolism compensation value are added to obtain the basal metabolism compensation value of the user to be detected.

[0019] The preliminary user basal metabolism information is adjusted based on the basal metabolism compensation value of the user to be detected to obtain the basal energy consumption of the user to be detected.

[0020] The muscle information of the user to be detected includes muscle mass information and muscle distribution information, and the user body digital twin model is created based on the weight information, the height information, the body dimension information, the muscle mass information and the muscle distribution information of the user to be detected.

[0021] Preferably, a high-definition camera is acquired and installed, and a user's fitness action is captured in real time based on the high-definition camera to obtain captured video information;

[0022] The captured video information is subjected to data preprocessing operations, i.e., image scaling operations and RGB conversion operations, to obtain video data preprocessing information;

[0023] Based on a data enhancement strategy, the video data preprocessing information is subjected to time enhancement processing and spatial enhancement processing to obtain video data enhancement information;

[0024] The video data enhancement information is subjected to data smoothing and noise reduction processing using a Savitzky-Golay filter to obtain video data smoothing information;

[0025] A video feature extractor is acquired, and a signal connection link between the video feature extractor and the high-definition camera is established;

[0026] Based on MediaPipe in the video feature extractor, posture detection is performed on the captured video information to extract multi-dimensional features of key features, thereby obtaining posture multi-dimensional key feature information;

[0027] The posture multi-dimensional key feature information is displayed on the user's body digital twin model;

[0028] Based on the posture multi-dimensional key feature information, it is determined whether there is missing frame key point data, and if there is missing frame key point data, linear interpolation compensation processing is performed to obtain key point interpolation information;

[0029] The video data smoothing information and the key point interpolation information are combined to form key video frame information;

[0030] Based on the key video frame information, the user's body digital twin model is adjusted, improved, and updated.

[0031] Preferably, based on MediaPipe, posture detection is performed on the captured video information to identify the center of the user's hips and the user's shoulder width. The center of the user's hips is used as the origin for coordinate normalization, and the user's shoulder width is used as the scale factor for standardization to obtain normalized coordinate feature information;

[0032] Based on MediaPipe, posture detection is performed on the captured video information to identify the angles between the various parts of the user's body to obtain body part angle feature information;

[0033] Based on MediaPipe, posture detection is performed on the captured video information to identify the distances between the various parts of the user's body to obtain body part distance feature information;

[0034] The MediaPipe is used for posture detection on the photographed video information, and the speed of each joint of the body of the to-be-detected user and the acceleration of each joint of the body of the to-be-detected user are recognized to obtain motion feature information;

[0035] The MediaPipe is used for posture detection on the photographed video information, and the symmetry of the left and right limbs of the to-be-detected user is recognized to obtain symmetry difference feature information;

[0036] The normalized coordinate feature information, the body part angle feature information, the body part distance feature information, the motion feature information, and the symmetry difference feature information are combined to form posture multi-dimensional key feature information.

[0037] Preferably, a deep learning model is obtained, and the deep learning model architecture mixes a convolutional neural network and a bidirectional long short-term memory network;

[0038] The convolutional neural network in the deep learning model effectively captures and extracts the local mode of the fitness action of the to-be-detected user in the key video frame information, and outputs user local spatiotemporal feature information;

[0039] The user local spatiotemporal feature information is input into the bidirectional long short-term memory network, the bidirectional long short-term memory network learns the forward and backward dependency relationship of each fitness action sequence in the user local spatiotemporal feature information, understands and judges the complete cycle of each fitness action of the to-be-detected user, and obtains action preliminary judgment information;

[0040] An attention mechanism is introduced, based on the action preliminary judgment information, different time step features are dynamically weighted, the importance of the key frame of the fitness action of the to-be-detected user is highlighted, and action category judgment information is output;

[0041] Based on the full connection layer in the deep learning model, residual connection operation is performed on the action category judgment information, and neural network regularization is performed based on LayerNorm + Dropout, the probability distribution of different action categories in the fitness action of the to-be-detected user is judged, and action category probability distribution information is output.

[0042] Preferably, a fitness standard action database is obtained, and the fitness standard action database includes standard action force line directions of each fitness action category;

[0043] The standard action force line directions of each fitness action category in the fitness standard action database are used as training data, and the user body digital twin model is trained to generate standard action force line directions of each fitness action category of the to-be-detected user, and user customized action force line directions are output;

[0044] The force line direction of each fitness motion of the to-be-tested user is recognized based on the user local spatiotemporal feature information and the motion category judgment information, and the user motion force line direction is obtained.

[0045] The user motion force line direction is compared with the user customized motion force line direction to determine whether there is a difference in the force line direction. If there is a difference, it is determined that the force line direction of the to-be-tested user is abnormal, a force line direction abnormality signal is output, and force line direction abnormality information is recorded. The force line direction abnormality information includes force line direction abnormality position, force line direction abnormality timestamp, and force line direction abnormality degree.

[0046] The force line direction abnormality information is displayed in real time on the user body digital twin model.

[0047] Preferably, based on the motion category judgment information and the motion category probability distribution information, a corresponding motion state machine is created for each fitness motion of the to-be-tested user.

[0048] The historical fitness motion video of the to-be-tested user is obtained, and the actual motion amplitude of the to-be-tested user is determined based on the historical fitness motion video of the to-be-tested user, and the motion state threshold in the motion state machine is pre-set.

[0049] Based on each motion state machine, each fitness motion of the to-be-tested user in the key video frame information is counted respectively to obtain the motion quantity of each motion category.

[0050] Preferably, a MET value database is obtained, and the MET value database includes MET values of each fitness motion.

[0051] The completion time of each fitness motion of the to-be-tested user is counted to obtain a plurality of sub-motion completion times.

[0052] The additional weight of each fitness motion of the to-be-tested user is counted to obtain additional weight information.

[0053] Based on the motion category judgment information, the motion category probability distribution information, the MET values of each fitness motion, the sub-motion completion times, and the additional weight information, the energy consumed by each fitness motion of the to-be-tested user is determined to obtain the motion energy consumption of each fitness motion.

[0054] The motion energy consumption of each fitness motion is superimposed to obtain the motion energy consumption of the to-be-tested user.

[0055] The basic energy consumption of the to-be-tested user and the motion energy consumption of the to-be-tested user are superimposed to obtain the total energy consumption of the to-be-tested user.

[0056] Preferably, a human-computer interaction interface is obtained, and the human-computer interaction interface includes a camera picture area and an information panel area.

[0057] The camera picture area is used for center display of the key video frame information;

[0058] The overlay is used for superimposing the human key point skeleton and the joint point on the center displayed key video frame information, wherein the human key point skeleton is marked by a green connecting line, and the human joint point is marked by a red dot, and the user's body digital twin model is superimposed on the key video frame information;

[0059] When the force line direction abnormal signal is received, the force line direction abnormal information is displayed by flashing marking in the key video frame information;

[0060] The information panel area is used for real-time visual display of the action category judgment information, the action category probability distribution information, the action motion quantity of each action category, the motion energy consumption of each fitness action and the total energy consumption of the to-be-measured user.

[0061] In summary, the present application has the following at least one beneficial technical effect:

[0062] 1. The posture of the shooting video information is detected by MediaPipe to obtain normalized coordinate feature information, the angle between each part of the body of the to-be-measured user is recognized to obtain body part angle feature information, the distance between each part of the body of the to-be-measured user is recognized to obtain body part distance feature information, the speed of each joint of the body of the to-be-measured user and the acceleration of each joint of the body of the to-be-measured user are recognized to obtain motion feature information, the symmetry of the left and right limbs of the to-be-measured user is recognized to obtain symmetry difference feature information, and multi-modal feature fusion is performed on the fitness action of the to-be-measured user, thereby improving the feature extraction accuracy of the fitness action data of the to-be-measured user, and further improving the electronic data processing efficiency based on deep learning.

[0063] 2. The deep learning model architecture of the deep learning model architecture mixed with the convolutional neural network and the bidirectional long short-term memory network is used, and the attention mechanism is introduced to focus on the key frame of the fitness action of the to-be-measured user, thereby improving the fitness action category judgment accuracy of the to-be-measured user, and further improving the electronic data processing efficiency based on deep learning. BRIEF DESCRIPTION OF DRAWINGS

[0064] Figure 1 The present embodiment mainly embodies the module schematic diagram of the electronic data processing system based on deep learning.

[0065] Mark: 1, basal metabolism analysis module; 2, feature extraction module; 3, action category analysis module; 4, force line direction judgment module; 5, action counting module; 6, heat calculation module; 7, interaction recording module. DETAILED DESCRIPTION

[0066] The application will be further described in detail below with reference to the accompanying drawings.

[0067] The embodiment of the application discloses an electronic data processing system based on deep learning.

[0068] With reference to Figure 1 An electronic data processing system based on deep learning comprises:

[0069] A basal metabolism analysis module is configured to, when a user to be tested is ready, acquire basal information of the user to be tested, judge basal metabolism of the user to be tested based on the basal information, obtain basal energy consumption of the user to be tested, and create a digital twin model of a body of the user based on the basal information of the user to be tested.

[0070] The basal metabolism analysis module has the following specific implementation manners:

[0071] The basal information of the user to be tested is acquired and cached, and the basal information of the user to be tested includes age information, gender information, weight information, height information, body shape dimension information and muscle information.

[0072] It is judged whether the environment and state of the user to be tested are suitable, and if so, a user readiness signal is output, and if not, the environment and state of the user to be tested are adjusted until the user readiness signal is output when the detection state is suitable.

[0073] In actual application, basal metabolism refers to the minimum energy required by all organs of the human body to maintain life. The basal metabolism determination method is the energy metabolism rate of the human body in a clear and quiet state without the influence of muscle activity, environmental temperature and mental stress.

[0074] Specifically, suitable detection parameter information is acquired, and the suitable detection parameter information includes a suitable heart rate interval, a suitable respiratory frequency interval and a suitable environmental temperature range.

[0075] The heart rate of the user to be tested is detected, and it is judged whether the user to be tested is in the suitable heart rate interval, and if so, a first type of user preparation completion signal is output.

[0076] The respiratory frequency of the user to be tested is detected, and it is judged whether the user to be tested is in the suitable respiratory frequency interval, and if so, a second type of user preparation completion signal is output.

[0077] The ambient temperature of the user to be tested is detected, and it is judged whether the user to be tested is in the suitable ambient temperature range, and if so, a third type of user preparation completion signal is output.

[0078] When the first type of user preparation completion signal is received, and the second type of user preparation completion signal is received, and the third type of user preparation completion signal is received, it is judged that the user to be tested is ready, and the detection of the basal metabolism of the user to be tested can be carried out, and a user ready signal is output.

[0079] When the user ready signal is received, the basal metabolism of the user to be tested is preliminarily detected based on the age information, gender information, weight information and height information of the user to be tested, and preliminary user basal metabolism information is obtained.

[0080] Based on the muscle information of the user to be tested, the influence of the muscle mass of the user to be tested on the basal metabolism of the user to be tested is judged to obtain a first basal metabolism compensation value. Wherein, the greater the muscle mass information is, the higher the first basal metabolism compensation value is.

[0081] Based on the body dimension information of the user to be tested, the body surface area of the user to be tested is judged, and based on the body surface area of the user to be tested, the influence of the body surface area on the basal metabolism of the user to be tested is judged to obtain a second basal metabolism compensation value. In actual application, the body dimension affects the body surface area, the greater the body surface area is, the greater the heat dissipation of the body to the external environment is, and the higher the basal metabolism is, therefore, the greater the body surface area is, the higher the second basal metabolism compensation value is.

[0082] The first basal metabolism compensation value and the second basal metabolism compensation value are added to obtain the basal metabolism compensation value of the user to be tested.

[0083] Based on the basal metabolism compensation value of the user to be tested, the preliminary user basal metabolism information is adjusted to obtain the basal energy consumption of the user to be tested.

[0084] The muscle information of the user to be tested includes muscle mass information and muscle distribution information, and based on the weight information, height information, body dimension information, muscle mass information and muscle distribution information of the user to be tested, a user body digital twin model is created.

[0085] The feature extraction module is configured to obtain shooting video information based on the high-definition camera shooting the fitness action of the user to be tested, perform data processing operation and multi-dimensional feature extraction on the shooting video information to obtain key video frame information, and adjust the user body digital twin model.

[0086] Specific execution modes of the action category analysis module include:

[0087] The high-definition camera is obtained and installed, and the high-definition camera is used to shoot the fitness action of the user to be tested in real time to obtain shooting video information.

[0088] The shooting video information is subjected to data preprocessing operation, i.e. image scaling operation and RGB conversion operation, to obtain video data preprocessing information.

[0089] Based on the data enhancement strategy, the video data pre-processing information is subjected to time enhancement processing and space enhancement processing to obtain video data enhancement information.

[0090] It should be pointed out that, in the embodiments of the present application, time enhancement processing is performed by frame skipping sampling or interpolation sampling, and space enhancement processing is performed by adding Gaussian noise, thereby improving data robustness.

[0091] The Savitzky-Golay filter is used to perform data smoothing and noise reduction processing on the video data enhancement information to obtain video data smoothing information.

[0092] In actual application, the Savitzky-Golay filter is a digital filter that can be applied to a group of data to smooth the data and improve the accuracy of the data without changing the trend and width of the signal.

[0093] A video feature extractor is acquired, and a signal connection link between the video feature extractor and the high-definition camera is established.

[0094] Based on MediaPipe in the video feature extractor, posture detection is performed to extract multi-dimensional features of key features in the photographed video information, and posture multi-dimensional key feature information is obtained. Specifically, it includes:

[0095] Based on MediaPipe, posture detection is performed on the photographed video information to identify the hip center of the user to be tested and the shoulder width of the user to be tested. The hip center of the user to be tested is taken as the origin for coordinate normalization, and the shoulder width of the user to be tested is taken as the scale factor for standardization to obtain normalized coordinate feature information.

[0096] Based on MediaPipe, posture detection is performed on the photographed video information to identify the angle between each part of the body of the user to be tested to obtain body part angle feature information.

[0097] Specifically, the body part angle feature information includes the angle of the left and right arms of the user to be tested (i.e., the angle of the shoulder-elbow-wrist of the left and right arms), the angle of the left and right legs of the user to be tested (i.e., the angle of the hip-knee-ankle of the left and right legs), the torso angle, and the spinal curvature angle.

[0098] Based on MediaPipe, posture detection is performed on the photographed video information to identify the distance between each part of the body of the user to be tested to obtain body part distance feature information.

[0099] Specifically, the body part distance feature information includes the distance between the two wrists of the user to be tested, the distance between the two ankles of the user to be tested, the shoulder width, the hip width, the body width, and the distance from the hand to the hip.

[0100] The MediaPipe is used for posture detection of the photographed video information, and the speed of each joint of the body of the to-be-detected user and the acceleration of each joint of the body of the to-be-detected user are recognized to obtain motion feature information.

[0101] The MediaPipe is used for posture detection of the photographed video information, and the symmetry of the left and right limbs of the to-be-detected user is recognized to obtain symmetry difference feature information.

[0102] Specifically, the symmetry difference feature information includes a length difference between left and right arms of the to-be-detected user and a length difference between left and right legs of the to-be-detected user.

[0103] The normalized coordinate feature information, the body part angle feature information, the body part distance feature information, the motion feature information, and the symmetry difference feature information are combined to form posture multi-dimensional key feature information.

[0104] In actual application, the MediaPipe is an open-source framework developed by Google Research, which is used to build machine learning pipelines, especially for processing time series data such as video and audio. It can be used for various tasks, including face detection, hand detection, and posture detection.

[0105] The posture multi-dimensional key feature information is displayed on the user body digital twin model.

[0106] Based on the posture multi-dimensional key feature information, it is determined whether there is missing key point data of the frame. If there is missing key point data of the frame, linear interpolation compensation processing is performed to obtain key point interpolation information.

[0107] The video data smoothing information and the key point interpolation information are combined to form key video frame information.

[0108] In actual application, the video feature extractor is a Python-based tool kit that efficiently extracts video features using deep convolutional neural networks.

[0109] Based on the key video frame information, the user body digital twin model is adjusted, improved, and updated.

[0110] The action category analysis module is configured to effectively capture the local pattern of the to-be-detected user's fitness action to obtain user local spatiotemporal feature information, analyze the complete cycle of the action to output action preliminary judgment information, dynamically weight to output action category judgment information, perform residual connection operation and regularization operation, judge the probability distribution of different action categories in the fitness action to obtain action category probability distribution information.

[0111] Specifically, the action category analysis module includes:

[0112] The deep learning model is obtained, and the deep learning model architecture mixes a convolutional neural network and a bidirectional long short-term memory network.

[0113] The convolutional neural network in the deep learning model effectively captures and extracts local patterns (such as the moment of arm bending) of the user's fitness action in the key video frame information, and outputs user local spatiotemporal feature information.

[0114] It should be pointed out that the convolutional neural network in the embodiment of the present application includes 3 layers of 1D convolutional layers (channel number 64→128→128), wherein each layer includes convolution, batch normalization, ReLU activation, pooling and Dropout operation, and the key video frame information is processed based on the 3 layers of 1D convolutional layers, the user's fitness action is captured, and the user's local spatiotemporal feature information is extracted.

[0115] The user local spatiotemporal feature information is input into the bidirectional long short-term memory network, the bidirectional long short-term memory network learns the forward and backward dependency relationship of each fitness action sequence in the user local spatiotemporal feature information, understands and judges the complete cycle of each fitness action of the user to be tested (such as the "down-up" cycle of push-ups), and obtains action preliminary judgment information.

[0116] The attention mechanism is introduced, based on the action preliminary judgment information, the features of different time steps are dynamically weighted, the importance of the key frame of the fitness action of the user to be tested is highlighted, and the action category judgment information is output. For example, in the deep squat action, the attention mechanism focuses on the key frame of the knee bending of the user to be tested, and improves the fitness action category judgment accuracy rate of the user to be tested.

[0117] Based on the full connection layer in the deep learning model, the residual connection operation is performed on the action category judgment information, the gradient disappearance is avoided and the original information is prevented from being lost in the deep network, and the neural network regularization is performed based on LayerNorm + Dropout. It should be pointed out that the deep learning model in the embodiment of the present application uses a label smoothing cross-entropy loss function in training to avoid overfitting. The probability distribution of different action categories in the fitness action of the user to be tested is judged, and the action category probability distribution information is output. The action categories include push-ups, deep squats, barbell curls, hammer curls, shoulder presses and other fitness action categories.

[0118] The force line direction judgment module is configured to judge whether the force line direction of the fitness action of the user to be tested is correct based on the user local spatiotemporal feature information, and if not, record the force line direction abnormal information.

[0119] Specific execution modes of the force line direction judgment module include:

[0120] Obtain a fitness standard action database, the fitness standard action database including force line directions of standard actions of various fitness action categories.

[0121] Take the force line directions of the standard actions of the various fitness action categories in the fitness standard action database as training data, combine the user body digital twin model to train, and generate force line directions of the standard actions of the various fitness action categories of the to-be-tested user, and output the user customized action force line directions.

[0122] Based on the user local spatiotemporal feature information and the action category judgment information, the force line directions of each fitness action of different action categories of the to-be-tested user are identified, and the user action force line directions are obtained.

[0123] The user action force line directions are compared with the user customized action force line directions, and it is judged whether there is a difference in the force line directions. If there is a difference, it is judged that the force line directions of the to-be-tested user are abnormal, a force line direction abnormality signal is output, and force line direction abnormality information is recorded. The force line direction abnormality information includes force line direction abnormality position, force line direction abnormality timestamp, and force line direction abnormality degree.

[0124] The force line direction abnormality information is displayed in real time on the user body digital twin model.

[0125] The action counting module is configured to create a plurality of action state machines, and count the various fitness action categories of the to-be-tested user based on each action state machine to obtain the action movement quantity of each action category.

[0126] The specific execution mode of the action counting module includes:

[0127] Based on the action category judgment information and the action category probability distribution information, a corresponding action state machine is created for each fitness action of the to-be-tested user.

[0128] Obtain the historical fitness action video of the to-be-tested user, judge the actual action amplitude of the to-be-tested user based on the historical fitness action video of the to-be-tested user, and pre-set the action state threshold in the action state machine.

[0129] Based on each action state machine, the various fitness actions of the to-be-tested user in the key video frame information are counted respectively to obtain the action movement quantity of each action category.

[0130] Now taking the action state machine process corresponding to the "hammer curl" as an example for illustration:

[0131] The action state threshold includes an elbow joint angular velocity threshold, an elbow joint angle minimum threshold, and an elbow joint angle maximum threshold.

[0132] The initial state of the to-be-tested user is that the arm naturally droops, and the elbow joint angle is large, which is the elbow joint angle maximum threshold.

[0133] When the elbow joint angle starts to decrease, the elbow joint angular velocity is detected and compared with the elbow joint angular velocity threshold value, if the elbow joint angular velocity exceeds the elbow joint angular velocity threshold value, the "elbow joint bending in" state is entered.

[0134] The elbow joint angle is detected and compared with the minimum elbow joint angle threshold value, when the elbow joint angle decreases to the preset minimum elbow joint angle threshold value (such as 60 degrees) and tends to be stable, the "bending complete" state is entered.

[0135] When the elbow joint angle starts to increase, the "stretching in" state is entered.

[0136] When the elbow joint angle returns to the vicinity of the initial maximum threshold value (such as 170 degrees), a complete action cycle is completed, and the count of "hammer bending" action is increased by 1, that is, the number of "hammer bending" action movements is increased by 1.

[0137] It should be pointed out that the state in the embodiment of the application requires to be maintained for several frames before being allowed to switch, so as to prevent misjudgment caused by jitter.

[0138] The heat calculation module is configured to obtain the exercise energy consumption of the user to be tested based on the action category judgment information, the action completion time and the additional weight, and obtain the total energy consumption of the user to be tested based on the exercise energy consumption of the user to be tested and the basal energy consumption.

[0139] The specific execution mode of the heat calculation module includes:

[0140] An MET value database is obtained, and the MET value database includes the MET values of various fitness actions (such as push-ups MET = 8.0).

[0141] The completion time of each fitness action of the user to be tested is counted to obtain a plurality of sub-action completion times.

[0142] The additional weight (i.e. the weight of the instrument for strength training) of each fitness action of the user to be tested is counted to obtain additional weight information.

[0143] Based on the action category judgment information, the action category probability distribution information, the MET values of various fitness actions, the sub-action completion times and the additional weight information, the energy consumption of each fitness action of the user to be tested is judged to obtain the exercise energy consumption of each fitness action.

[0144] The exercise energy consumptions of the various fitness actions are superimposed to obtain the exercise energy consumption of the user to be tested.

[0145] The basal energy consumption of the user to be tested and the exercise energy consumption of the user to be tested are superimposed to obtain the total energy consumption of the user to be tested.

[0146] An interaction recording module is configured to display the user body digital twin model, the key video frame information, the action category judgment information, the action motion quantity and the total energy consumption in real time.

[0147] The specific execution mode of the interaction recording module includes:

[0148] An interactive interface is obtained, and the interactive interface includes a camera picture area and an information panel area.

[0149] The camera picture area is used for displaying the key video frame information in the center.

[0150] The human body key point skeleton and the joint node are superimposed on the key video frame information displayed in the center based on overlay, wherein the human body key point skeleton is marked by a green connecting line, the joint node is marked by a red dot, and the user body digital twin model is superimposed on the key video frame information.

[0151] When the force line direction abnormal signal is received, the force line direction abnormal information is displayed in the key video frame information in a flashing manner to prompt the user to be tested to adjust the bodybuilding action.

[0152] The information panel area is used for displaying the action category judgment information, the action category probability distribution information, the action motion quantity of each action category, the motion energy consumption of each bodybuilding action and the total energy consumption of the user to be tested in real time.

[0153] Those skilled in the art should understand that the embodiments of the present application can be provided as a method, a system. Therefore, the present application can adopt a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present application can adopt the form of a computer program product implemented on one or more computer usable storage media (including but not limited to disk memory, CD-ROM, optical memory, etc.) containing computer usable program codes.

[0154] The present application is described with reference to the flowcharts and structural diagrams of the method and system according to the embodiments of the present application. It should be understood that the combination of each flow and module in the flowcharts and structural diagrams can be realized by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processor or other programmable data processing devices to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing devices produce a means for implementing the functions specified in the flowcharts and structural diagrams. Figure 1 The means for implementing the functions specified in the flowcharts and structural diagrams. Figure 1 The means for implementing the functions specified in the flowcharts and structural diagrams.

[0155] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and structures Figure 1 The function specified in one or more modules.

[0156] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and structures Figure 1 The steps of a specified function in one or more modules.

[0157] Although preferred embodiments of the invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including both the preferred embodiments and all changes and modifications falling within the scope of the invention.

[0158] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.

Claims

1. A deep learning-based electronic data processing system, characterized in that, include: The basal metabolic analysis module is configured to acquire and determine the basal metabolic status of the user based on the user's basic information when the user is ready, thereby obtaining the user's basal energy consumption, and creating a digital twin model of the user's body based on the user's basic information. The feature extraction module is configured to obtain video information by capturing the fitness movements of the user under test using a high-definition camera, perform data processing operations and multi-dimensional feature extraction to obtain key video frame information, and adjust the user's digital twin model. The action category analysis module is configured to effectively capture the local spatiotemporal feature information of the user's fitness movements, analyze the complete cycle of the movement and output preliminary action judgment information, dynamically weight the output of action category judgment information, perform residual connection operation and regularization operation, and determine the probability distribution of different action categories in the fitness movement to obtain action category probability distribution information. The force line direction judgment module is configured to judge whether the force line direction of the user's fitness movement is correct based on the user's local spatiotemporal feature information. If it is incorrect, the abnormal force line direction information is recorded. The motion counting module is configured to create multiple motion state machines, and count the number of motion movements for each category of fitness movements of the user under test based on each motion state machine. The calorie calculation module is configured to obtain the exercise energy consumption of the user under test based on the action category judgment information, the duration of action completion and the additional load, and to obtain the total energy consumption of the user under test based on the exercise energy consumption and the basal energy consumption. The interaction recording module is configured to visualize the user's digital twin model of the body, key video frame information, action category judgment information, number of actions and total energy consumption in real time.

2. The deep learning-based electronic data processing system according to claim 1, characterized in that, The specific execution methods of the basic metabolic analysis module include: The basic information of the user to be tested is obtained and cached. The basic information of the user to be tested includes age, gender, weight, height, body shape dimensions, and muscle mass. Determine whether the environment and state of the user under test are suitable. If suitable, output a user ready signal. If not suitable, adjust the environment and state of the user under test until they are suitable for detection, and then output a user ready signal. Upon receiving the user ready signal, based on the user's age, gender, weight, and height information, the basal metabolic rate of the user is initially detected to obtain preliminary user basal metabolic information. Based on the muscle information of the test user, the impact of the test user's muscle mass on the test user's basal metabolism is determined to obtain the first basal metabolic compensation value. The body surface area of ​​the test user is determined based on the body shape dimension information of the test user, and the impact of the body surface area on the basal metabolism of the test user is determined based on the body surface area of ​​the test user to obtain the second basal metabolic compensation value. The first basal metabolic compensation value and the second basal metabolic compensation value are added together to obtain the basal metabolic compensation value of the user to be tested. The basal metabolic information of the user under test is adjusted based on the basal metabolic compensation value of the user under test to obtain the basal energy consumption of the user under test. The muscle information of the user under test includes muscle mass information and muscle distribution information. Based on the user's weight information, height information, body shape dimension information, muscle mass information, and muscle distribution information, a digital twin model of the user's body is created.

3. The deep learning-based electronic data processing system according to claim 2, characterized in that, The specific execution method of the feature extraction module includes: A high-definition camera is acquired and installed, and the user's fitness movements are captured in real time using the high-definition camera to obtain video information. Perform data preprocessing operations on the captured video information, namely image scaling and RGB conversion, to obtain preprocessed video data information; Based on data augmentation strategies, temporal and spatial augmentation processes are performed on preprocessed video data to obtain video data augmentation information. The Savitzky-Golay filter is used to smooth and reduce noise in the video data enhancement information to obtain smooth video data information. Acquire a video feature extractor and establish a signal connection link between the video feature extractor and the high-definition camera; Pose detection is performed based on MediaPipe in the video feature extractor, and multi-dimensional feature extraction is performed on key features in the captured video information to obtain multi-dimensional key feature information of pose. Display multi-dimensional key features of posture on the user's digital twin model of body; Based on the multi-dimensional key feature information of the posture, it is determined whether there is missing frame key point data. If there is missing frame key point data, linear interpolation compensation processing is performed to obtain key point interpolation information. The video data smoothing information and the key point interpolation information are combined to form key video frame information; Based on the key video frame information, the user's digital twin model is adjusted, improved, and updated.

4. The deep learning-based electronic data processing system according to claim 3, characterized in that, The specific execution method for performing pose detection based on MediaPipe in the video feature extractor, and extracting multi-dimensional features from key features in the captured video information to obtain multi-dimensional key pose feature information includes: Based on MediaPipe, pose detection is performed on the captured video information to identify the hip center and shoulder width of the user under test. The coordinates are normalized with the hip center of the user under test as the origin and standardized with the shoulder width of the user under test as the scale factor to obtain normalized coordinate feature information. Based on MediaPipe, pose detection is performed on the captured video information to identify the angles between different parts of the user's body and obtain the body part angle feature information. Based on MediaPipe, pose detection is performed on the captured video information to identify the distances between different parts of the user's body and obtain body part distance feature information. Based on MediaPipe, pose detection is performed on the captured video information to identify the velocity and acceleration of each joint of the user's body, thereby obtaining motion feature information. Based on MediaPipe, pose detection is performed on the captured video information to identify the symmetry of the left and right limbs of the user under test and obtain symmetry difference feature information. The normalized coordinate feature information, the body part angle feature information, the body part distance feature information, the motion feature information, and the symmetry difference feature information are combined to form multi-dimensional key feature information of posture.

5. The deep learning-based electronic data processing system according to claim 4, characterized in that, The specific execution methods of the action category analysis module include: Obtain a deep learning model whose architecture combines a convolutional neural network and a bidirectional long short-term memory network; The convolutional neural network in the deep learning model effectively captures and extracts the local patterns of the fitness movements of the user under test in the key video frame information, and outputs the user's local spatiotemporal feature information. The user's local spatiotemporal feature information is input into the bidirectional long short-term memory network. The bidirectional long short-term memory network learns the forward and backward dependencies of each fitness movement sequence in the user's local spatiotemporal feature information, understands and judges the complete cycle of each fitness movement of the user under test, and obtains preliminary movement judgment information. An attention mechanism is introduced, which dynamically weights the features of different time steps based on the initial judgment information of the action, highlights the importance of the key frames of the fitness action of the user under test, and outputs the action category judgment information. Based on the fully connected layer in the deep learning model, residual connection operation is performed on the action category judgment information, and neural network regularization is performed based on LayerNorm + Dropout to determine the probability distribution of different action categories in the fitness movements of the user under test, and output the action category probability distribution information.

6. The deep learning-based electronic data processing system according to claim 5, characterized in that, The specific execution method of the force line direction determination module includes: Obtain a database of standard fitness movements, which includes the force line direction of standard movements for each category of fitness movements; The standard movement force line direction of each fitness movement category in the fitness standard movement database is used as training data. Combined with the user's body digital twin model, the standard movement force line direction of each fitness movement category of the user to be tested is generated, and the user-customized movement force line direction is output. Based on the user's local spatiotemporal feature information and action category judgment information, the force line direction of the user performing different fitness movements of different action categories is identified to obtain the force line direction of the user's movements. The force line of the user's action is compared with the force line of the user's customized action to determine whether there is a difference in the force line. If there is a difference, the force line of the user under test is determined to be abnormal, a force line abnormality signal is output, and the force line abnormality information is recorded. The force line abnormality information includes the abnormal force line location, the abnormal force line timestamp, and the degree of abnormal force line. The abnormal force line orientation information is displayed in real time on the user's digital twin model of the body.

7. The deep learning-based electronic data processing system according to claim 6, characterized in that, The specific execution methods of the action counting module include: Based on the action category judgment information and the action category probability distribution information, a corresponding action state machine is created for each fitness action of the user to be tested. The system acquires historical fitness video of the user to be tested, determines the actual range of motion of the user based on the historical fitness video, and pre-sets the motion state threshold in the motion state machine. Based on each action state machine, the fitness actions of the user under test in the key video frame information are counted to obtain the number of actions in each action category.

8. The deep learning-based electronic data processing system according to claim 7, characterized in that, The specific execution methods of the heat calculation and analysis module include: Obtain a MET value database, which includes the MET values ​​for each fitness exercise; The completion time of each fitness movement performed by the user under test was statistically analyzed to obtain the completion time of multiple sub-movements; The extra load carried by the user in each fitness exercise is recorded to obtain extra load information; Based on the action category judgment information, action category probability distribution information, MET value of each fitness action, completion time of each sub-action, and additional load information, the energy consumption of each fitness action is obtained by judging the energy consumed by the user in each fitness exercise. The energy expenditure of the user under test is calculated by summing up the energy expenditure of each fitness exercise. The total energy consumption of the user is calculated by superimposing the user's basal energy consumption and the user's exercise energy consumption.

9. The deep learning-based electronic data processing system according to claim 8, characterized in that, The specific execution methods of the interaction recording module include: A human-computer interaction interface is obtained, the human-computer interaction interface including a camera screen area and an information panel area; The camera's view area is used to center-display the key video frame information; Based on the overlay, the human body key point skeleton and joint points are superimposed on the key video frame information displayed in the center. The human body key point skeleton is marked with green connecting lines and the human body joint points are marked with red dots. Simultaneously, the user's digital twin model of the body is superimposed on the key video frame information. When an abnormal force line direction signal is received, the abnormal force line direction information will be flashed in the key video frame information; The information panel area is used to provide real-time visualization of action category judgment information, action category probability distribution information, the number of movements in each action category, the energy consumption of each fitness movement, and the total energy consumption of the user being tested.

Images