Computer vision-based motion data processing method, device, equipment and medium

Abstract

The present invention relates to the field of artificial intelligence, and provides a motion data processing method, device, equipment and medium based on computer vision; wherein the method comprises: obtaining a first video, wherein the first video comprises the behavior movements of a human body during exercise training; generating a plurality of consecutive first images according to the first video, and extracting features of the plurality of consecutive first images respectively to obtain a first feature sequence; establishing a first motion model according to the first feature sequence; comparing the first motion model with a preset second motion model to identify the motion movements included in the first motion model; the second motion model includes at least one motion movement; determining the motion recognition rate of the first video according to the identified motion movement; the motion data processing method based on computer vision directly extracts and describes features of the video, so as to more intuitively identify the behavior movements of the human body and improve the accuracy of motion recognition.

Description

Technical Field

[0001] The embodiments of the present invention relate to, but are not limited to, the fields of artificial intelligence and computer vision, and in particular to a motion data processing method, device, equipment and medium based on computer vision. Background Art

[0002] There are already many online fitness courses on the market. Users can train independently through these fitness video courses, identify human movements through computer vision, and then provide training guidance. It can be seen that the effect of online fitness is very dependent on the accurate recognition of human movements.

[0003] Human motion recognition is a very important field in the field of machine vision and artificial intelligence. Its purpose is to detect and identify the target's movements from the video. Video-based human motion recognition consists of two parts: human motion area detection and motion recognition. Human motion area detection uses the motion target detection method to detect the area where the human body moves from the video. Motion recognition refers to feature extraction in the human motion area and then processing the features for specific business scenarios.

[0004] In the prior art, when human motion is recognized, it is often necessary to accurately extract and track human joints, and to perform some preprocessing such as target detection, target tracking, filtering and denoising. However, due to factors such as self-occlusion, scale and posture changes, it is more difficult to solve this problem. Therefore, the prior art has great limitations. Summary of the invention

[0005] The following is a summary of the subject matter described in detail herein. This summary is not intended to limit the scope of the claims.

[0006] The embodiments of the present invention provide a motion data processing method, device, equipment and medium based on computer vision, which can more intuitively identify human behavior and actions by directly extracting and describing features from videos, thereby improving the accuracy of action recognition.

[0007] In a first aspect, an embodiment of the present invention provides a motion data processing method based on computer vision, comprising:

[0008] Acquire a first video, wherein the first video includes a human body performing sports training;

[0009] Generate a plurality of consecutive frames of first images according to the first video, and extract features from the plurality of consecutive frames of first images to obtain a first feature sequence; wherein the first feature sequence includes a plurality of feature descriptors, and the feature descriptors correspond to the first images one by one;

[0010] Establishing a first motion model according to the first feature sequence;

[0011] Comparing the first motion model with a preset second motion model to identify motion actions included in the first motion model; wherein the second motion model includes at least one motion action;

[0012] The motion recognition rate of the first video is determined according to the recognized motion action.

[0013] In some embodiments, extracting features from the first images of the plurality of consecutive frames to obtain a first feature sequence includes:

[0014] Calculating a first gradient field and a first optical flow field of the first image;

[0015] Multiplying the intensity of the first gradient field by the first optical flow field to obtain a second optical flow field;

[0016] Calculating a directional gradient histogram of the second optical flow field as a feature descriptor of the first image;

[0017] The feature descriptors of all the first images are arranged in time sequence to form a first feature sequence.

[0018] In some embodiments, establishing a first motion model according to the first feature sequence includes:

[0019] Establishing a motion model of the first video to map the first feature sequence from a non-Euclidean space to a reproducing kernel Hilbert space;

[0020] The model parameters of the motion model are determined according to the first feature sequence to obtain a first motion model.

[0021] In some embodiments, the motion model of the first feature sequence is:

[0022]

[0023] Wherein, A is the state transfer matrix, B represents the influence of input noise on state change, C is a linear operator used to convert the change state into the output of the system, and x t represents the dynamic part of the first video, the y t is the apparent feature of the first video, the v t and w t are the model noise and observation noise at time t, respectively.

[0024] In some embodiments, determining the model parameters of the motion model according to the first feature sequence to obtain the first motion model includes:

[0025] Calculating a kernel matrix of the first feature sequence to obtain a first matrix;

[0026] Performing singular value decomposition on the first matrix to obtain a plurality of eigenvectors, and obtaining a kernel principal component corresponding to each of the eigenvectors;

[0027] The model parameters of the motion model are determined according to the first matrix and the kernel principal components corresponding to each eigenvector to obtain a first motion model.

[0028] In some embodiments, comparing the first motion model with a preset second motion model to identify the motion action included in the first motion model includes:

[0029] Calculate the metric distance between the first motion model and a preset second motion model:

[0030] It is determined whether the measured distance is within a set threshold range, and if so, the motion action of the second motion model is used as the motion action of the first motion model.

[0031] In some embodiments, the calculation formula of the metric distance is:

[0032]

[0033] or:

[0034]

[0035] Where y1 is the spatial metric of the first motion model, y2 is the spatial metric of the second motion model, and y 1；j is the jth feature descriptor in the first motion model, y 2；j is the j-th feature descriptor in the second motion model.

[0036] In a second aspect, an embodiment of the present invention further provides a motion data processing device based on computer vision, comprising:

[0037] An acquisition module, configured to acquire a first video, wherein the first video includes a behavior of a human body performing exercise training;

[0038] a feature extraction module, configured to generate a plurality of consecutive frames of first images according to the first video, and extract features from the plurality of consecutive frames of first images to obtain a first feature sequence; wherein the first feature sequence includes a plurality of feature descriptors, and the feature descriptors correspond to the first images one by one;

[0039] A model building module, used for building a first motion model according to the first feature sequence;

[0040] a motion recognition module, configured to compare the first motion model with a preset second motion model to recognize the motion included in the first motion model; wherein the second motion model includes at least one motion;

[0041] The output module is used to determine the motion recognition rate of the first video according to the recognized motion action.

[0042] In a third aspect, an embodiment of the present invention further provides an electronic device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the motion data processing method based on computer vision as described in the first aspect is implemented.

[0043] In a fourth aspect, an embodiment of the present invention further provides a storage medium, wherein executable instructions are stored in the storage medium, and when the executable instructions are executed by a processor, the motion data processing method based on computer vision as described in the first aspect is implemented.

[0044] The embodiment of the present invention includes: obtaining a first video, the first video includes the behavior of a human body in sports training; generating a plurality of consecutive frames of first images according to the first video, extracting features from the plurality of consecutive frames of first images respectively, and obtaining a first feature sequence; wherein the first feature sequence includes a plurality of feature descriptors, and the feature descriptors correspond to the first images one by one; establishing a first motion model according to the first feature sequence; comparing the first motion model with a pre-set second motion model to identify the motion included in the first motion model; and determining the motion recognition rate of the first video according to the identified motion. The motion data processing method based on computer vision directly extracts and describes features from the video, more intuitively identifies the behavior of the human body, and improves the accuracy of motion recognition.

[0045] Other features and advantages of the present invention will be described in the following description, and partly become apparent from the description, or understood by practicing the present invention. The purpose and other advantages of the present invention can be realized and obtained by the structures particularly pointed out in the description, claims and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] The accompanying drawings are used to provide a further understanding of the technical solution of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the technical solution of the present invention and do not constitute a limitation on the technical solution of the present invention.

[0047] Figure 1 is a flow chart of a motion data processing method based on computer vision according to an embodiment of the present invention;

[0048] Figure 2 yes Figure 1 Specific flow chart of step S200;

[0049] Figure 3 yes Figure 1 Specific flow chart of step S300;

[0050] Figure 4 yes Figure 1 Specific flow chart of step S320;

[0051] Figure 5 yes Figure 1 Specific flow chart of step S400;

[0052] Figure 6 is a structural diagram of a motion data processing device based on computer vision in an embodiment of the present invention;

[0053] Figure 7 It is a structural diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0054] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0055] It should be noted that, although the functional modules are divided in the device schematic diagram and the logical order is shown in the flow chart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flow chart. The terms "first", "second", etc. in the specification, claims and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. In the description of the present invention, the meaning of "several" is one or more, the meaning of "many" is more than two, greater than, less than, exceed, etc. are understood to exclude the number itself, and above, below, within, etc. are understood to include the number itself.

[0056] The present invention provides a motion data processing method, device, equipment and medium based on computer vision. The method includes: obtaining a first video, wherein the first video includes the behavior of a human body during exercise training; generating a first image of multiple consecutive frames according to the first video, and extracting features from the first images of multiple consecutive frames to obtain a first feature sequence; wherein the first feature sequence includes multiple feature descriptors, and the feature descriptors correspond to the first images one by one; establishing a first motion model according to the first feature sequence; comparing the first motion model with a pre-set second motion model to identify the motion contained in the first motion model; wherein the second motion model includes at least one motion; and determining the motion recognition rate of the first video according to the identified motion. The motion data processing method based on computer vision directly extracts and describes features from the video, more intuitively identifies the behavior of the human body, and improves the accuracy of motion recognition.

[0057] The embodiments of the present invention are further described below in conjunction with the accompanying drawings.

[0058] Reference Figure 1 , Figure 1 The invention is a flowchart of a motion data processing method based on computer vision.

[0059] like Figure 1 As shown, a motion data processing method based on computer vision includes:

[0060] Step S100, obtaining a first video, wherein the first video includes a human body performing sports training;

[0061] For step S100, the first video obtained is a video of a time period during the human body's exercise training. Generally, considering the need for subsequent image processing, the time period of the first video should be long enough, and specifically, at least one action is included in a time period. In some embodiments, the action includes at least one of the following: bending, jumping forward, jumping up, running, sideways running, running on one foot, walking slowly, waving with one arm, and waving with both arms.

[0062] Step S200: generating a plurality of consecutive first images according to the first video, and extracting features from the plurality of consecutive first images to obtain a first feature sequence; wherein the first feature sequence includes a plurality of feature descriptors, and the feature descriptors correspond to the first images one by one;

[0063] In some embodiments, the acquired first video is sampled at a set sampling frequency to obtain a first image of multiple consecutive frames. It should be considered that in order to ensure that the subsequently extracted motion actions are not lost, the set sampling frequency should not be too small; and in order to increase the learning speed of the model, the set sampling frequency should not be too large; in a preferred embodiment, the set sampling frequency is 20 frames / second to 30 frames / second. It should also be noted that the feature descriptor is some data extracted from the first image to represent the behavioral action.

[0064] Reference Figure 2 , for step S200, the specific steps are as follows:

[0065] Step S210, calculating a first gradient field and a first optical flow field of the first image;

[0066] It should be noted that, in the process of the camera capturing the first video, when the camera captures the movement of the human body in the camera area, or the camera area of ​​the camera changes, each pixel in the first video has a velocity vector, forming a motion field of image change. Generally, the optical flow field is consistent with the motion field. The image change within the time interval can be expressed by the optical flow field, and the optical flow field can be used to estimate the motion. The optical flow field is a velocity field that represents the three-dimensional motion of an object through a two-dimensional image, and the optical flow direction histogram describes the motion characteristics of the image. The recognition of human behavior, that is, the understanding and description of human behavior, refers to the analysis and recognition of human motion patterns. The motion information of the human body plays an irreplaceable role in the process of behavior recognition. The calculation of optical flow is intended to extract the motion information of the target human body. The characteristics of optical flow are used to characterize the motion contour of the target human body at each moment, and this feature has the characteristics of scale and direction invariance.

[0067] Step S220: multiplying the intensity of the first gradient field by the first optical flow field to obtain a second optical flow field;

[0068] Step S230, calculating a directional gradient histogram of the second optical flow field as a feature descriptor of the first image;

[0069] In some embodiments, for step S230, the specific steps are as follows:

[0070] Step S231, determining key points of the second optical flow field;

[0071] It should be noted that key points are some very prominent points that will not disappear due to factors such as lighting, scale, rotation, etc., such as corner points, edge points, bright spots in dark areas, and dark spots in bright areas. This step is to search for image positions in all scale spaces. In some embodiments, potential scale- and rotation-invariant points of interest are identified by using Gaussian differential functions.

[0072] Step S232, determining the radius of the image area where the key point is located;

[0073]

[0074] Among them, r is the radius of the image area, σ is the scale of the group where the key point is located, and d=4 represents the division into 4×4 sub-blocks.

[0075] It should be noted that for each key point, there are three pieces of information: position, scale, and direction. A descriptor is created for each key point, and a set of vectors is used to describe the key point so that it does not change with various changes, such as changes in lighting, changes in viewing angle, etc. This descriptor includes not only the key point, but also the pixels around the key point that contribute to it, and the descriptor should have a high degree of uniqueness to increase the probability of correct matching of feature points.

[0076] Step S233, dividing the pixels in the neighborhood of the key point into multiple sub-domains, dividing each sub-domain into multiple blocks, respectively calculating the gradient direction histograms of 8 directions in each block, and obtaining a descriptor vector of each block;

[0077] In some embodiments, the pixels in the neighborhood of the key point are divided into 16×16 sub-domains, each sub-domain is divided into 4×4 blocks, and the gradient direction histograms of 8 directions in each block are calculated respectively to obtain 4×4×8=128 descriptor vectors.

[0078] Step S234, normalizing each of the descriptor vectors to obtain a descriptor vector of the key point;

[0079] In one embodiment, the ratio of each of the descriptor vectors to all of the descriptor vectors is used as the normalized descriptor vector.

[0080] Step S235: Use the descriptor vectors of all key points in the first image as feature descriptors of the first image.

[0081] It is worth noting that the normalization of the descriptor vector of the key point can remove the lighting effect in the first image. The overall drift of the grayscale value in the first image can also be removed because the gradient of each point in the first image is obtained by subtracting the neighboring pixels.

[0082] Step S240: Form a first feature sequence by chronologically arranging the feature descriptors of all the first images.

[0083] It should be noted that the optical flow field is sensitive to noise, and the optical flow field existing in the background will complicate the recognition and lead to inaccurate classification.

[0084] In the embodiment provided by the present invention, in order to overcome the disadvantages of using a single first optical flow field, in this embodiment, the first gradient field and the first optical flow field are fused to generate a second optical flow field; then the second optical flow field is made into a histogram according to the method of the directional gradient histogram, that is, the density distribution of the second optical flow field is used to replace the second optical flow field as the feature descriptor of the first image. Among them, the directional gradient histogram (Histogram of Oriented Gradient, HOG) feature is a feature descriptor used for object detection in computer vision and image processing. The directional gradient histogram can be used to determine the structure and local information of the second optical flow field. Since the gradient field usually has the largest gradient intensity at its edge and contour, that is, the gradient value of the human body contour part is the largest. Therefore, after the weighting of this weight coefficient is obtained, the intensity of the optical flow field along the contour part of the human body shape is correspondingly enhanced. Then this feature descriptor can well describe the behavioral action characteristics of the human body. The feature descriptor in this embodiment contains both the motion information in the first optical flow field and the contour information of the human body posture. Using the descriptor vector of key points as the feature descriptor of the first image can accurately and comprehensively describe the motion characteristics of the human body, laying a good foundation for subsequent dynamic system modeling.

[0085] In this embodiment, the first feature sequence provides a normalized directional gradient histogram (feature descriptor) of the first video at multiple consecutive time points; in the embodiment provided by the present invention, in order to determine the changes in human body movements, it is necessary to compare the feature descriptors of two frames of the first image, and identify the human body's behavior movements by comparing the results of the feature descriptors.

[0086] Step S300: establishing a first motion model according to the first feature sequence;

[0087] Reference Figure 3 , for step S300, the specific steps are as follows:

[0088] Step S310: establishing a motion model of the first video to map the first feature sequence from a non-Euclidean space to a reproducing kernel Hilbert space;

[0089] The motion model includes a dynamic part and a noise part of the first video.

[0090] Step S320: Determine the model parameters of the motion model according to the first feature sequence to obtain a first motion model.

[0091] For step S310, the motion model of the first feature sequence is:

[0092]

[0093] Wherein, A is the state transfer matrix, B represents the influence of input noise on state change, C is a linear operator used to convert the change state into the output of the system, and x t represents the dynamic part of the first video, the y t is the apparent feature of the first video, the v t and w t are the model noise and observation noise at time t, respectively.

[0094] The dynamic part of the first video x t It is represented as a time evolution state process; x0 is the initialization condition of the system; the apparent feature y of the first video t is a linear function of the current state vector and the observation noise, x t ∈R n , x0∈R n ,y t ∈R m , w∈R m , A∈R n×n , n and m are both spatial dimensions.

[0095] In this embodiment, the motion model is used to transform the first characteristic sequence of the non-Euclidean space Transformed into a reproducing kernel Hilbert space eigensequence A motion model expressing the first eigensequence in a reproducing kernel Hilbert space;

[0096] It should be noted that when a video is given to train a motion model, the primary task is to learn the model parameters of the motion model from the feature trajectory (optical flow field) of the video. In the above embodiment, the model parameters of the motion model corresponding to the video can be expressed by an array M = (A, B, C, x t ) to indicate.

[0097] This embodiment assumes that the noise is a zero-mean independent and identically distributed normal random Gaussian process. The main purpose of the learning system is to calculate the parameters A, B and the change sequence x when the first matrix is ​​determined. t .

[0098] Reference Figure 4 , for step S320, the specific steps are as follows:

[0099] Step S321, calculating the kernel matrix of the first feature sequence to obtain a first matrix;

[0100] In this embodiment, the first characteristic sequence is y t =[y t；1 ,y t；2 ,...,y t；i,...,y t；n ] T ;

[0101]

[0102] Where i is the number of the feature descriptor in the first feature sequence; n is the total number of all feature descriptors in the first feature sequence; y t；i is the i-th feature descriptor in the first feature sequence at time t;

[0103] The calculation formula of the first matrix is:

[0104]

[0105] where e = [1,...,1] T , e∈R N , N is the spatial dimension;

[0106] K is the kernel matrix, k(y i ,y j )=φ(y i ) T φ(y j );

[0107] y i is the i-th feature descriptor, y j is the jth feature descriptor.

[0108] In this embodiment, a first matrix is ​​used to represent the kernel of zero-mean data (first characteristic sequence) in a high-dimensional space.

[0109] Step S322, performing singular value decomposition on the first matrix to obtain multiple eigenvectors, and obtaining a kernel principal component corresponding to each of the eigenvectors;

[0110] Perform singular value decomposition on the first matrix and get Denote the jth eigenvector in eigenvector V as v j ; According to the feature vector v j Get the jth kernel principal component Where α i,j represents the jth weight vector α j The i-th component of The weight vector is calculated according to the eigenvalue Sort in descending order.

[0111] Step S323: Determine the model parameters of the motion model according to the first matrix and the kernel principal components corresponding to each eigenvector to obtain a first motion model.

[0112] In this embodiment, after obtaining the kernel principal component and the first matrix, the implicit state X of the motion model is [x0, ..., x N-1 ] and the state transfer matrix A can be calculated:

[0113] Specifically, the following formula is first used for calculation:

[0114]

[0115] A=[x0,...,x N-1 ][x0,...,x N-2 ] T ;

[0116] Where X is the implicit state of the motion model; A is the state transfer matrix;

[0117] The model noise at time t is:

[0118] v t =x t -Ax t-1 ,

[0119] In the formula, v t is the model noise at time t, x t is the dynamic part at time t;

[0120] The covariance matrix of the noise at time t is:

[0121] In the formula, is the covariance matrix of the noise at time t;

[0122] Next, we use the square root method to decompose Then there is Thus, the influence of input noise on state change B is determined.

[0123] In the embodiment provided by the present invention, the first video is used as a motion model to reflect the behavior of the human body in the first video. The motion model divides the visual component and the internal dynamic part of the first video into two random processes. The dynamic part of the first video is represented as a time evolution state process x t ∈R n , and the apparent feature y of the video frame t ∈R m is a linear function of the current state vector and the observation noise.

[0124] Step S400: comparing the first motion model with a preset second motion model to identify motion actions included in the first motion model; wherein the second motion model includes at least one motion action;

[0125] Reference Figure 5 , for step S400, the specific steps are as follows:

[0126] Step S410: Calculate the metric distance between the first motion model and a preset second motion model:

[0127] Step S420: determine whether the measured distance is within a set threshold range, and if so, use the motion action of the second motion model as the motion action of the first motion model.

[0128] In the embodiment provided by the present invention, the construction of similarity measurement is focused on by mapping the first feature sequence (h∈H) to a high-dimensional (even infinite-dimensional) Hilbert space, where the mapping is: φ:H→F. F is a Hilbert space.

[0129] For points existing in the nonlinear manifold H, the first kernel function k(y1,y2)=φ(y1) is given T φ(y2), so we can calculate the kernel function on the original representation and then calculate the similarity on the reproduced kernel Hilbert space.

[0130] A distance sum between points in the reproducing kernel Hilbert space, which is the space generated by the first feature sequence. For two feature descriptors (such as histograms generated from linearly changing data), they can be directly calculated in the metric space; for the first motion model and the pre-set second motion model, the calculation formula of the metric distance is:

[0131]

[0132] or:

[0133]

[0134] Where y1 is the spatial metric of the first motion model, y2 is the spatial metric of the second motion model, and y 1；j is the jth feature descriptor in the first motion model, y 2；j is the j-th feature descriptor in the second motion model.

[0135] In another embodiment, the Binet-Cauchy maximum single-value kernel may be used to measure the metric distance between the motion model and the standard model.

[0136] It can be understood that after calculating the spatial distance between all feature descriptors in the first feature sequence and the corresponding feature descriptors in the second motion model, the metric distance between the first motion model and the second motion model can be expressed as a whole; if the metric distance is within the set threshold range, the motion action of the second motion model is taken as the motion action of the first motion model. It should be noted that the set threshold range reflects the requirements for motion actions and needs to be set according to actual conditions. For example, when the value of the threshold range is set to a smaller value, the requirements for motion actions are higher.

[0137] Step S500: determining a motion recognition rate of the first video according to the recognized motion action.

[0138] It should be noted that the mathematical model of the pre-set second motion model described in this embodiment is consistent with the first motion model, so that distance measurement can be performed in the same metric space; in some embodiments, the second motion model is established by the same method as the first motion model, and in another embodiment, the number of the first videos is not limited. For example, when one or more students are trained on the same or several motions, multiple first videos can be shot, and multiple first motion models can be established for the multiple first video packages, and then the multiple first motion models can be compared with the second motion model to identify the motion; in this way, multiple motions of a student or multiple students can be identified respectively; because the embodiment provided by the present invention does not need to perform some preprocessing such as target detection, target tracking, filtering and denoising on the acquired first video, the first video can be directly feature extracted and represented, and then a motion model can be established to perform metric distance measurement, which has a high accuracy. Accurate and targeted training feedback can be provided.

[0139] In some embodiments, the motion recognition rate of each acquired video is analyzed, and the recognition rate is set to a gradient value. For example, a recognition rate over 90% represents very good, 80%-90% represents accurate motion, and a rate below 80% represents irregular motion that needs to be improved. Based on this model, the user's entire set of sports movements is scored, and the scoring results are fed back to the user. After a comprehensive analysis of each motion recognition rate, a user training report is given, and irregular motions in the user's training are fed back, and reasonable suggestions are given. Based on each training result of the user and combined with the user's own situation, a training course that is more suitable for the user's current situation is provided.

[0140] The following is an experimental data of an embodiment of the present invention. In this experiment, a total of 9 behavioral actions were tested, each action was performed by 20 people, that is, each action had 20 videos. The statistical data of Table 1 below was obtained.

[0141] Table 1: Action recognition results

[0142]

[0143] In the embodiment provided by the present invention, the behavioral actions of users during training are analyzed and fed back based on the recognition of human body movement movements, which greatly improves the practical value of online fitness courses. Compared with traditional online fitness courses, by adopting the embodiment provided by the present invention, it has the effect of training in a close proximity to offline gyms, which is equivalent to the effect of training by offline fitness coaches. By scoring the training actions of users and providing analysis reports, users can know the effectiveness of their training each time and whether their actions are accurate. According to the analysis reports, they can adjust their irregular training actions, thereby achieving the best effect in training again and again.

[0144] It should be noted that the embodiments of the present invention can acquire and process relevant data based on artificial intelligence technology. Artificial Intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0145] AI basic technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics, etc. AI software technologies mainly include computer vision technology, robotics technology, biometrics technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0146] In addition, refer to Figure 6 One embodiment of the present invention further provides a motion data processing device based on computer vision, comprising:

[0147] An acquisition module 100 is used to acquire a first video, wherein the first video includes a human body performing sports training;

[0148] A feature extraction module 200 is used to generate a plurality of consecutive frames of first images according to the first video, and to extract features from the plurality of consecutive frames of first images to obtain a first feature sequence; wherein the first feature sequence includes a plurality of feature descriptors, and the feature descriptors correspond to the first images one by one;

[0149] A model building module 300, configured to build a first motion model according to the first feature sequence;

[0150] The motion recognition module 400 is used to compare the first motion model with a preset second motion model to identify the motion included in the first motion model; wherein the second motion model includes at least one motion;

[0151] The output module 500 is used to determine the motion recognition rate of the first video according to the recognized motion action.

[0152] In one embodiment, in the feature extraction module 200, the feature extraction is performed on the first images of the consecutive multiple frames respectively to obtain a first feature sequence, including:

[0153] Calculating a first gradient field and a first optical flow field of the first image;

[0154] Multiplying the intensity of the first gradient field by the first optical flow field to obtain a second optical flow field;

[0155] Calculating a directional gradient histogram of the second optical flow field as a feature descriptor of the first image;

[0156] The feature descriptors of all the first images are arranged in time sequence to form a first feature sequence.

[0157] In one embodiment, the model building module 300 is specifically used for:

[0158] Establishing a motion model of the first video to map the first feature sequence from a non-Euclidean space to a reproducing kernel Hilbert space;

[0159] The model parameters of the motion model are determined according to the first feature sequence to obtain a first motion model.

[0160] In one embodiment, determining the model parameters of the motion model according to the first feature sequence to obtain the first motion model includes:

[0161] Calculating a kernel matrix of the first feature sequence to obtain a first matrix;

[0162] Performing singular value decomposition on the first matrix to obtain a plurality of eigenvectors, and obtaining a kernel principal component corresponding to each of the eigenvectors;

[0163] The model parameters of the motion model are determined according to the first matrix and the kernel principal components corresponding to each eigenvector to obtain a first motion model.

[0164] In one embodiment, the action recognition module 400 is specifically used for:

[0165] Calculate the metric distance between the first motion model and a preset second motion model:

[0166] It is determined whether the measured distance is within a set threshold range, and if so, the motion action of the second motion model is used as the motion action of the first motion model.

[0167] In addition, refer to Figure 7 An embodiment of the present invention further provides an electronic device, the electronic device 10 comprising: a memory 11, a processor 12, and a computer program stored in the memory 11 and executable on the processor 12.

[0168] The processor 12 and the memory 11 may be connected via a bus or other means.

[0169] The non-transient software program and instructions required to implement the motion data processing method based on computer vision in the above embodiment are stored in the memory 11. When executed by the processor 12, the motion data processing method based on computer vision in the above embodiment is executed, for example, the above described Figure 1 Steps S100 to S500 of the method, Figure 2 Steps S210 to S240 of the method, Figure 3 Steps S310 to S320 of the method, Figure 4 Steps S321 to S323 of the method, Figure 5 The method comprises steps S410 to S420.

[0170] The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0171] In addition, an embodiment of the present invention further provides a computer-readable storage medium, which stores computer-executable instructions. The computer-executable instructions are executed by a processor or a controller, for example, by a processor, so that the processor can execute the motion data processing method based on computer vision in the above embodiment, for example, execute the above described Figure 1 Steps S100 to S500 of the method, Figure 2 Steps S210 to S240 of the method, Figure 3 Steps S310 to S320 of the method, Figure 4 Steps S321 to S323 of the method, Figure 5 The method comprises steps S410 to S420.

[0172] It should be noted that the embodiments provided by the present invention can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronic devices, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The embodiments provided by the present invention can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0173] It will be appreciated by those skilled in the art that all or some of the steps and systems in the disclosed method above may be implemented as software, firmware, hardware and appropriate combinations thereof. Some physical components or all physical components may be implemented as software executed by a processor, such as a central processing unit, a digital signal processor or a microprocessor, or may be implemented as hardware, or may be implemented as an integrated circuit, such as an application specific integrated circuit. Such software may be distributed on a computer-readable medium, which may include a computer storage medium (or a non-transitory medium) and a communication medium (or a temporary medium). As known to those skilled in the art, the term computer storage medium includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules or other data). Computer storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technologies, CD-ROM, digital versatile disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tapes, disk storage or other magnetic storage devices, or any other medium that may be used to store desired information and may be accessed by a computer. Furthermore, it is well known to those skilled in the art that communication media typically embodies computer readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transport mechanism, and may include any information delivery media.

[0174] The above is a specific description of the preferred implementation of the present invention, but the present invention is not limited to the above-mentioned implementation mode. Technical personnel familiar with the field can also make various equivalent deformations or substitutions without violating the spirit of the present invention. These equivalent deformations or substitutions are all included in the scope defined by the claims of the present invention.

Claims

1. A motion data processing method based on computer vision, characterized in that: include: Acquire a first video, wherein the first video includes a human body performing sports training; Generate a plurality of consecutive frames of first images according to the first video, and extract features from the plurality of consecutive frames of first images to obtain a first feature sequence; wherein the first feature sequence includes a plurality of feature descriptors, and the feature descriptors correspond to the first images one by one; Establishing a first motion model according to the first feature sequence; Calculating a metric distance between the first motion model and a preset second motion model; wherein the second motion model includes at least one motion action, and a mathematical model of the second motion model is consistent with the first motion model; determining whether the measured distance is within a set threshold range, and if so, taking the motion action of the second motion model as the motion action of the first motion model; Determining a motion recognition rate of the first video according to the recognized motion action; The step of establishing a first motion model according to the first feature sequence includes: Establishing a motion model of the first video to map the first feature sequence from a non-Euclidean space to a reproducing kernel Hilbert space; The model parameters of the motion model are determined according to the first feature sequence to obtain a first motion model. The motion model of the first feature sequence is: ; Among them, the is the state transfer matrix, represents the influence of input noise on state change, C is a linear operator used to convert the change state into the output of the system, represents the dynamic part of the first video, the is the apparent feature of the first video, and are the model noise and observation noise at time t, respectively.

2. The motion data processing method based on computer vision according to claim 1, characterized in that: The extracting features of the first images of the continuous multiple frames to obtain a first feature sequence comprises: Calculating a first gradient field and a first optical flow field of the first image; Multiplying the intensity of the first gradient field by the first optical flow field to obtain a second optical flow field; Calculating a directional gradient histogram of the second optical flow field as a feature descriptor of the first image; The feature descriptors of all the first images are arranged in time sequence to form a first feature sequence.

3. The motion data processing method based on computer vision according to claim 1, characterized in that: The step of determining the model parameters of the motion model according to the first feature sequence to obtain the first motion model includes: Calculating a kernel matrix of the first feature sequence to obtain a first matrix; Performing singular value decomposition on the first matrix to obtain a plurality of eigenvectors, and obtaining a kernel principal component corresponding to each of the eigenvectors; The model parameters of the motion model are determined according to the first matrix and the kernel principal components corresponding to each eigenvector to obtain a first motion model.

4. The motion data processing method based on computer vision according to claim 1, characterized in that: The calculation formula of the metric distance is: ； or: ； in, is the spatial metric of the first motion model, is the spatial metric of the second motion model, is the j-th feature descriptor in the first motion model, is the j-th feature descriptor in the second motion model.

5. A motion data processing device based on computer vision, characterized in that: include: An acquisition module, configured to acquire a first video, wherein the first video includes a behavior of a human body performing exercise training; a feature extraction module, configured to generate a plurality of consecutive frames of first images according to the first video, and extract features from the plurality of consecutive frames of first images to obtain a first feature sequence; wherein the first feature sequence includes a plurality of feature descriptors, and the feature descriptors correspond to the first images one by one; A model building module, used for building a first motion model according to the first feature sequence; a motion recognition module, configured to calculate a metric distance between the first motion model and a preset second motion model, determine whether the metric distance is within a preset threshold range, and if so, use the motion of the second motion model as the motion of the first motion model; wherein the second motion model includes at least one motion, and the mathematical model of the second motion model is consistent with the first motion model; An output module, configured to determine a motion recognition rate of the first video according to the recognized motion action; The step of establishing a first motion model according to the first feature sequence includes: Establishing a motion model of the first video to map the first feature sequence from a non-Euclidean space to a reproducing kernel Hilbert space; The model parameters of the motion model are determined according to the first feature sequence to obtain a first motion model. The motion model of the first feature sequence is: ; Among them, the is the state transfer matrix, represents the influence of input noise on state change, C is a linear operator used to convert the change state into the output of the system, represents the dynamic part of the first video, the is the apparent feature of the first video, and are the model noise and observation noise at time t, respectively.

6. An electronic device comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the computer vision-based motion data processing method as described in any one of claims 1 to 4 when executing the computer program.

7. A storage medium, characterized in that: The storage medium stores executable instructions, and when the executable instructions are executed by the processor, the motion data processing method based on computer vision as described in any one of claims 1 to 4 is implemented.

Images