Intelligent motion analysis system, method and device and computer readable storage medium

Abstract

The invention discloses an intelligent motion analysis system, method and device and a computer readable storage medium. The intelligent motion analysis system comprises a color visual sensor which is used for acquiring frame-by-frame images of a sportsman and forming a motion video sequence by the frame-by-frame images; the attitude estimation model is used for converting the motion video sequence into two attitude feature vectors and then converting the two attitude feature vectors into three-dimensional coordinate information of skeleton key points of the moving personnel; and the analysis module is used for obtaining joint angular velocity, limb consistency analysis and joint three-dimensional angles according to the three-dimensional coordinate information of the skeleton key points. The motion video sequence of the sportsman is collected through the color visual sensor, the sportsman does not need to wear a physical sensor, the influence of wearing the physical sensor on the actual performance of the sportsman is reduced, and the motion analysis of the sportsman is completely free of contact and interference.

Description

Technical Field

[0001] The present invention generally relates to the field of electronic information technology, and specifically, especially to an intelligent motion analysis system, method, device and computer-readable storage medium. Background Art

[0002] Traditional motion detection methods usually install physical sensors at specific bone locations on the human body to obtain indicators of interest, but the addition of such equipment will undoubtedly have a certain impact on the actual performance of athletes, thereby causing difficulties in subsequent evaluations. Summary of the invention

[0003] In view of the above-mentioned defects or deficiencies in the prior art, it is desirable to provide a contactless, non-interference intelligent motion analysis system, method, device and computer-readable storage medium.

[0004] In a first aspect, the intelligent motion analysis system of the present invention is characterized by comprising:

[0005] A color vision sensor, which is used to obtain frame-by-frame images of a moving person and compose the frame-by-frame images into a moving video sequence;

[0006] A posture estimation model, which first converts the motion video sequence into two posture feature vectors, and then converts the two posture feature vectors into three-dimensional coordinate information of the skeleton key points of the motion person;

[0007] An analysis module obtains joint angular velocity, limb consistency analysis and joint three-dimensional angle according to the three-dimensional coordinate information of the skeleton key points.

[0008] In a second aspect, the intelligent motion analysis method of the present invention is characterized by comprising the following steps:

[0009] S100, acquiring frame-by-frame images of a moving person, and composing the frame-by-frame images into a moving video sequence;

[0010] S200, first converting the motion video sequence into two-gesture feature vectors, and then converting the two-gesture feature vectors into three-dimensional coordinate information of skeleton key points of the athlete;

[0011] S300, obtaining joint speed, limb consistency analysis and joint three-dimensional angle according to the three-dimensional coordinate information of the skeleton key points.

[0012] In a third aspect, the intelligent motion analysis device of the present invention is characterized in that it includes a processor and a memory, the processor is configured to execute instructions stored in the memory, and the processor implements the intelligent motion analysis method by executing the instructions.

[0013] In a fourth aspect, the computer-readable storage medium of the present invention is characterized in that instructions are stored in the computer-readable storage medium, and when the computer-readable storage medium is run on a computer, the computer executes an intelligent motion analysis method.

[0014] According to the technical solution provided in the embodiment of the present application, the motion video sequence of the athlete is collected by a color vision sensor, and the athlete does not need to wear physical sensors, thereby reducing the impact of wearing physical sensors on the actual performance of the athlete. The present invention performs the motion analysis of the athlete completely non-contact and non-interference. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Other features, objects and advantages of the present application will become more apparent by reading the detailed description of non-limiting embodiments made with reference to the following drawings:

[0016] Figure 1 A schematic diagram of the structure of a two-dimensional human posture regression model of an intelligent motion analysis system according to an embodiment of the present invention;

[0017] Figure 2 A schematic diagram of the structure of a CBAM module of an intelligent motion analysis system according to an embodiment of the present invention;

[0018] Figure 3 A schematic diagram of the structure of a CBS module of an intelligent motion analysis system according to an embodiment of the present invention;

[0019] Figure 4 A schematic diagram of the structure of a first basic block of an intelligent motion analysis system according to an embodiment of the present invention;

[0020] Figure 5 A schematic diagram of the structure of a second basic block of an intelligent motion analysis system according to an embodiment of the present invention;

[0021] Figure 6 A schematic diagram of the structure of a three-dimensional human body posture skeleton key point regression model of an intelligent motion analysis system according to an embodiment of the present invention;

[0022] Figure 7 A schematic diagram of the structure of a first FNR module of an intelligent motion analysis system according to an embodiment of the present invention;

[0023] Figure 8 A schematic diagram of the structure of the key points of bones of an intelligent motion analysis system according to an embodiment of the present invention;

[0024] Fig. 9 A schematic diagram of a flow chart of an intelligent motion analysis method according to another embodiment of the present invention;

[0025] Fig.10A schematic diagram of the structure of an intelligent motion analysis device according to another embodiment of the present invention;

[0026] Fig.11 A schematic diagram of the structure of an intelligent motion analysis device according to another embodiment of the present invention;

[0027] Fig.12 The figure is a schematic diagram of the structure of a computer-readable storage medium according to another embodiment of the present invention. DETAILED DESCRIPTION

[0028] The present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are only used to explain the relevant invention, rather than to limit the invention. It is also necessary to explain that, for ease of description, only the parts related to the invention are shown in the accompanying drawings.

[0029] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0030] One embodiment of the present invention is, please refer to Figures 1 to 8 The intelligent motion analysis system of the present invention includes: a color visual sensor, which is used to obtain frame-by-frame images of a sports person and compose the frame-by-frame images into a motion video sequence; a posture estimation model, which first converts the motion video sequence into two posture feature vectors, and then converts the two posture feature vectors into three-dimensional coordinate information of the skeleton key points of the sports person; an analysis module, which obtains joint angular velocity, limb consistency analysis and joint three-dimensional angle according to the three-dimensional coordinate information of the skeleton key points.

[0031] In an embodiment of the present invention, the posture estimation model (DF-POSE) uses an end-to-end model training method. The video sequence is first input into a two-dimensional posture detector as input, and a two-dimensional posture feature vector is generated through a two-dimensional human posture detection method; the two-dimensional posture feature vector is input into a three-dimensional posture detector, and the three-dimensional coordinate information of the main skeleton key points of the human body is generated in combination with the SMPL model. The skeletal point coordinate information of the three-dimensional posture of the human body can be directly obtained from the video sequence image, and end-to-end estimation can be achieved, and real-time estimation can be performed. The two-dimensional human posture detection method generates a two-dimensional posture feature vector without the need to regress two-dimensional key points.

[0032] refer to Figure 8 The key points of the skeleton include eyes, nose, left and right shoulders, left and right elbows, left and right wrists, left and right hips, left and right knees, left and right ankles, the midpoint of the two shoulders, the midpoint of the two hips and the spine point, a total of 17 key points.

[0033] Furthermore, the posture estimation model includes a two-dimensional human posture regression model and a three-dimensional human posture skeleton key point regression model. The two-dimensional human posture regression model is based on the ResNet-50 network structure. The two-dimensional human posture regression model includes a CBAM module, a CBS module, a downsampling module and a first fully connected layer arranged in sequence. The CBAM module includes a parallel spatial attention branch, a channel attention branch, and a residual branch. The CBS module includes a series Conv layer, a first BN layer and a first ReLu layer. The three-dimensional human posture skeleton key point regression model includes a first FNR module, an average pooling module, a second FNR module and a second fully connected layer arranged in sequence. The first FNR module includes a random drop layer, a third fully connected layer, a second BN layer and a second RULu layer. The second FNR module has the same structure as the first FNR module.

[0034] Furthermore, the two-dimensional human posture regression model includes a first basic block and a second basic block. The downsampling module stacks the first basic block and the second basic block four times, stacks a first basic block and two second basic blocks once, stacks a first basic block and three second basic blocks twice, stacks a first basic block and five second basic blocks three times, and stacks a first basic block and two second basic blocks four times.

[0035] In an embodiment of the present invention, an improved CBAM module is first added to the ResNet-50 network structure to increase the degree of attention of the convolutional neural network to different feature channels and spatial positions, improve the model's ability to extract complex features, and improve the performance of the model; secondly, the basic blocks B1 and B2 in the original network are adjusted to accelerate the model. The backbone network will therefore be divided into four parts. The first part includes one B1 and two B2s; the second part will include one B1 and three B2s; the third part includes one B1 and five B2s; the fourth part includes one B1 and two B2s, and finally a two-dimensional posture vector for the human body will be obtained. Finally, the final three-dimensional human posture key point coordinates are obtained through the three-dimensional human posture detection model.

[0036] refer to Figure 4 and 5 ,Furthermore, the first basic block and the second basic block respectively include a depthwise separable convolution consisting of a 3x3 depthwise convolution and a 1x1 pointwise convolution.

[0037] In an embodiment of the present invention, the traditional 3x3 convolution blocks are uniformly replaced by depthwise separable convolutions. Depthwise separable convolutions are composed of a 3x3 depthwise convolution and a 1x1 pointwise convolution. Acceleration model.

[0038] Furthermore, according to the three-dimensional coordinate information of the key points of the skeleton, the three-dimensional angle of the joint is obtained, including:

[0039] Select any three connected bone key points P1 = (x1, y1, z1), P2 = (x2, y2, z2), P3 = (x3, y3, z3), then the joint three-dimensional angle θ is obtained by the following formula:

[0040]

[0041] Further, the first slope between P1 and P2 is obtained by the following formula,

[0042]

[0043] The second slope between P2 and P3 is obtained by the following formula,

[0044]

[0045] The second norm of the first slope and the second slope is obtained by the following formula:

[0046]

[0047] If the second norm of the first slope and the second slope is less than the threshold value 0.1, it indicates that P1, P2 and P3 are in a consistent state.

[0048] In the embodiment of the present invention, if the second norm of the first slope and the second slope is less than the threshold value 0.1, it indicates that P1, P2 and P3 are in a consistent state, that is, the three points P1, P2 and P3 are collinear.

[0049] If the second norm of the first slope and the second slope is greater than the threshold value 0.1, it indicates that P1, P2 and P3 are in an inconsistent state, that is, the three points P1, P2 and P3 are not collinear.

[0050] Furthermore, after obtaining the angle information of all frames in the video sequence, the angle line is fitted by the polynomial regression method, and the first-order angle curve is obtained by derivation. The joint angular velocity at a certain moment is obtained by the following formula:

[0051]

[0052] In an embodiment of the present invention, after obtaining the angle information of all frames of the video sequence, the angle line is fitted by a polynomial regression method:

[0053] θ(t)=a0+a1t+a2t 2 +...+a n t n .

[0054] And take the derivative to obtain the first-order angle curve, then the angular velocity at a certain moment can be expressed by the following formula:

[0055]

[0056] Afterwards, the burrs in the angular velocity are removed by using the improved Kalman filter algorithm, and the improved Jacobian matrix is ​​calculated. When the state is updated, the modified extended Kalman filter iteratively corrects the gain and state estimation value to make the result more accurate.

[0057] First, create the state vector:

[0058]

[0059] f is the nonlinear state transfer function, u k is the control input, based on the physical relationship between angle and angular velocity:

[0060] X K|K-1 =f(X k-1 ,u k ),

[0061] Then calculate the Jacobian matrix F of the state transfer function to the state variable k-1 :

[0062]

[0063] Then calculate the predicted state covariance, where Q k is the process noise covariance:

[0064]

[0065] Then calculate the Kalman gain:

[0066]

[0067] Among them, P k|k-1 is the predicted state covariance, H k is the observation matrix, R k is the observation noise covariance.

[0068] Then update the initial state:

[0069] x k|k =x k|k-1 +K k (z k -h(x k|k-1 ))

[0070] Among them, z k are the observed values, and h(x) is the nonlinear model.

[0071] After that, iterative correction is performed. First, the increment of the state is calculated:

[0072] Δxk =K k (z k -h(x k|k-1 )).

[0073] Afterwards, the state estimate is revised, and each iteration is based on the increment Δx k Update the state estimate:

[0074]

[0075] Where i represents the number of iterations.

[0076] After each iteration, the Kalman gain is recalculated, the state estimate is further adjusted, and the Kalman gain is updated:

[0077]

[0078] Until the change in state estimation is less than the preset threshold β = 10 -5 , or the maximum number of iterations is reached, the iteration is terminated.

[0079] After each iteration, update the state covariance P k|k-1 :

[0080]

[0081] After multiple iterations, the final state estimation This is the optimized angular velocity.

[0082] Another embodiment of the present invention is, referring to Fig. 9 An intelligent motion analysis method comprises the following steps: S100, acquiring frame-by-frame images of a sports person, and composing the frame-by-frame images into a motion video sequence; S200, first converting the motion video sequence into a two-posture feature vector, and then converting the two-posture feature vector into three-dimensional coordinate information of the skeleton key points of the sports person; S300, obtaining joint speed, limb consistency analysis and joint three-dimensional angle according to the three-dimensional coordinate information of the skeleton key points.

[0083] Another embodiment of the present invention is, referring to Fig.10 and 11 , an intelligent motion analysis device, comprising a processor and a memory, wherein the processor is configured to execute instructions stored in the memory, and the processor implements an intelligent motion analysis method by executing the instructions.

[0084] In an embodiment of the present invention, an intelligent motion analysis device is composed of a display screen 1 and an internally hollow metal cabinet 12, which are connected by a connecting rod 5; four pulleys 13 are installed under the metal cabinet to facilitate users to quickly transfer the equipment.

[0085] Preferably, the display screen 1 is characterized in that the display screen includes a color camera 2, a fill light 3 and a button 4 for controlling the display screen to be turned on and off. In addition, the power cord 8 of the display screen and the display screen adapter 9 are arranged downward through the space inside the connecting rod 5 to maintain a certain aesthetic and safety.

[0086] Preferably, the hollow metal cabinet 12 is characterized in that the metal cabinet has a freely openable metal door to facilitate access to the internal space. In addition, an operating keyboard 7 and a mouse 6 are placed on the upper surface of the metal cabinet. The keyboard connection cable 9 and the mouse connection cable 11 are arranged vertically downward along the metal cabinet.

[0087] Preferably, refer to Fig.10 and 11 The metal cabinet has a freely openable metal door to facilitate access to the internal space, and is characterized in that the metal cabinet door can be opened up to 180 degrees. A host 14 of a computer system is installed in the internal space of the cabinet, and the power cord 8 of the display screen, the display screen adapter cable 9, the keyboard connection cable 9 and the mouse connection cable 11 are all connected to the host 14 through the internal space.

[0088] Another embodiment of the present invention is, referring to Fig.12 , a computer-readable storage medium, wherein instructions are stored in the computer-readable storage medium, and when the computer-readable storage medium is run on a computer, the computer executes an intelligent motion analysis method.

[0089] The computer system includes a central processing unit (CPU) 501, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 502 or a program loaded from a storage portion into a random access memory (RAM) 503. In the RAM 503, various programs and data required for system operation are also stored. The CPU 501, the ROM 502, and the RAM 503 are connected to each other via a bus 504. An input / output (I / O) interface 505 is also connected to the bus 504.

[0090] The following components are connected to the I / O interface 505: an input section 506 including a keyboard, a mouse, etc.; an output section including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 508 including a hard disk, etc.; and a communication section 509 including a network interface card such as a LAN card, a modem, etc. The communication section 509 performs communication processing via a network such as the Internet. A drive is also connected to the I / O interface 505 as needed. A removable medium 511, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 510 as needed, so that a computer program read therefrom is installed into the storage section 508 as needed.

[0091] In particular, according to an embodiment of the present invention, the above reference process Fig. 9 The described process can be implemented as a computer software program. For example, an embodiment of the present invention includes a computer program product, which includes a computer program carried on a computer readable medium, and the computer program contains program code for executing the method shown in the flow chart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication part, and / or installed from a removable medium. When the computer program is executed by a central processing unit (CPU) 501, the above-mentioned functions defined in the system of the present application are executed.

[0092] It should be noted that the computer-readable medium shown in the present invention may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present invention, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In the present invention, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, optical cable, RF, etc., or any suitable combination of the above.

[0093] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present invention. In this regard, each box in the flow chart or block diagram can represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0094] The units involved in the embodiments of the present invention may be implemented by software or hardware, and the units described may also be arranged in a processor. The names of these units do not, in some cases, limit the units themselves. The units or modules described may also be arranged in a processor, for example, they may be described as: a processor including a color vision sensor, a posture estimation model, and an analysis module. The names of these units or modules may not, in some cases, limit the units or modules themselves, for example, a color vision sensor may also be described as "a color vision sensor for acquiring frame-by-frame images of a moving person and composing the frame-by-frame images into a motion video sequence".

[0095] As another aspect, the present application also provides a computer-readable medium, which may be included in the electronic device described in the above embodiment; or may exist independently without being assembled into the electronic device. The above computer-readable medium carries one or more programs, and when the above one or more programs are executed by an electronic device, the electronic device implements the intelligent motion analysis method described in the above embodiment.

[0096] For example, the electronic device may implement Fig. 9 As shown in: S100, acquiring frame-by-frame images of a sports person, and composing the frame-by-frame images into a sports video sequence; S200, first converting the sports video sequence into a two-posture feature vector, and then converting the two-posture feature vector into three-dimensional coordinate information of the skeleton key points of the sports person; S300, obtaining joint speed, limb consistency analysis and joint three-dimensional angle according to the three-dimensional coordinate information of the skeleton key points.

[0097] It should be noted that, although several modules or units of the device for action execution are mentioned in the above detailed description, this division is not mandatory. In fact, according to the embodiments of the present disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. On the contrary, the features and functions of one module or unit described above can be further divided into multiple modules or units to be embodied.

[0098] In addition, although the steps of the method in the present disclosure are described in a specific order in the drawings, this does not require or imply that the steps must be performed in this specific order, or that all the steps shown must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps, etc.

[0099] Through the description of the above implementations, those skilled in the art can easily understand that the example implementations described here can be implemented by software, or by combining software with necessary hardware.

[0100] The above description is only a preferred embodiment of the present application and an explanation of the technical principles used. Those skilled in the art should understand that the scope of the invention involved in the present application is not limited to the technical solution formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the inventive concept. For example, the above features are replaced with (but not limited to) technical features with similar functions disclosed in the present application.

Claims

1. An intelligent motion analysis system, characterized in that: include: A color vision sensor, which is used to obtain frame-by-frame images of a moving person and compose the frame-by-frame images into a moving video sequence; A posture estimation model, which first converts the motion video sequence into two posture feature vectors, and then converts the two posture feature vectors into three-dimensional coordinate information of the skeleton key points of the motion person; An analysis module obtains joint angular velocity, limb consistency analysis and joint three-dimensional angle according to the three-dimensional coordinate information of the skeleton key points.

2. The intelligent motion analysis system according to claim 1, characterized in that: The posture estimation model includes a two-dimensional human posture regression model and a three-dimensional human posture skeleton key point regression model. The two-dimensional human posture regression model is based on the ResNet-50 network structure. The two-dimensional human posture regression model includes a CBAM module, a CBS module, a downsampling module and a first fully connected layer arranged in sequence. The CBAM module includes a parallel spatial attention branch, a channel attention branch, and a residual branch. The CBS module includes a series-connected Conv layer, a first BN layer and a first ReLu layer. The three-dimensional human posture skeleton key point regression model includes a first FNR module, an average pooling module, a second FNR module and a second fully connected layer which are arranged in sequence. The first FNR module includes a random discard layer, a third fully connected layer, a second BN layer and a second RULu layer. The second FNR module has the same structure as the first FNR module.

3. The intelligent motion analysis system according to claim 2, characterized in that: The two-dimensional human posture regression model includes a first basic block and a second basic block, and the downsampling module stacks the first basic block and the second basic block four times. Stack one first basic block and two second basic blocks. One first basic block and three second basic blocks are stacked twice. One first basic block and five second basic blocks are stacked three times. One first basic block and two second basic blocks are stacked four times.

4. The intelligent motion analysis system according to claim 3, characterized in that: The first basic block and the second basic block respectively include a depthwise separable convolution consisting of a 3x3 depthwise convolution and a 1x1 pointwise convolution.

5. The intelligent motion analysis system according to claim 1, characterized in that: According to the three-dimensional coordinate information of the skeleton key points, the three-dimensional angle of the joint is obtained, including: Select any three connected bone key points P1 = (x1, y1, z1), P2 = (x2, y2, z2), P3 = (x3, y3, z3), then the joint three-dimensional angle θ is obtained by the following formula:

6. The intelligent motion analysis system according to claim 5, characterized in that: The first slope between P1 and P2 is obtained by the following formula, The second slope between P2 and P3 is obtained by the following formula, The second norm of the first slope and the second slope is obtained by the following formula: If the second norm of the first slope and the second slope is less than the threshold value 0.1, it indicates that P1, P2 and P3 are in a consistent state.

7. The intelligent motion analysis system according to claim 5, characterized in that: After obtaining the angle information of all frames in the video sequence, the angle line is fitted by the polynomial regression method, and the first-order angle curve is obtained by derivation. The joint angular velocity at a certain moment is obtained by the following formula:

8. An intelligent motion analysis method, characterized in that: The following steps are involved: S100, acquiring frame-by-frame images of a moving person, and composing the frame-by-frame images into a moving video sequence; S200, first converting the motion video sequence into two-gesture feature vectors, and then converting the two-gesture feature vectors into three-dimensional coordinate information of skeleton key points of the athlete; S300, obtaining joint speed, limb consistency analysis and joint three-dimensional angle according to the three-dimensional coordinate information of the skeleton key points.

9. An intelligent motion analysis device, characterized in that: The invention comprises a processor and a memory, wherein the processor is configured to execute instructions stored in the memory, and the processor implements the intelligent motion analysis method according to claim 8 by executing the instructions.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores instructions, and when the computer-readable storage medium is run on a computer, the computer is enabled to execute the intelligent motion analysis method according to claim 8 .

Images