Computer vision method for evaluating the performance of movements by a person

Abstract

The invention relates to evaluating the correctness of a person's performance of movements. An evaluation method includes creating a database of a reference performance of a test movement, with an atlas of reference values for joint angles in all the degrees of freedom of the joints in question. Images of the performance of the test movement are obtained, and a three-dimensional model is constructed of the test subject in a frame, said model including indications of reference points on a skeleton model of the test subject. The spatial position of said points is evaluated and the angular displacement of the points in relation to one another is used to determine the joint angle values, after which the obtained joint angle values are compared with the reference angle values. During performance of the test movement, personal values are obtained for the angle of each analyzed joint in each degree of freedom of the joint. For each analyzed joint, the variation between the reference angle values and the personal angle values is calculated. The mean variation is evaluated for each joint, and the evaluations are compared with threshold values. The mean value of all the measurements of correctness of the performance of the test movement is evaluated, and the evaluation is compared with threshold values for the test movement, thus providing an evaluation of the performance of the test movement. The invention provides greater accuracy in the evaluation of a person's performance of movements.

Description

[0001] METHOD OF ASSESSING THE PERFORMANCE OF HUMAN MOVEMENTS USING A MACHINE

[0002] VISION

[0003] DESCRIPTION

[0004] The invention relates to methods for assessing the correctness of the performance of certain human movements. The claimed method can be applied in various areas of human activity, such as medicine, sports, production, the service sector, etc. [A63B24 / 00, A61B5 / 00, G06T7 / 00],

[0005] A METHOD FOR MONITORING AND ANALYSING AN ATHLETE'S MOVEMENTS is known from the prior art [RU2361637, published 20.07.2009] consisting of filming the movements being studied with a video camera, transmitting the resulting video fragment to a computer, setting control points on the athlete's body or apparatus in the first frame, tracking their position in all subsequent frames, calculating the trajectories of the movement of the control points and mathematically analyzing them using a computer, characterized in that at least two video cameras are used during filming, which are connected to the computer via a synchronizing device and installed so that a possible obstacle appearing in front of one of the cameras and blocking a control point for it at a certain moment does not block this point for the other cameras at the same time, and when calculating the trajectories, frames are identified in the video fragments of all cameras where the control point is blocked by the obstacle, and these frames are excluded from further processing,and for the remaining frames, the coordinates of the control point are calculated, then for the same frames, the calculated coordinates are averaged over all video fragments.

[0006] The disadvantage of the analogue is its limited area of ​​use, namely the area of ​​sports.

[0007] Also known from the prior art is a METHOD FOR DIAGNOSIS OF DISORDERS IN THE MUSCULOSKELETAL SYSTEM [RU2780164, publ. 20.09.2022], characterized by the fact that initially in the initial data module a database is formed, including encyclopedic, scientific and practical knowledge about the etiology and pathogenesis of diseases of the musculoskeletal system, means and methods of their treatment, and a database of images of the standard performance of diagnostic exercises, myofascial and functional diagnostics of deviations of the musculoskeletal system with an atlas of reference values ​​​​of the degrees of freedom of the joints is formed, after which stages of collecting a person's life history, disease history, myofascial, functional and software diagnostics of the musculoskeletal system are carried out, with the formation of a recommended list of specialized kinesitherapeutic exercises for the correction and treatment of the identified deviations in the functionality of the patient's musculoskeletal system,wherein at the stage of software diagnostics of the state of the musculoskeletal system, images of the patient performing diagnostic exercises are obtained, according to which, using machine vision technology, a three-dimensional model of the patient is constructed in the frame, including designations of the location of key joints on the image of the patient's musculoskeletal system in the form of reference points on his skeletal model, while an assessment is made of the spatial position of the said points and, based on their angular movements relative to each other, the degrees of freedom of the patient's joints are determined, after which the obtained values ​​of the degrees of freedom of the joints are compared with the values ​​of the degrees of freedom of the joints specified in the atlas of reference values ​​of the degrees of freedom of the joints,and based on the obtained deviations, on the basis of the created database, a recommended list of specialized kinesitherapeutic exercises is formed to correct and treat the identified deviations in the functionality of the patient's musculoskeletal system.

[0008] The disadvantage of the analogue is its limited scope of use, namely, for diagnosing disorders in the musculoskeletal system. Another disadvantage of the analogue is the low accuracy of diagnosis, due to the fact that statistical processing of the obtained values ​​of the degrees of freedom of the joints is not implemented.

[0009] The closest in its technical essence is the METHOD OF ANALYZING ANGULAR DISPLACEMENTS OF REFERENCE POINTS OF A SKELETAL MODEL OF THE HUMAN MUSCULOSKELETAL SYSTEM [RU2786306, published 12 / 19 / 2022], characterized in that for its implementation, a video image stream is continuously generated, on which a person is recognized in the frame, a skeletal model of the human musculoskeletal system is constructed based on the recognized image of a person in the video stream, a set of reference points is formed on the skeletal model, with a continuous assessment of the angular displacements of the said reference points of the said 3d model, characterized in that in order to assess the movement of the reference points, local coordinate systems are constructed in each of them, while the said coordinate systems are constructed on the basis of the Gram-Schmidt algorithm, according to which each coordinate system depends on the position of the parent bone of the human skeletal model, taking into account its spatial rotation.The main technical problem of the prototype is the low accuracy of the application of the specified method for assessing the correctness of human movement, due to the fact that the prototype does not provide for statistical processing of the results obtained during the implementation of the method.

[0010] The objective of the invention is to eliminate the shortcomings of the prototype and analogues.

[0011] The technical result of the invention consists in increasing the accuracy of assessing the performance of human movements.

[0012] The specified technical result is achieved due to the fact that the method for assessing the performance of human movements by means of machine vision, characterized by the fact that initially in the initial data module a database of images of the reference performance of the control movement is formed with an atlas of reference angular values ​​of joint opening in degrees for all degrees of freedom of these joints, then images of the performance of the control movement by the controlled person are obtained, according to which, using machine vision technology, a three-dimensional model of the controlled person is constructed in the frame, including designations of the location of key joints on the image of the musculoskeletal system of the controlled person in the form of reference points on his skeletal model,wherein the spatial position of the said points is assessed and, based on their angular movements relative to each other, the personal angular values ​​of the joint opening in degrees are determined for all degrees of freedom of these joints of the controlled person, after which the obtained personal angular values ​​of the joint opening are compared with the reference angular values ​​of the joint opening specified in the said atlas of reference values, characterized in that reference values ​​of the degrees of the joint opening angles in each degree of freedom are generated for each joint, and a set of threshold values ​​is also generated for each joint and the control movement itself, when performing the control movement, personal values ​​of the degrees of the opening angles of each analyzed joint in each degree of its freedom are generated based on the images obtained by means of machine vision,then, for each analyzed joint, the deviations between the reference and personal angular values ​​of the joint opening are calculated, then the average deviation is assessed for each joint, the average deviation estimates for the joints are compared with the threshold values ​​for the joints, thereby forming numerical measures of the correctness of the execution of the control movement for the joints, then the average value of all measures of the correctness of the execution of the control movement for the analyzed joints is assessed, which is compared with the threshold values ​​for the control movement, forming an assessment of the execution of the control movement of the person.,

[0013] Brief description of the drawings.

[0014] Fig. 1 shows a skeletal model of the human musculoskeletal system.

[0015] Implementation of the invention.

[0016] A method for assessing the performance of human movements using machine vision, characterized by the fact that initially in the initial data module a database is formed of images of the reference performance of control movements with an atlas of reference angular values ​​of joint opening in degrees for all degrees of freedom of these joints. An array of reference values ​​of degrees of joint opening angles in each degree of freedom is formed for each joint in accordance with the used model of the musculoskeletal system (MSS). The array of reference values ​​characterizes the correct location and opening of the joint in time during the performance of the control movement. A set of threshold values ​​is formed for each control movement, and a set of threshold values ​​is also formed for each joint (and the corresponding array of reference values).

[0017] Next, the person whose movements are subject to assessment is placed in front of a video camera that transmits his image to the software analysis module, in which, using machine vision technology, a search is made for the image of the given monitored person in the frame and an assessment of the position of his musculoskeletal system is made, after which a three-dimensional model of the monitored person in the frame is constructed, including designations of the location of key joints on the image of the monitored person's musculoskeletal system in the form of reference points on his skeletal model. In this case, reference points are placed at the locations of the joints, as well as on the upper and lower spine, at the level of the shoulder girdle and hip joints, respectively.

[0018] Next, the controlled person (in the implementation variant - using a set of test video files) performs a control movement. During the control movement, a video image of the movement is transmitted to the software analysis module. In the software analysis module, the spatial movement of the reference points of the skeletal model of the musculoskeletal system of the controlled person is processed and evaluated, determining the corresponding personal values ​​of the degrees of the opening angles of his joints in each degree of freedom. Thus, for each joint, arrays of real values ​​are formed that characterize the location of the joint of the controlled person - in time.

[0019] Next, for each joint, the elements of the array of reference and personal values ​​are compared with each other. The comparison is implemented as follows: initially, the arrays are time-calibrated so that the values ​​of the personal and reference elements of the arrays correspond to each other in terms of time chronology. Next, the deviations of the values ​​of the degrees of the joint opening angles between the personal and reference values ​​of the arrays are calculated, thereby forming an array of deviations. The deviations are calculated between the elements of the arrays corresponding to one moment in time. The calculation of each deviation is implemented as the modulus of the difference between the reference value and the personal value corresponding to it. Next, the average deviation is estimated for each array of deviations. The average is estimated by summing up all the elements of the array and dividing the obtained value by the number of elements of the array.In this way, average deviations of the value are formed for each of the joints.

[0020] The obtained average deviation for a specific joint is compared with the threshold values ​​for this joint, depending on the comparison results, an assessment of the correctness of the movement execution for a specific joint is formed, expressed as a number (in the embodiment, expressed as a percentage) (a measure of the correctness of the control movement execution for a specific joint). A similar procedure is performed for all joints - forming an array of assessments of the correctness of the movement execution for all joints. Next, an assessment of the average value of the correctness of the movement execution is carried out - for which all elements of the array of assessments of the correctness of the movement execution for all joints are summed up and the resulting number is divided by the number of elements of the array. The obtained assessment of the average value of the correctness of the movement execution is compared with the threshold values ​​for the controlled movement. Based on the comparison, an assessment of the control movement execution is carried out.

[0021] An example of the method implementation is the assessment of the correctness of the performance of a gymnastic exercise.

[0022] The model in Fig. 1 serves as a three-dimensional skeletal 3D model of a human, according to which the angular displacements of its support points located in the key joints of the human skeleton during the execution of control movements are calculated. Table 1 shows the support points of the model.

[0023] Initially, a database of images of the standard execution of the control warm-up gymnastic movement "squat" with an atlas of the angular values ​​of the opening of the joints in degrees for all degrees of freedom of these joints is formed in the initial data module. The database is formed using a recording of the exercise performed by a professional athlete.

[0024] Table 1

[0025] For example, the reference (table 2) angular values ​​of the leg joint opening in degrees when performing the exercise in the time interval from 0 to 1.24 seconds are given. For the joint "24,26,28" three threshold values ​​are formed: 14,18,20, where: the range from 0 to 14 characterizes "a good level of exercise performance" - the numerical characteristic "3";

[0026] Table 2 the range from 14 to 18 characterizes the "average level of exercise performance" - numerical characteristic "2"; the range from 18 to 20 characterizes the "low level of exercise performance" - numerical characteristic "1"; the range from 20 characterizes the "incorrect exercise performance" - numerical characteristic "0".

[0027] For the joint "23,25,27" three threshold values ​​are formed: 12,14,18, where: the range from 0 to 12 characterizes "good level of exercise performance" - numerical characteristic "3"; the range from 12 to 14 characterizes "average level of exercise performance" - numerical characteristic "2"; the range from 14 to 18 characterizes "low level of exercise performance" - numerical characteristic "1"; the range from 18 characterizes "incorrect exercise performance" - numerical characteristic "0".

[0028] For the joint "12,24,26" three threshold values ​​are formed: 12,15,20, where: the range from 0 to 12 characterizes "good level of exercise performance" - numerical characteristic "3"; the range from 12 to 15 characterizes "average level of exercise performance" - numerical characteristic "2"; the range from 15 to 20 characterizes "low level of exercise performance" - numerical characteristic "1"; the range from 20 characterizes "incorrect exercise performance" - numerical characteristic "0".

[0029] For the joint "11,23,25" three threshold values ​​are formed: 12,19,23, where: the range from 0 to 12 characterizes "good level of exercise performance" - numerical characteristic "3"; the range from 12 to 19 characterizes "average level of exercise performance" - numerical characteristic "2"; the range from 19 to 23 characterizes "low level of exercise performance" - numerical characteristic "1"; the range from 23 characterizes "incorrect exercise performance" - numerical characteristic "0".

[0030] For the squat exercise, two threshold values ​​are generally formed: 2.3. The range from 0 to 2 is characterized by an unsatisfactory assessment for the exercise; The range from 2 to 2.8 is characterized by a satisfactory assessment for the exercise; The range from 2.8 to 3 is characterized by an excellent assessment for the exercise. Then the controlled person performs a control movement, the corresponding degrees of freedom for each joint are determined in the software analysis module, and arrays of real values ​​for each joint are formed at the output (Table 3).

[0031] Table 3.

[0032] Next, for each joint, a comparison is made between the elements of the array of reference and personal values, forming a table of deviations (table 4).

[0033] Next, the average deviation is estimated for each array of deviations: for the joint “24;26;28” this value is 9.235; for the joint “23;25;27” this value is 10.512; for the joint “22;24;26” this value is 11.667; for the joint “11;23;25” this value is 16.022.

[0034] In accordance with the deviations, each joint was given an assessment and a corresponding numerical characteristic; for the joint “24;26;28” the numerical characteristic was 3; for the joint “23;25;27” the numerical characteristic was 3; for the joint “22;24;26” the numerical characteristic was 3; for the joint “11;23;25” the numerical characteristic was 2.

[0035] Thus, an array of assessments of the correctness of the exercise execution for all joints "3.3, 3.2" was formed. Then, the average value of the correctness of the exercise execution is assessed - the value of 2.75 is obtained. The resulting assessment of the average value of the correctness of the exercise execution is compared with the threshold values ​​for this movement - 2.75 falls within the range from 2 to 2.8, which allows for a satisfactory assessment of the exercise.

[0036] Table 4

[0037] In the embodiment of the claimed method, the threshold values, as well as the deviation tables, can be normalized in such a way that the final assessment is expressed as a percentage.

[0038] The scope of application of the claimed method is not limited to the field of sports. The solution can be used for educational purposes, in the service sector, production, etc. - anywhere where the correct execution of certain movements is required to achieve a result. In various embodiments of the claimed method, the average deviations for each joint can be calculated using both software and software-hardware and hardware devices. For example, using a subtractor block.

[0039] In various embodiments of the claimed method, the average deviation for each joint can be estimated using software, software, and hardware devices. For example, using threshold devices.

[0040] The claimed technical result - increasing the accuracy of assessing the performance of human movements - is achieved by using machine vision technologies, which allows calculating the angular values ​​of joint opening in degrees for all degrees of freedom of the joints based on the model of the musculoskeletal system. The claimed technical result is also achieved due to the fact that statistical processing of the results occurs, namely, the claimed method technically implements the procedure for assessing the average deviation of personal angular values ​​of joint opening from the reference ones, as well as assessing the average for all joints to form a final assessment of the correctness of the movement. The said statistical processing allows "smoothing" anomalous statistical values ​​​​that appear due to imperfect operation of the equipment, thereby ensuring an increase in the accuracy of the assessment.

[0041] The applicant implemented the claimed method in 2023, in particular, the claimed method was used for medical purposes to diagnose disorders in the musculoskeletal system. Testing of the method confirmed the claimed technical result - an increase in the accuracy of the assessment led to an increase in the accuracy of diagnostics by an average of 10-20%.

Claims

FORMULA 1. A method for assessing the performance of human movements using machine vision, characterized by the fact that initially, in the initial data module, a database of images of the reference performance of a control movement is formed with an atlas of reference angular values ​​of joint opening in degrees for all degrees of freedom of these joints, images of the controlled person performing the control movement are obtained, based on which, using machine vision technology, a three-dimensional model of the controlled person is constructed in a frame, including designations of the location of joints on the image of the musculoskeletal system of the controlled person in the form of reference points on his skeletal model, which are placed at the locations of the joints, as well as on the upper and lower spine, at the level of the shoulder girdle and hip joints, respectively,wherein the spatial position of the said reference points is assessed and, based on their angular movements relative to each other, the personal angular values ​​of the joint opening in degrees are determined for all degrees of freedom of the given joints of the controlled person; the obtained personal angular values ​​of the joint opening are compared with the reference angular values ​​of the joint opening specified in the said atlas of reference values; for each joint, reference values ​​of the degrees of the joint opening angles in each degree of freedom are generated; also, for each joint and the control movement itself, a set of threshold values ​​is generated; when the control movement is performed, personal values ​​of the degrees of the opening angles of each analyzed joint in each degree of its freedom are generated based on the images obtained by machine vision,the elements of the array of reference values ​​and the elements of the array of personal values ​​of the degrees of joint opening angles in each degree of freedom are formed, for each joint the elements of the array of reference and personal values ​​are compared with each other, the deviations of the values ​​of the degrees of joint opening angles between the personal and reference values ​​of the arrays are calculated as the modulus of the difference between the reference value and the personal value corresponding to it, thereby forming an array of deviations, the average deviation is estimated for each array of deviations by summing up all the elements of the array, 14 SUBSTITUTE SHEET (RULE 26) deviations and dividing the obtained value by the number of elements of the deviation array, thereby forming the average deviations of the value for each of the joints, the obtained average deviation for a particular joint is compared with the threshold values ​​for this joint, an assessment of the correctness of the movement execution for a particular joint is formed, expressed as a number, an array of assessments of the correctness of the movement execution for all joints is similarly formed, an assessment of the average value of the correctness of the control movement is carried out, for which all elements of the array of assessments of the correctness of the movement execution for all joints are summed up and the resulting number is divided by the number of elements of the array, a comparison is made of the assessment of the average value of the correctness of the control movement execution with the threshold values ​​for the control movement.

2. The method for assessing the performance of human movements using machine vision according to claim 1, characterized in that reference points are used that characterize: the left shoulder, the right shoulder, the left elbow, the right elbow, the left wrist, the right wrist, the little finger of the left hand, the little finger of the right hand, the index finger of the left hand, the index finger of the right hand, the thumb of the left hand, the thumb of the right hand, the left thigh, the right thigh, the left knee, the right knee, the left ankle, the right ankle, the left heel, the right heel, the index toe of the left foot, the index toe of the left foot, the index toe of the right foot, the lower vertebral reference point, the upper vertebral reference point. 15 SUBSTITUTE SHEET (RULE 26)

Images