A rehabilitation action evaluation method and system based on image recognition

Abstract

This invention discloses a rehabilitation movement assessment method and system based on image recognition, belonging to the field of image recognition technology. It includes calculating the human joint point coordinate sequence corresponding to each image frame, and calculating a similarity matrix between the human joint point coordinate sequence corresponding to each image frame and the human joint point coordinate sequence corresponding to each sample image frame in a sample rehabilitation movement video. Based on the coordinate similarity with the largest corresponding value in each row and column of the similarity matrix, the image frame and sample image frame corresponding to the same rehabilitation movement are determined. This aligns the image frame sequence of the user's rehabilitation movement with the sample image frame sequence of the sample rehabilitation movement video, and calculates the similarity between each image frame in the image frame sequence and the corresponding image frame in the sample image frame sequence to determine the user's rehabilitation movement completion score. This invention can eliminate the problem of low accuracy in rehabilitation movement assessment due to asynchrony, and improve the accuracy of rehabilitation movement completion assessment.

Description

Technical Field

[0001] This invention belongs to the field of image recognition technology, specifically relating to a rehabilitation movement assessment method and system based on image recognition. Background Technology

[0002] In rehabilitation medicine, accurate assessment of the completion of rehabilitation movements is a core element in developing personalized training programs and tracking rehabilitation progress. With the development of computer vision and artificial intelligence technologies, automated assessment methods based on image recognition are gradually becoming mainstream. These methods can effectively solve the problems of traditional assessments, which rely on therapists' manual observation and are characterized by strong subjectivity, low efficiency, and difficulty in quantification.

[0003] Image recognition-based rehabilitation movement assessment technology primarily achieves objective quantitative evaluation by analyzing the similarity between user movements and standard movements in reference videos, significantly improving assessment efficiency and repeatability. However, existing methods still face key challenges when dealing with dynamic scenarios. Due to differences in physical fitness and reaction speed, users' actual movement progress may not be synchronized with the reference video, such as delayed or premature movements, directly causing errors in similarity calculation and reducing assessment accuracy.

[0004] Therefore, how to provide an effective solution to avoid the problem of low accuracy in rehabilitation movement assessment due to inconsistencies between user rehabilitation movements and standard movements in reference videos has become an urgent problem to be solved in existing technologies. Summary of the Invention

[0005] The purpose of this invention is to provide a rehabilitation movement assessment method and system based on image recognition, in order to solve the above-mentioned problems existing in the prior art.

[0006] To achieve the above objectives, the present invention adopts the following technical solution: In a first aspect, the present invention provides a rehabilitation movement assessment method based on image recognition, comprising: Obtain the user's rehabilitation exercise videos; Image frames are extracted from the rehabilitation exercise video at preset time intervals to obtain an image frame sequence; Identify human joint points in each image frame of the image frame sequence; For each image frame in the image frame sequence, a coordinate system is established with the same reference point as the origin and the same direction as the coordinate axis. The coordinates of all human joints in each image frame are calculated to obtain the human joint coordinate sequence corresponding to each image frame. Calculate the similarity matrix between the human joint coordinate sequence corresponding to each image frame and the human joint coordinate sequence corresponding to each sample image frame extracted from the sample rehabilitation exercise video at preset time intervals. Select at least one coordinate similarity with the largest corresponding value from each row of the similarity matrix to obtain a first similarity list; select at least one coordinate similarity with the largest corresponding value from each column of the similarity matrix to obtain a second similarity list. If there are multiple sets of the same coordinate similarity in the first similarity list and the second similarity list, and the position of the corresponding image frame in the rehabilitation exercise video is consistent with the position of the corresponding sample image frame in the sample rehabilitation exercise video, then the image frames and sample image frames corresponding to each set of the same coordinate similarity are determined to be image frames of the same rehabilitation action, and the image frame sequence is aligned with the sample image frame sequence of the sample rehabilitation exercise video. After aligning the image frame sequence with the sample image frame sequence of the sample rehabilitation exercise video, the similarity between each image frame in the image frame sequence and the corresponding sample image frame in the sample image frame sequence is calculated. Based on the similarity between each image frame in the image frame sequence and the corresponding sample image frame in the sample image frame sequence, the user's rehabilitation action completion score is determined.

[0007] In one possible design, the human joints in each image frame include the nasal joint, left shoulder joint, right shoulder joint, left elbow joint, right elbow joint, left wrist joint, right wrist joint, spine joint, left knee joint, right knee joint, left ankle joint, and right ankle joint. The establishment of a coordinate system for each image frame in the image frame sequence, with the same reference point as the origin and the same direction as the coordinate axis, includes: For each image frame in the image frame sequence, a coordinate system is established with the center of the left and right shoulder joints as the origin, the direction from the left shoulder joint to the right shoulder joint as the horizontal axis, and the direction from the center of the left and right shoulder joints away from the spinal joint as the vertical axis.

[0008] In one possible design, the similarity matrix between the sequence of human joint coordinates corresponding to each image frame and the sequence of human joint coordinates corresponding to each sample image frame extracted from the sample rehabilitation exercise video at preset time intervals is calculated, including: Calculate the similarity between the human joint coordinate sequence corresponding to each image frame and the sample human joint coordinate sequence corresponding to each sample image frame, and obtain the similarity between the human joint coordinate sequence corresponding to each image frame and the sample human joint coordinate sequence corresponding to each sample image frame. Let the similarity between the human joint coordinate sequence corresponding to the i-th image frame and the human joint coordinate sequence corresponding to the j-th sample image frame be used as the element in the i-th row and j-th column to construct a similarity matrix, and obtain the similarity matrix between the human joint coordinate sequence corresponding to each image frame and the human joint coordinate sequence of each sample. Where i and j are both positive integers, and i≤I, j≤J, I represents the total number of image frames in the image frame sequence, and J represents the total number of sample image frames.

[0009] In one possible design, the similarity between the sequence of human joint coordinates corresponding to each image frame and the sequence of human joint coordinates corresponding to each sample image frame is calculated, including: Calculate the Euclidean distance between the human joint coordinate sequence corresponding to each image frame and the sample human joint coordinate sequence corresponding to each sample image frame to obtain the similarity between the human joint coordinate sequence corresponding to each image frame and the sample human joint coordinate sequence corresponding to each sample image frame.

[0010] In one possible design, aligning the image frame sequence with a sample image frame sequence from a sample rehabilitation exercise video includes: Based on the timestamps of the image frames and sample image frames corresponding to the same coordinate similarity in each group, the timestamps of each image frame in the image frame sequence corresponding to the rehabilitation exercise video are adjusted so that the timestamps of the image frames and sample image frames corresponding to the same coordinate similarity in each group are consistent.

[0011] In one possible design, the user's rehabilitation action completion score is determined based on the similarity between each image frame in the image frame sequence and the corresponding image frame in the sample image frame sequence, including: Based on the time axis distance between each image frame in the image frame sequence and the image frames in the image frame sequence that are determined to be the same rehabilitation action, the weight corresponding to each image frame in the image frame sequence is determined. The user's rehabilitation action completion score is calculated by weighting each image frame in the image frame sequence and the similarity between each image frame in the image frame sequence and the corresponding sample image frame in the sample image frame sequence.

[0012] In one possible design, the weight corresponding to any image frame in the image frame sequence is negatively correlated with the distance between any image frame and the image frames in the image frame sequence that are determined to be the same rehabilitation action.

[0013] Secondly, the present invention provides a rehabilitation movement assessment system based on image recognition, comprising: The acquisition unit is used to acquire the user's rehabilitation exercise videos; The extraction unit is used to extract image frames from the rehabilitation exercise video at preset time intervals to obtain an image frame sequence; The recognition unit is used to identify human joint points in each image frame of the image frame sequence; The coordinate calculation unit is used to establish a coordinate system for each image frame of the image frame sequence, with the same reference point as the origin and the same direction as the coordinate axis, and to calculate the coordinates of all human joints in each image frame to obtain the human joint coordinate sequence corresponding to each image frame. The first similarity calculation unit is used to calculate the similarity matrix between the human joint coordinate sequence corresponding to each image frame and the human joint coordinate sequence corresponding to each sample image frame extracted from the sample rehabilitation exercise video at preset time intervals. The selection unit is used to select at least one coordinate similarity with the largest corresponding value from each row of the similarity matrix to obtain a first similarity list, and to select at least one coordinate similarity with the largest corresponding value from each column of the similarity matrix to obtain a second similarity list; The alignment unit is configured to, if there are multiple sets of identical coordinate similarities in the first similarity list and the second similarity list, and the position of the corresponding image frame in the rehabilitation exercise video is consistent with the position of the corresponding sample image frame in the sample rehabilitation exercise video, then determine the image frames corresponding to each set of identical coordinate similarities and the sample image frame as image frames of the same rehabilitation action and align the image frame sequence with the sample image frame sequence of the sample rehabilitation exercise video; The second similarity calculation unit is used to calculate the similarity between each image frame in the image frame sequence and the corresponding sample image frame in the sample image frame sequence after aligning the image frame sequence with the sample image frame sequence of the sample rehabilitation exercise video, and to determine the user's rehabilitation action completion score based on the similarity between each image frame in the image frame sequence and the corresponding sample image frame in the sample image frame sequence.

[0014] Thirdly, the present invention provides an electronic device comprising a memory, a processor, and a transceiver connected in sequence, wherein the memory is used to store a computer program, the transceiver is used to send and receive messages, and the processor is used to read the computer program and execute the image recognition-based rehabilitation movement assessment method as described in the first aspect or any possible design of the first aspect.

[0015] Fourthly, the present invention provides a computer-readable storage medium storing instructions that, when executed on a computer, perform the image recognition-based rehabilitation movement assessment method described in the first aspect or any possible design of the first aspect.

[0016] Fifthly, the present invention provides a computer program product containing instructions that, when executed on a computer, cause the computer to perform the image recognition-based rehabilitation movement assessment method as described in the first aspect or any possible design of the first aspect.

[0017] Beneficial effects: This invention discloses an image recognition-based rehabilitation movement assessment scheme, which can eliminate the problem of low accuracy in rehabilitation movement assessment caused by the asynchronous progress of the user's actual movements and the reference video. Specifically, the scheme involves: acquiring the user's rehabilitation exercise video; extracting image frames from the rehabilitation exercise video at preset time intervals to obtain an image frame sequence; identifying the human joints in each image frame of the image frame sequence; establishing a coordinate system for each image frame of the image frame sequence, with the same reference point as the origin and the same direction as the coordinate axis, and calculating the coordinates of all human joints in each image frame to obtain a human joint coordinate sequence corresponding to each image frame; calculating the similarity matrix between the human joint coordinate sequence corresponding to each image frame and the sample human joint coordinate sequence corresponding to each sample image frame extracted from the sample rehabilitation exercise video at preset time intervals; selecting at least one coordinate similarity with the largest corresponding value from each row of the similarity matrix to obtain a first similarity list; and selecting at least one coordinate with the largest corresponding value from each column of the similarity matrix. Similarity is used to obtain a second similarity list. If there are multiple sets of identical coordinate similarities between the first and second similarity lists, and the position of the corresponding image frame in the rehabilitation exercise video is consistent with the position of the corresponding sample image frame in the sample rehabilitation exercise video, then the image frames and sample image frames corresponding to each set of identical coordinate similarities are determined to be image frames of the same rehabilitation action, and the image frame sequence is aligned with the sample image frame sequence of the sample rehabilitation exercise video. After aligning the image frame sequence with the sample image frame sequence of the sample rehabilitation exercise video, the similarity between each image frame in the image frame sequence and the corresponding sample image frame in the sample image frame sequence is calculated, and based on the similarity between each image frame in the image frame sequence and the corresponding sample image frame in the sample image frame sequence, the user's rehabilitation action completion score is determined. Thus, by calculating the human joint coordinate sequence corresponding to each image frame, and calculating the similarity matrix between the human joint coordinate sequence corresponding to each image frame and the human joint coordinate sequence corresponding to each sample image frame of the sample rehabilitation exercise video, and based on the coordinate similarity with the largest corresponding value in each row and column of the similarity matrix, the image frame and sample image frame corresponding to the same rehabilitation movement are determined. This aligns the image frame sequence of the user's rehabilitation movement with the sample image frame sequence of the sample rehabilitation exercise video, thereby solving the problem of the user's actual movement completion progress being out of sync with the reference video. It eliminates the problem of low accuracy in rehabilitation movement assessment caused by the asynchrony between the user's actual movement completion progress and the reference video, improves the accuracy of rehabilitation movement completion assessment, and provides an important basis for developing personalized training programs and tracking rehabilitation progress, facilitating practical application and promotion. Attached Figure Description

[0018] Figure 1A flowchart of the image recognition-based rehabilitation movement assessment method provided in the embodiments of this application; Figure 2 A schematic diagram of a rehabilitation movement assessment system based on image recognition provided in this application embodiment; Figure 3 This is a block diagram of an electronic device provided in an embodiment of this application. Detailed Implementation

[0019] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the present invention will be briefly introduced below in conjunction with the accompanying drawings and descriptions of the embodiments or the prior art. Obviously, the following description of the structure of the accompanying drawings is only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. It should be noted that the description of these embodiments is for the purpose of helping to understand the present invention, but does not constitute a limitation of the present invention.

[0020] It should be understood that although the terms first, second, etc., may be used herein to describe various units, these units should not be limited by these terms. These terms are only used to distinguish one unit from another. For example, a first unit may be referred to as a second unit, and similarly, a second unit may be referred to as a first unit, without departing from the scope of the exemplary embodiments of the invention.

[0021] It should be understood that the term "and / or" that may appear in this document is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can mean: A exists alone, B exists alone, and A and B exist simultaneously. The term " / and" that may appear in this document describes another relationship between related objects, indicating that two relationships can exist. For example, A / and B can mean: A exists alone, and A and B exist alone. In addition, the character " / " that may appear in this document generally indicates that the related objects before and after it are in an "or" relationship.

[0022] It should be understood that specific details are provided in the following description to facilitate a complete understanding of the exemplary embodiments. However, those skilled in the art will understand that the exemplary embodiments can be implemented without these specific details. For example, the system may be shown in block diagrams to avoid obscuring the example with unnecessary details. In other instances, well-known processes, structures, and techniques may be shown without unnecessary details to avoid obscuring the exemplary embodiments.

[0023] To avoid the problem of low accuracy in rehabilitation action assessment due to inconsistencies between user rehabilitation actions and standard actions in reference videos, this application provides a rehabilitation action assessment method and system based on image recognition. This image recognition-based rehabilitation action assessment method and system can eliminate the problem of low accuracy in rehabilitation action assessment caused by the asynchronous progress of the user's actual actions compared to the reference video, thereby improving the accuracy of rehabilitation action completion assessment.

[0024] The image recognition-based rehabilitation movement assessment method provided in this application can be applied to user terminals or servers. It is understood that the described execution entity does not constitute a limitation on the embodiments of this application.

[0025] The image recognition-based rehabilitation movement assessment method provided in the embodiments of this application will be described in detail below.

[0026] like Figure 1 The diagram shown is a flowchart of a rehabilitation movement assessment method based on image recognition provided in the first aspect of the present application. The rehabilitation movement assessment method based on image recognition may include, but is not limited to, the following steps S101-S108.

[0027] Step S101. Obtain the user's rehabilitation exercise video.

[0028] Among them, the user's rehabilitation exercise video can be a video in which the user performs rehabilitation exercises by referring to a standard rehabilitation exercise video.

[0029] Step S102. Extract image frames from the rehabilitation exercise video according to a preset time interval to obtain an image frame sequence.

[0030] The preset time interval can be set according to the actual situation. For example, a frame can be extracted from the rehabilitation exercise video every 1 second.

[0031] Step S103. Identify human joints in each image frame of the image frame sequence.

[0032] In one or more embodiments, human joints in each image frame of an image frame sequence can be identified using existing human joint recognition algorithms, which will not be elaborated further here.

[0033] Human joints in each image frame may include, but are not limited to, the nasal joint, left shoulder joint, right shoulder joint, left elbow joint, right elbow joint, left wrist joint, right wrist joint, spinal joint, left knee joint, right knee joint, left ankle joint, and right ankle joint.

[0034] Step S104. For each image frame in the image frame sequence, establish a coordinate system with the same reference point as the origin and the same direction as the coordinate axis, and calculate the coordinates of all human joints in each image frame to obtain the human joint coordinate sequence corresponding to each image frame.

[0035] Since the relative positions of the left and right shoulder joints are fixed, when establishing a coordinate system, we can use the center of the left and right shoulder joints as the origin, the direction from the left shoulder joint to the right shoulder joint as the horizontal axis, and the direction from the center of the left and right shoulder joints away from the spinal joint as the vertical axis for each image frame in the image frame sequence.

[0036] Understandably, in some other embodiments, a coordinate system can also be established using other relatively fixed human joint points as reference points.

[0037] Assuming that the human body has 12 joints, including the nasal joint, left shoulder joint, right shoulder joint, left elbow joint, right elbow joint, left wrist joint, right wrist joint, spine joint, left knee joint, right knee joint, left ankle joint, and right ankle joint, then the coordinate sequence of the human body joints corresponding to each image frame is a coordinate sequence containing the coordinates of the 12 joints: nasal joint, left shoulder joint, right shoulder joint, left elbow joint, right elbow joint, left wrist joint, right wrist joint, spine joint, left knee joint, right knee joint, left ankle joint, and right ankle joint.

[0038] Step S105. Calculate the similarity matrix between the human joint coordinate sequence corresponding to each image frame and the human joint coordinate sequence corresponding to each sample image frame extracted from the sample rehabilitation exercise video at preset time intervals.

[0039] In one or more embodiments, calculating the similarity matrix between the sequence of human joint coordinates corresponding to each image frame and the sequence of human joint coordinates corresponding to each sample image frame extracted from the sample rehabilitation exercise video at preset time intervals may include, but is not limited to, the following steps S1051-S1052.

[0040] Step S1051. Calculate the similarity between the human joint coordinate sequence corresponding to each image frame and the sample human joint coordinate sequence corresponding to each sample image frame, and obtain the similarity between the human joint coordinate sequence corresponding to each image frame and the sample human joint coordinate sequence corresponding to each sample image frame.

[0041] Step S1052. Let the similarity between the human joint coordinate sequence corresponding to the i-th image frame and the human joint coordinate sequence corresponding to the j-th sample image frame be used as the element in the i-th row and j-th column to construct a similarity matrix, thereby obtaining the similarity matrix between the human joint coordinate sequence corresponding to each image frame and the human joint coordinate sequence of each sample.

[0042] Where i and j are both positive integers, and i≤I, j≤J, I represents the total number of image frames in the image frame sequence, and J represents the total number of sample image frames.

[0043] For example, if an image frame sequence contains 20 image frames and a sample image frame sequence (for ease of explanation, the sequence consisting of all sample image frames is called the sample image frame sequence) contains 21 sample image frames, then a 20×21 similarity matrix can be calculated.

[0044] When calculating the similarity between the human joint coordinate sequence corresponding to each image frame and the sample human joint coordinate sequence corresponding to each sample image frame, the Euclidean distance between the human joint coordinate sequence corresponding to each image frame and the sample human joint coordinate sequence corresponding to each sample image frame can be calculated. The similarity between the human joint coordinate sequence corresponding to each image frame and the sample human joint coordinate sequence corresponding to each sample image frame can be obtained based on the Euclidean distance. The smaller the Euclidean distance, the greater the corresponding similarity.

[0045] Understandably, in other embodiments, the similarity matrix between the human joint coordinate sequence corresponding to each image frame and the human joint coordinate sequence of each sample can also be determined by calculating Mahalanobis distance.

[0046] Step S106. Select at least one coordinate similarity with the largest corresponding value from each row of the similarity matrix to obtain the first similarity list. Select at least one coordinate similarity with the largest corresponding value from each column of the similarity matrix to obtain the second similarity list.

[0047] Step S107. If there are multiple sets of the same coordinate similarity in the first similarity list and the second similarity list, and the position of the corresponding image frame in the rehabilitation exercise video is consistent with the position of the corresponding sample image frame in the sample rehabilitation exercise video, then the image frames and sample image frames corresponding to each set of the same coordinate similarity are determined to be image frames of the same rehabilitation action, and the image frame sequence is aligned with the sample image frame sequence of the sample rehabilitation exercise video.

[0048] The position of the image frame corresponding to a certain set of coordinate similarity in the rehabilitation exercise video is consistent with the position of the corresponding sample image frame in the sample rehabilitation exercise video. This can mean that the position of the image frame corresponding to a certain set of coordinate similarity in the rehabilitation exercise video is roughly the same as the position of the sample image frame corresponding to a certain set of coordinate similarity in the sample rehabilitation exercise video (e.g., both are in the foreground, near the middle, or in the background).

[0049] The image frame corresponding to coordinate similarity refers to the image frame corresponding to the human joint coordinate sequence corresponding to coordinate similarity, while the sample image frame corresponding to coordinate similarity refers to the sample image frame corresponding to the sample human joint coordinate sequence corresponding to coordinate similarity.

[0050] In this context, any set of identical coordinate similarities can mean that the two coordinate similarities in that set correspond to the same image frame and the same sample image frame. For example, if there is coordinate similarity XS1 in the first similarity list and coordinate similarity XS2 in the second similarity list, assuming that coordinate similarity XS1 is the similarity between the human joint coordinate sequence corresponding to image frame TX1 and the human joint coordinate sequence corresponding to sample image frame YTX1, and coordinate similarity XS2 is also the similarity between the human joint coordinate sequence corresponding to image frame TX1 and the human joint coordinate sequence corresponding to sample image frame YTX1, then coordinate similarity XS1 and coordinate similarity XS2 can be considered as a set of identical coordinate similarities.

[0051] In one or more embodiments, when aligning the image frame sequence with the sample image frame sequence of the sample rehabilitation exercise video, the timestamps of each image frame in the image frame sequence corresponding to the rehabilitation exercise video can be adjusted based on the timestamps of the image frames corresponding to the same coordinate similarity in each group and the sample image frames, so that the timestamps of the image frames corresponding to the same coordinate similarity in each group and the sample image frames are consistent.

[0052] Step S108. After aligning the image frame sequence with the sample image frame sequence of the sample rehabilitation exercise video, calculate the similarity between each image frame in the image frame sequence and the corresponding image frame in the sample image frame sequence, and determine the user's rehabilitation action completion score based on the similarity between each image frame in the image frame sequence and the corresponding image frame in the sample image frame sequence.

[0053] In one or more embodiments, the weights corresponding to each image frame in the image frame sequence can be determined based on the time-axis distance between each image frame in the image frame sequence and the image frames in the image frame sequence that are identified as performing the same rehabilitation action. Then, based on the weights corresponding to each image frame in the image frame sequence and the similarity between each image frame in the image frame sequence and the corresponding sample image frames in the sample image frame sequence, a weighted calculation of the user's rehabilitation action completion score is performed. The weight corresponding to any image frame in the image frame sequence is negatively correlated with the distance between that image frame and the image frames in the image frame sequence that are identified as performing the same rehabilitation action.

[0054] The present invention provides an image recognition-based rehabilitation movement assessment method, which involves: acquiring a user's rehabilitation exercise video; extracting image frames from the rehabilitation exercise video at preset time intervals to obtain an image frame sequence; identifying human joint points in each image frame of the image frame sequence; establishing a coordinate system for each image frame of the image frame sequence, with the same reference point as the origin and the same direction as the coordinate axis, and calculating the coordinates of all human joint points in each image frame to obtain a human joint point coordinate sequence corresponding to each image frame; calculating a similarity matrix between the human joint point coordinate sequence corresponding to each image frame and the sample human joint point coordinate sequence corresponding to each sample image frame extracted from the sample rehabilitation exercise video at preset time intervals; selecting at least one coordinate similarity with the largest corresponding value from each row of the similarity matrix to obtain a first similarity list; and selecting at least one coordinate similarity value from each column of the similarity matrix. The first similarity list and the second similarity list are used to obtain the coordinate similarity with the largest corresponding value. If there are multiple sets of the same coordinate similarity in the first similarity list and the second similarity list, and the position of the corresponding image frame in the rehabilitation exercise video is consistent with the position of the corresponding sample image frame in the sample rehabilitation exercise video, then the image frames and sample image frames corresponding to each set of the same coordinate similarity are determined to be image frames of the same rehabilitation action, and the image frame sequence is aligned with the sample image frame sequence of the sample rehabilitation exercise video. After aligning the image frame sequence with the sample image frame sequence of the sample rehabilitation exercise video, the similarity between each image frame in the image frame sequence and the corresponding sample image frame in the sample image frame sequence is calculated, and the user's rehabilitation action completion score is determined based on the similarity between each image frame in the image frame sequence and the corresponding sample image frame in the sample image frame sequence. Thus, by calculating the human joint coordinate sequence corresponding to each image frame, and calculating the similarity matrix between the human joint coordinate sequence corresponding to each image frame and the human joint coordinate sequence corresponding to each sample image frame of the sample rehabilitation exercise video, and based on the coordinate similarity with the largest corresponding value in each row and column of the similarity matrix, the image frame and sample image frame corresponding to the same rehabilitation movement are determined. This aligns the image frame sequence of the user's rehabilitation movement with the sample image frame sequence of the sample rehabilitation exercise video, thereby solving the problem of the user's actual movement completion progress being out of sync with the reference video. This eliminates the problem of low accuracy in rehabilitation movement assessment caused by the asynchrony between the user's actual movement completion progress and the reference video, improves the accuracy of rehabilitation movement completion assessment, and provides an important basis for developing personalized training programs and tracking rehabilitation progress, facilitating practical application and promotion.

[0055] Please see Figure 2 The second aspect of this application provides an image recognition-based rehabilitation movement assessment system, which includes: The acquisition unit is used to acquire the user's rehabilitation exercise videos; The extraction unit is used to extract image frames from the rehabilitation exercise video at preset time intervals to obtain an image frame sequence; The recognition unit is used to identify human joint points in each image frame of the image frame sequence; The coordinate calculation unit is used to establish a coordinate system for each image frame of the image frame sequence, with the same reference point as the origin and the same direction as the coordinate axis, and to calculate the coordinates of all human joints in each image frame to obtain the human joint coordinate sequence corresponding to each image frame. The first similarity calculation unit is used to calculate the similarity matrix between the human joint coordinate sequence corresponding to each image frame and the human joint coordinate sequence corresponding to each sample image frame extracted from the sample rehabilitation exercise video at preset time intervals. The selection unit is used to select at least one coordinate similarity with the largest corresponding value from each row of the similarity matrix to obtain a first similarity list, and to select at least one coordinate similarity with the largest corresponding value from each column of the similarity matrix to obtain a second similarity list; The alignment unit is configured to, if there are multiple sets of identical coordinate similarities in the first similarity list and the second similarity list, and the position of the corresponding image frame in the rehabilitation exercise video is consistent with the position of the corresponding sample image frame in the sample rehabilitation exercise video, then determine the image frames corresponding to each set of identical coordinate similarities and the sample image frame as image frames of the same rehabilitation action and align the image frame sequence with the sample image frame sequence of the sample rehabilitation exercise video; The second similarity calculation unit is used to calculate the similarity between each image frame in the image frame sequence and the corresponding sample image frame in the sample image frame sequence after aligning the image frame sequence with the sample image frame sequence of the sample rehabilitation exercise video, and to determine the user's rehabilitation action completion score based on the similarity between each image frame in the image frame sequence and the corresponding sample image frame in the sample image frame sequence.

[0056] The working process, working details and technical effects of the image recognition-based rehabilitation movement assessment system provided in the second aspect of this embodiment can be found in the first aspect of the embodiment, and will not be repeated here.

[0057] Please see Figure 3 The third aspect of this application provides an electronic device, including a memory, a processor, and a transceiver that are sequentially and communicatively connected, wherein the memory is used to store a computer program, the transceiver is used to send and receive messages, and the processor is used to read the computer program and execute the image recognition-based rehabilitation action assessment method as described in the first aspect of the application.

[0058] Specifically, the memory may include, but is not limited to, random access memory (RAM), read-only memory (ROM), flash memory, first-in-first-out (FIFO) memory, and / or last-in-first-out (FILO) memory, etc.; the processor may not be limited to microprocessors of the STM32F105 series, ARM (Advanced RISC Machines), x86 architecture processors, or processors with integrated NPU (neural-network processing units); the transceiver may be, but is not limited to, WiFi (Wireless Fidelity) wireless transceivers, Bluetooth wireless transceivers, General Packet Radio Service (GPRS) wireless transceivers, ZigBee (a low-power LAN protocol based on the IEEE 802.15.4 standard), 3G transceivers, 4G transceivers, and / or 5G transceivers, etc.

[0059] This fourth aspect of the embodiment provides a computer-readable storage medium storing instructions containing the image recognition-based rehabilitation movement assessment method described in the first aspect of the embodiment. Specifically, the computer-readable storage medium stores instructions that, when executed on a computer, perform the image recognition-based rehabilitation movement assessment method as described in the first aspect. The computer-readable storage medium refers to a data storage medium, which may include, but is not limited to, floppy disks, optical disks, hard disks, flash memory, USB flash drives, and / or Memory Sticks. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices.

[0060] The fifth aspect of this embodiment provides a computer program product containing instructions that, when executed on a computer, cause the computer to perform the image recognition-based rehabilitation action assessment method as described in the first aspect of the embodiment, wherein the computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device.

[0061] Finally, it should be noted that the above description is merely a preferred embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. An image recognition-based rehabilitation movement evaluation method, characterized by, include: Obtain the user's rehabilitation exercise videos; Image frames are extracted from the rehabilitation exercise video at preset time intervals to obtain an image frame sequence; Identify human joint points in each image frame of the image frame sequence; For each image frame in the image frame sequence, a coordinate system is established with the same reference point as the origin and the same direction as the coordinate axis. The coordinates of all human joints in each image frame are calculated to obtain the human joint coordinate sequence corresponding to each image frame. Calculate the similarity matrix between the human joint coordinate sequence corresponding to each image frame and the human joint coordinate sequence corresponding to each sample image frame extracted from the sample rehabilitation exercise video at preset time intervals. Select at least one coordinate similarity with the largest corresponding value from each row of the similarity matrix to obtain a first similarity list; select at least one coordinate similarity with the largest corresponding value from each column of the similarity matrix to obtain a second similarity list. If there are multiple sets of the same coordinate similarity in the first similarity list and the second similarity list, and the position of the corresponding image frame in the rehabilitation exercise video is consistent with the position of the corresponding sample image frame in the sample rehabilitation exercise video, then the image frames and sample image frames corresponding to each set of the same coordinate similarity are determined to be image frames of the same rehabilitation action, and the image frame sequence is aligned with the sample image frame sequence of the sample rehabilitation exercise video. After aligning the image frame sequence with the sample image frame sequence of the sample rehabilitation exercise video, the similarity between each image frame in the image frame sequence and the corresponding sample image frame in the sample image frame sequence is calculated. Based on the similarity between each image frame in the image frame sequence and the corresponding sample image frame in the sample image frame sequence, the user's rehabilitation action completion score is determined.

2. The image recognition-based rehabilitation movement evaluation method according to claim 1, characterized in that, The human joints in each image frame include the nasal joint, left shoulder joint, right shoulder joint, left elbow joint, right elbow joint, left wrist joint, right wrist joint, spine joint, left knee joint, right knee joint, left ankle joint, and right ankle joint. The establishment of a coordinate system for each image frame in the image frame sequence, with the same reference point as the origin and the same direction as the coordinate axis, includes: For each image frame in the image frame sequence, a coordinate system is established with the center of the left and right shoulder joints as the origin, the direction from the left shoulder joint to the right shoulder joint as the horizontal axis, and the direction from the center of the left and right shoulder joints away from the spinal joint as the vertical axis.

3. The image recognition-based rehabilitation movement evaluation method according to claim 1, characterized in that, Calculate the similarity matrix between the sequence of human joint coordinates corresponding to each image frame and the sequence of human joint coordinates corresponding to each sample image frame extracted from the sample rehabilitation exercise video at preset time intervals, including: Calculate the similarity between the human joint coordinate sequence corresponding to each image frame and the sample human joint coordinate sequence corresponding to each sample image frame, and obtain the similarity between the human joint coordinate sequence corresponding to each image frame and the sample human joint coordinate sequence corresponding to each sample image frame. Let the similarity between the human joint coordinate sequence corresponding to the i-th image frame and the human joint coordinate sequence corresponding to the j-th sample image frame be used as the element in the i-th row and j-th column to construct a similarity matrix, and obtain the similarity matrix between the human joint coordinate sequence corresponding to each image frame and the human joint coordinate sequence of each sample. Where i and j are both positive integers, and i≤I, j≤J, I represents the total number of image frames in the image frame sequence, and J represents the total number of sample image frames.

4. The image recognition-based rehabilitation movement evaluation method according to claim 3, characterized in that, Calculate the similarity between the sequence of human joint coordinates corresponding to each image frame and the sequence of human joint coordinates corresponding to each sample image frame, including: Calculate the Euclidean distance between the human joint coordinate sequence corresponding to each image frame and the sample human joint coordinate sequence corresponding to each sample image frame to obtain the similarity between the human joint coordinate sequence corresponding to each image frame and the sample human joint coordinate sequence corresponding to each sample image frame.

5. The image recognition-based rehabilitation movement evaluation method according to claim 1, characterized in that, Aligning the image frame sequence with the sample image frame sequence of the sample rehabilitation exercise video includes: Based on the timestamps of the image frames and sample image frames corresponding to the same coordinate similarity in each group, the timestamps of each image frame in the image frame sequence corresponding to the rehabilitation exercise video are adjusted so that the timestamps of the image frames and sample image frames corresponding to the same coordinate similarity in each group are consistent.

6. The image recognition-based rehabilitation movement evaluation method according to claim 1, characterized in that, Based on the similarity between each image frame in the image frame sequence and the corresponding image frame in the sample image frame sequence, the user's rehabilitation action completion score is determined, including: Based on the time axis distance between each image frame in the image frame sequence and the image frames in the image frame sequence that are determined to be the same rehabilitation action, the weight corresponding to each image frame in the image frame sequence is determined. The user's rehabilitation action completion score is calculated by weighting each image frame in the image frame sequence and the similarity between each image frame in the image frame sequence and the corresponding sample image frame in the sample image frame sequence.

7. The image recognition-based rehabilitation movement evaluation method according to claim 6, characterized in that, The weight corresponding to any image frame in the image frame sequence is negatively correlated with the distance between any image frame and the image frames in the image frame sequence that are determined to be the same rehabilitation action.

8. An image recognition-based rehabilitation movement evaluation system, characterized by, include: The acquisition unit is used to acquire the user's rehabilitation exercise videos; The extraction unit is used to extract image frames from the rehabilitation exercise video at preset time intervals to obtain an image frame sequence; The recognition unit is used to identify human joint points in each image frame of the image frame sequence; The coordinate calculation unit is used to establish a coordinate system for each image frame of the image frame sequence, with the same reference point as the origin and the same direction as the coordinate axis, and to calculate the coordinates of all human joints in each image frame to obtain the human joint coordinate sequence corresponding to each image frame. The first similarity calculation unit is used to calculate the similarity matrix between the human joint coordinate sequence corresponding to each image frame and the human joint coordinate sequence corresponding to each sample image frame extracted from the sample rehabilitation exercise video at preset time intervals. The selection unit is used to select at least one coordinate similarity with the largest corresponding value from each row of the similarity matrix to obtain a first similarity list, and to select at least one coordinate similarity with the largest corresponding value from each column of the similarity matrix to obtain a second similarity list; The alignment unit is configured to, if there are multiple sets of identical coordinate similarities in the first similarity list and the second similarity list, and the position of the corresponding image frame in the rehabilitation exercise video is consistent with the position of the corresponding sample image frame in the sample rehabilitation exercise video, then determine the image frames corresponding to each set of identical coordinate similarities and the sample image frame as image frames of the same rehabilitation action and align the image frame sequence with the sample image frame sequence of the sample rehabilitation exercise video; The second similarity calculation unit is used to calculate the similarity between each image frame in the image frame sequence and the corresponding sample image frame in the sample image frame sequence after aligning the image frame sequence with the sample image frame sequence of the sample rehabilitation exercise video, and to determine the user's rehabilitation action completion score based on the similarity between each image frame in the image frame sequence and the corresponding sample image frame in the sample image frame sequence.

9. An electronic device, comprising: The device includes a memory, a processor, and a transceiver that are sequentially and communicatively connected. The memory is used to store a computer program, the transceiver is used to send and receive messages, and the processor is used to read the computer program and execute the image recognition-based rehabilitation movement assessment method as described in any one of claims 1 to 7.

10. A computer program product comprising computer programs or instructions, characterized in that, When the computer program or the instructions are executed by the computer, they implement the image recognition-based rehabilitation movement assessment method as described in any one of claims 1 to 7.

Images