Motion guidance method, controller, training device, program product, and medium

Abstract

The motion guidance method, the controller, the training device, the program product and the medium provided in the embodiments of the present disclosure comprise the following steps: based on the recognition of target image data taken from a trainee, the position information of a plurality of key points on the human body of the trainee related to a part to be trained is recognized; a guidance position to which a motion key point needs to reach in performing a preset action is determined, and guidance is formed on the trainee according to the guidance position; based on a target image data set collected in time sequence from the trainee in the process of performing the preset action, position motion time sequence information of the plurality of key points is obtained; and in response to the position motion time sequence information indicating that at least one fixed key point has a deviation event, it is determined that the trainee uses a compensatory action when performing the preset action. The present disclosure is based on image recognition of human body key points and position information of a motion process, accurately determines the action state of the trainee and correctly guides the accurate guidance position, and can detect an undesirable compensatory action, thereby improving the rehabilitation training effect.

Description

Technical Field

[0001] This disclosure relates to the field of rehabilitation medical device technology, and in particular to motion guidance methods, controllers, training equipment, program products and media. Background Technology

[0002] As an important branch of medical robotics, rehabilitation robots encompass numerous fields, including rehabilitation medicine, biomechanics, mechanics, electronics, materials science, computer science, and robotics, and have become a research hotspot in the international robotics field. Rehabilitation robots have been widely applied in rehabilitation nursing, prosthetics, and rehabilitation therapy, which has not only promoted the development of rehabilitation medicine but also driven the development of new technologies and theories in related fields.

[0003] Rehabilitation robots guide patients' limb movements to achieve rehabilitation training goals. However, existing rehabilitation robots can only roughly estimate the patient's level of participation and condition in movement by using motion data obtained from encoders or sensors in the drive motors. This estimation is inaccurate and often deviates significantly from the actual movement. For example, there are deviations in the distance and orientation of the actual movement, making it difficult to assess the limb's movement status and detect a series of negative movements such as limb compensation. This affects the user experience and is detrimental to the effectiveness of rehabilitation training.

[0004] In addition, during this process, the guiding position of the terminal effector or exoskeleton, which is the actuator for force interaction with the patient, is often a fixed or random position, and the parameters for generating the guiding position are not adjusted according to the patient's recovery status and progress, which is not flexible enough and will also affect the user experience.

[0005] These and other issues have become urgent technical problems that the industry needs to solve. Summary of the Invention

[0006] In view of the shortcomings of the prior art described above, the purpose of this disclosure is to provide motion guidance methods, controllers, training equipment, program products and media to solve the problems in the related art.

[0007] The first aspect of this disclosure provides a motion guidance method applied to a motion training device, the motion training device including an interactive motion mechanism capable of force interaction and movement with at least one affected limb of a trainee; the method includes: obtaining positional information of multiple key points on the trainee's body related to the training area based on the recognition of target image data acquired from the trainee; wherein the multiple key points include: fixed key points defined as fixed in position during a preset action, and motion key points that move; determining the guidance position that the motion key points need to reach during the execution of the preset action, and guiding the trainee according to the guidance position; acquiring positional motion temporal information of the multiple key points based on a target image dataset acquired sequentially during the execution of the preset action by the trainee; and determining that the trainee uses a compensatory action when performing the preset action in response to the positional motion temporal information indicating that at least one of the fixed key points has shifted.

[0008] In an embodiment of the first aspect, the step of determining that the trainee performs a compensatory action in response to the positional motion timing information indicating that at least one of the fixed key points has shifted includes at least one of the following: 1) determining that the trainee performs a compensatory action when the rotation angle of at least one of the fixed key points reaches a preset angle threshold; 2) determining that the trainee performs a compensatory action when the rotation angle of at least one of the fixed key points reaches the preset angle threshold and continues for a preset duration; 3) determining that the trainee performs a compensatory action when the displacement distance of at least one of the fixed key points reaches a preset displacement threshold; 4) determining that the trainee performs a compensatory action when the displacement distance of at least one of the fixed key points reaches the preset displacement threshold and continues for a preset duration.

[0009] In an embodiment of the first aspect, a first current position of the interactive motion mechanism is estimated based on the drive stroke data of the interactive motion mechanism; a second current position of the interactive motion mechanism is determined based on the currently captured target image data; the deviation between the first current position and the second current position is compared; if the comparison result indicates that the deviation is large enough to meet a preset deviation condition, the current position of the interactive motion mechanism is corrected to the second current position; and the positions of the multiple key points of the trainee identified in the target image data at the current and / or previous times are corrected according to the second current position.

[0010] In an embodiment of the first aspect, determining the guiding position that the key movement point needs to reach in performing a preset action includes: acquiring the movement intention information of the trainee's affected limb; determining the guiding position that conforms to the movement intention; and forming a guide.

[0011] In an embodiment of the first aspect, determining the guiding position that conforms to the movement intention and forming the guidance includes at least one of the following: 1) obtaining the first position movement timing information of each key point of the trainee's healthy limb and the second position movement timing information of each key point of the affected limb paired with the healthy limb based on a target image dataset acquired in a time sequence, and determining the contralateral movement that coordinates with or is symmetrical to the movement of the healthy limb as the target pose of the affected limb, and determining the guiding position of the movement key points on the affected limb based on the target pose; 2) obtaining the first position movement timing information of each key point of the trainee's healthy limb and the second position movement timing information of each key point of the affected limb paired with the healthy limb based on a target image dataset acquired in a time sequence. The system includes: 1) determining the second positional motion timing information of key points of the body; 2) predicting a first preset movement of the unaffected limb based on the first positional motion timing information, and determining a second preset movement of the opposite side that is symmetrical or coordinated with the first preset movement; 3) determining a guiding position of the affected limb that conforms to the trainee's movement intention based on the second preset movement; 4) determining a guiding position that conforms to the movement intention based on instruction information indicating the trainee's movement intention of the affected limb, and forming guidance; 5) determining a guiding position that conforms to the movement intention based on the expected movement corresponding to the compensatory movement, and forming guidance based on the expected movement corresponding to the compensatory movement, in response to detecting that the trainee is using a compensatory movement; the expected movement includes at least one of the following: a preset movement; a reference movement of the unaffected limb. In an embodiment of the first aspect, determining a guiding position that conforms to the movement intention based on instruction information indicating the trainee's movement intention of the affected limb and forming guidance includes: determining a movement intention corresponding to the biometric information output by the trainee, in response to biometric information output by the trainee, determining a guiding position that conforms to the movement intention, and forming guidance.

[0012] In an embodiment of the first aspect, the motion guidance method further includes: in response to detecting that the trainee is using compensatory movements and / or meets the condition of insufficient movement completion ability, controlling the interactive motion mechanism to move the trainee's affected limb to bring the key movement point to the guidance position.

[0013] In an embodiment of the first aspect, the insufficient action completion capability condition includes at least one of the following: the static state of the interactive motion mechanism lasts for a preset duration; the movement distance of the interactive motion mechanism within the preset duration is less than a preset distance threshold; the movement speed of the interactive motion mechanism is lower than a preset speed threshold.

[0014] In an embodiment of the first aspect, guiding the trainee according to the guidance position includes: outputting guidance information prompting the trainee to perform an action so that the key points of movement reach the guidance position, the guidance information including one of the following: voice, image, text or assistance from the interactive motion mechanism; and / or, the method further includes: displaying a mapping image on a display, the movement of a virtual object in the mapping image being correlated with the action performed by the trainee.

[0015] In an embodiment of the first aspect, the target image data includes at least one of the following: color images and depth information with at least partial pixel overlap; multiple color images acquired from different viewpoints and all containing the trainee; multiple color images and depth information acquired from different viewpoints and with at least partial pixel overlap.

[0016] In an embodiment of the first aspect, the training area is located on the upper limb of the human body, including at least one of the shoulder joint and the elbow joint; the multiple key points related to the elbow include: fixed key points including the elbow joint, shoulder joint and head representation points, and motion key points including the hand representation point.

[0017] A second aspect of this disclosure provides a controller, comprising: a processor and a memory; the memory storing program instructions; the processor being configured to execute the program instructions to perform the motion guidance method as described in any one aspect of the first disclosure.

[0018] This disclosure provides a sports training device in a third aspect, comprising: a device body forming a recognition area; an interactive motion mechanism movably disposed in the recognition area; a 3D image sensor disposed in the device body for capturing target image data; the field of view of the 3D image sensor covering the recognition area; and a controller as described in the second aspect, communicatively connected to the 3D image sensor.

[0019] In a third embodiment, the device body includes: a height-adjustable table with its desktop forming the recognition area; the interactive motion mechanism is disposed on the desktop; and / or, the motion training device further includes: a display disposed on the device body and communicatively connected to the controller; and / or, the 3D image sensor is disposed above the display.

[0020] This disclosure provides a fourth aspect of a computer program product, comprising: program instructions for executing the motion guidance method as described in any of the first aspects.

[0021] The fifth aspect of this disclosure provides a computer-readable storage medium storing program instructions that are executed to perform the motion guidance method as described in any one of the first aspects.

[0022] As described above, this disclosure provides a motion guidance method, controller, training device, program product, and medium. The method includes: identifying the positional information of multiple key points on the trainee's body related to the training area based on the recognition of target image data captured from the trainee; determining the guidance position that the motion key points need to reach during the execution of a preset action, and guiding the trainee according to the guidance position; acquiring the positional motion temporal information of multiple key points based on a target image dataset collected sequentially during the execution of the preset action; and determining that the trainee uses a compensatory action when performing the preset action in response to an event indicating that at least one fixed key point has shifted according to the positional motion temporal information. This disclosure is based on image recognition of the positional information of the trainee's body key points and the motion process, accurately determining the trainee's motion state, and providing accurate guidance positions for correct guidance, thereby improving the rehabilitation training effect.

[0023] Therefore, on the one hand, when faced with complex symptoms, it can also overcome the limitations of motor encoders and sensors, accurately record the movement trajectory of the user's upper limbs in all aspects, and provide feedback to the therapist and the user.

[0024] On the other hand, because it uses non-contact detection based on image recognition, it can facilitate and speed up rehabilitation training, take into account patients with mobility impairments, avoid the inconvenience of wearable detection, and effectively and accurately record data during exercise.

[0025] On the other hand, this disclosed solution can eliminate the uncertainty in estimating the user's joint movements caused by current terminal or exoskeleton rehabilitation robots, and meet the data recording (image recording) requirements of patients' rehabilitation exercises in the visual dimension. Based on this, it can effectively improve the user's correct training posture and provide patients and therapists with multi-dimensional recording and analysis channels. Attached Figure Description

[0026] Figure 1 A schematic diagram of the structure of a sports training device according to one embodiment of the present disclosure is shown.

[0027] Figure 2 A schematic diagram of the module structure of the controller in one embodiment of this disclosure is shown.

[0028] Figure 3 A flowchart illustrating a motion guidance method according to one embodiment of this disclosure is shown.

[0029] Figure 4 A schematic diagram illustrating the key points for the correct execution of preset actions in one embodiment of this disclosure is shown.

[0030] Figure 5 A schematic diagram illustrating the key points of the compensatory action execution in one embodiment of this disclosure is shown.

[0031] Figure 6 A flowchart illustrating an exemplary judgment logic for a compensatory action in one embodiment of this disclosure is shown.

[0032] Figure 7 A schematic diagram illustrating key points of the affected limb guided by reference to the healthy limb in one embodiment of this disclosure.

[0033] Figure 7 A schematic diagram illustrating key points of the affected limb guided by reference to the healthy limb in one embodiment of this disclosure.

[0034] Figure 8 This illustration shows a flowchart of a process for calibrating key point location information identified from target image data in one embodiment of the present disclosure.

[0035] Figure 9 Demonstrated in a specific embodiment Figure 8 A simplified diagram illustrating the principle of the process flow.

[0036] Figure 10 A schematic diagram of a motion guidance device module is shown in one embodiment of this disclosure. Detailed Implementation

[0037] The following specific examples illustrate the implementation of this disclosure. Those skilled in the art can easily understand other advantages and effects of this disclosure from the information disclosed herein. This disclosure can also be implemented or applied through other different specific embodiments, and various details in this disclosure can be modified or changed according to different viewpoints and application modules without departing from the spirit of this disclosure. It should be noted that, unless otherwise specified, the embodiments and features in the embodiments of this disclosure can be combined with each other.

[0038] The embodiments of this disclosure will now be described in detail with reference to the accompanying drawings, so that those skilled in the art to which this disclosure pertains can readily implement it. This disclosure may be embodied in many different forms and is not limited to the embodiments described herein.

[0039] In this disclosure, references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic represented in connection with that embodiment or example is included in at least one embodiment or example of this disclosure. Furthermore, the specific features, structures, materials, or characteristics represented may be combined in any suitable manner in any one or a group of embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples represented in this disclosure, as well as the features of those different embodiments or examples.

[0040] Furthermore, the terms "first" and "second" are used for illustrative purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the representation of this disclosure, "a set" means two or more, unless otherwise explicitly specified.

[0041] For the purpose of clarity, devices unrelated to the description are omitted, and the same or similar components throughout the specification are given the same reference numerals.

[0042] Throughout this specification, when it is said that a device is "connected" to another device, this includes not only "direct connection" but also "indirect connection" by placing other components in between. Furthermore, when it is said that a device "comprises" a certain constituent element, unless otherwise stated otherwise, this does not exclude other constituent elements, but rather implies that other constituent elements may be included.

[0043] While the terms first, second, etc., are used in some examples herein to refer to various elements, these elements should not be limited by these terms. These terms are used only to distinguish one element from another. For example, first interface and second interface, etc., are used. Furthermore, as used herein, the singular forms “a,” “an,” and “the” are intended to also include the plural forms unless the context indicates otherwise. It should be further understood that the terms “comprising,” “including,” indicate the presence of the stated feature, step, operation, element, module, item, kind, and / or group, but do not exclude the presence, occurrence, or addition of one or more other features, steps, operations, elements, modules, items, kinds, and / or groups. The terms “or” and “and / or” as used herein are interpreted as inclusive, or mean any one or any combination thereof. Thus, “A, B, or C” or “A, B, and / or C” means “any one of: A; B; C; A and B; A and C; B and C; A, B, and C.” Exceptions to this definition will only occur if the combination of elements, functions, steps, or operations is inherently mutually exclusive in some way.

[0044] The technical terms used herein are for reference only to specific embodiments and are not intended to limit the scope of this disclosure. The singular form used herein includes the plural form unless the statement explicitly indicates otherwise. The word "comprising" as used in this specification means to specify a particular characteristic, region, integer, step, operation, element, and / or component, and does not exclude the presence or addition of other characteristics, regions, integers, steps, operations, elements, and / or components.

[0045] Although not explicitly defined, all terms, including technical and scientific terms used herein, shall have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure pertains. Terms defined in commonly used dictionaries shall be further interpreted as having a meaning consistent with the relevant technical literature and the message of the present disclosure, and shall not be over-interpreted as having an ideal or overly formulaic meaning unless otherwise defined.

[0046] Rehabilitation robots have a wide range of applications, including various rehabilitation training and treatments such as physical therapy, exercise training, speech rehabilitation, and neurological rehabilitation. Taking physical therapy as an example, rehabilitation robots can help patients perform various limb movements, such as arm extension, knee flexion and extension, and ankle joint movements, to strengthen muscle strength and joint mobility. In neurological rehabilitation, rehabilitation robots can stimulate the nervous system to promote neural reconstruction and recovery.

[0047] However, existing rehabilitation robots can only roughly estimate the patient's level of participation and condition in movement by using motion data obtained from encoders or sensors of drive motors when guiding the patient's limb movements. This is not accurate and often has a large error compared to the patient's actual movement. This error makes it difficult to assess the limb's movement status and detect negative movements such as limb compensation, which is detrimental to rehabilitation.

[0048] Furthermore, current rehabilitation robots guide patients to random or fixed locations, and cannot adjust their positions according to the patient's recovery progress, which is inflexible and affects the user experience.

[0049] In view of the above, this disclosure provides a motion guidance method for use in sports training equipment to solve the problems in the related art.

[0050] First, the application scenarios of the methods in the embodiments of this disclosure are introduced. The exercise training device can be for training the patient's lower limbs, the patient's upper limbs, or other parts of the patient's body. The following explanation will focus on training the patient's upper limbs.

[0051] like Figure 1 The diagram shown illustrates the structure of a sports training device according to an embodiment of this disclosure.

[0052] For example, the exercise training device 100 may be a rehabilitation robot. The exercise training device 100 may include a device body 101, an interactive motion mechanism 102, a 3D image sensor 103, and a controller 104. For example, in Figure 1 The exercise training equipment 100 is used for upper limb training of patients.

[0053] Specifically, the device body 101 forms a recognition area, which can be a spatial area used for the patient's limb movements and can be used for image acquisition. In a further example, corresponding to the needs of upper limb training, the device body 101 may include a height-adjustable table 111, with the recognition area formed above its tabletop. Exemplarily, the table 111 is height-adjustably mounted on a base 112, which may have feet. Exemplarily, the bottom of the feet may be flat, forming a stable support. Alternatively, the feet may be movable, such as rollers or casters, facilitating the movement of the base 112. Exemplarily, the lifting and lowering of the table 111 can be driven by a drive source such as a cylinder or motor, and can be achieved with a structure such as a sliding guide (e.g., a slider and a slide rail). A mechanism for changing the direction of motion (e.g., a lead screw or gear) can also be used to convert the power direction of the drive source to drive the table 111 up and down.

[0054] exist Figure 1 In this embodiment, the interactive motion mechanism 102 includes an end effector 121, which is a device that allows the trainee's affected limb to interact with and is movably positioned within the recognition area. Exemplarily, the end effector 121 can receive force applied by the trainee's affected limb and move accordingly. The end effector 121 can also be driven by a drive motor, which can be controlled by a controller 104, allowing the end effector 121 to perform controlled active movement to move the trainee's affected limb.

[0055] exist Figure 1 In this embodiment, the interactive motion mechanism 102 is located on the desktop and can move laterally within the plane of the desktop via a quadrilateral movable linkage mechanism. Since the device in this example trains the patient's upper limb, the hand of the affected limb can grasp the terminal effector 121, and when the hand of the affected limb moves, the terminal effector 121 can be moved accordingly. In other embodiments, if the training area of ​​the motion training device 100 changes, such as limb movement, the type of the interactive motion mechanism 102 also changes accordingly, for example, it can be changed from the terminal effector 121 to an exoskeleton, etc. This is for illustrative purposes only and is not limited to applications such as... Figure 1 The sports training equipment shown is of type 100, but it can also be of other types.

[0056] The 3D image sensor 103 is configured to capture images within the recognition area, thereby acquiring images of the trainee's movement relative to the terminal effector 121. This allows for the determination of the spatial position information of each point within the area, enabling the identification and analysis of the spatial pose of key points on the patient's limbs. For example, these key points may be key points constituting the human skeleton, such as joints or key parts of the head, hands, elbows, shoulders, etc. Exemplarily, the 3D image sensor 103 can be a motion-sensing camera, also known as a depth camera, capable of acquiring RGB-D information, where RGB represents the pixel value of each pixel and D represents depth information. It integrates two sub-modules. One sub-module includes a color (RGB) camera for capturing color planar images. The other sub-module includes a depth sensor. Exemplarily, the depth sensor can use infrared light as the emitted signal for distance measurement. In one example, the depth sensor can measure distance based on the Time of Flight (ToF) method. Since the speed of infrared light in air is known, the distance between the target and the 3D image sensor 103 can be calculated by combining the time from the emission of the infrared light to the time it takes to receive the echo signal reflected after the infrared light hits the target. In another example, the depth sensor can measure distance using structured light, which projects structured light with encoded information onto the surface of the target, thereby forming a light stripe pattern on the surface of the target. Subsequently, the image acquisition system acquires an image of the light stripe pattern to calculate the distance.

[0057] In other embodiments, the 3D image sensor 103 can also be a binocular camera with two calibrated RGB cameras. By acquiring two RGB images with overlapping portions, the coordinates in the image coordinate systems of the two RGB cameras can be unified to a spatial coordinate system using calibrated intrinsic and extrinsic parameters, thereby obtaining spatial position information. In some embodiments, the 3D image sensor 103 can also be a binocular structured light camera. The field of view of the 3D image sensor 103 covers the recognition area. Figure 1 In the example, the 3D image sensor 103 may be located above the desktop, facing downwards towards the desktop, with its field of view covering the recognition area.

[0058] It should be noted that, in Figure 1 In this configuration, the controller 104 is concealed because it is encapsulated within the device body 101. The controller 104 can be positioned in a suitable location within the device body 101 as needed, such as inside a box at the rear of the table 111, or under the table 111.

[0059] The controller 104 is communicatively connected to the 3D image sensor 103 to acquire target image data from the 3D image sensor 103. The content of the target image data may vary depending on the possible implementation of the motion-sensing camera or depth camera described in the previous examples. For example, for a 3D image sensor 103 based on a depth sensor in conjunction with an RGB camera, the target image data may include: a color image acquired by the RGB camera with at least partial pixel overlap, and depth information acquired by the depth sensor. For a 3D image sensor 103 based on binocular or more cameras, the target image data may include: multiple color images acquired from different viewpoints, each containing the trainee. For a binocular camera with a depth sensor, the target image data may include multiple color images (acquired by a binocular RGB camera) acquired from different viewpoints with at least partial pixel overlap, and depth information (acquired by the depth sensor).

[0060] Furthermore, the controller 104 can obtain precise positional information changes of key points related to the training area on the trainee's body during the execution of training movements in a non-contact manner by recognizing and analyzing target image data. This allows for accurate judgment of whether the training area has correctly performed the training movement or performed an inappropriate compensatory movement. Therefore, this solves the problem of inaccurate judgment in related technologies that rely on motor encoders or sensors to indirectly determine movement posture. It eliminates the need for wearable sensors, accurately detects compensatory movements, and can provide corresponding assistance to enhance the rehabilitation training effect.

[0061] Understandably, "compensatory movement" refers to a movement relative to a correctly executed movement. It describes a situation where, due to insufficient strength in the movement pattern, target joint, or target muscle, abnormal motor function occurs, leading to compensatory movement in adjacent non-target joints or muscles to achieve the intended movement goal. (See comparative text.) Figure 4 and Figure 5 visible, Figure 4 The image demonstrates the correct execution of elbow rotation. As the right elbow rotates to the left from its original position, the hand moves to the left to the guiding position, while the head and shoulders remain fixed. Figure 5 As can be seen, what is being shown is Figure 4 A compensatory movement for a pre-set action. Although the hand moves to... Figure 4 When in the same guiding position, but with deviations in both the shoulder and head, the elbow joint rotation becomes incomplete, significantly impacting the effectiveness of elbow joint rehabilitation training. Therefore, it is crucial to detect and address compensatory movements during rehabilitation training to ensure the desired results are achieved.

[0062] In some embodiments, exemplarily, upper limb rehabilitation training is used as an example. The training area is located in the upper limb of the human body, including at least one of the shoulder joint and elbow joint. The plurality of key points includes key points directly related to the training area, as well as other key points that cooperate with the key points to complete the movement. For example, the plurality of key points may include representation points of the hand, elbow joint, shoulder joint, and head, etc. During elbow rotation, the key points of the hand will swing with the elbow rotation. Alternatively, in other embodiments, if the training area is located in the lower limb of the human body, the plurality of key points may include, correspondingly, the foot, knee joint, hip joint, etc. In some embodiments, the position information may include position and posture information, the position information may be represented by spatial three-dimensional coordinates within the recognition area, and the posture information may include at least one of the following: direction angle, velocity, acceleration, etc.

[0063] In some embodiments, key point recognition and pose recognition of the human body based on target image data can also be performed by the 3D image sensor 103. The 3D image sensor 103 can integrate a processor for performing the above recognition, thereby directly identifying and analyzing key points on the patient's body and their location information based on the captured target image data. Furthermore, using this location information, compensatory movements occurring in the patient can be accurately detected, allowing for timely intervention.

[0064] In some embodiments, the human keypoint recognition based on target image data can be achieved using a human pose estimation keypoint detection algorithm based on deep neural networks (RTMPOSE). RTMPOSE is a real-time multi-person keypoint detection algorithm for human pose estimation. By analyzing human poses in videos or images, it can accurately obtain the positional information of human keypoints in real time, thereby achieving the recognition and analysis of human poses. The principle of RTMPOSE is based on deep learning and computer vision technology. It employs a Convolutional Neural Network (CNN) model for human keypoint detection. The CNN learns features from the original image and extracts image feature information through multi-layer convolution and pooling operations. After constructing the initial network framework, the network is trained using videos or images containing humans as training data. The network parameters are adjusted by calculating the positional loss between predicted keypoints and actual keypoints until convergence yields a usable human keypoint detection model.

[0065] An exemplary implementation of keypoint recognition and keypoint location information acquisition is provided. Based on target image data carrying depth information acquired by a 3D image sensor 103, the spatial coordinates of each point can be calculated using image coordinate transformation. Furthermore, by combining the RTMPOSE model, the spatial coordinates of multiple keypoints on the patient's body in the target image data can be identified. Each identified keypoint is categorized into a corresponding body part label, such as head, hand, elbow, etc. By connecting these joints, a skeletal diagram of the trainee's body can be obtained. As an example, each identified keypoint can be assigned a unique ID for differentiation and can be used to index corresponding labels, location information, etc. Further, in the time dimension, by identifying the changes in the spatial coordinates of the same keypoint in two consecutive image frames, and combining this with the time difference of the timestamps between the two frames, the temporal sequence information of the keypoint's position and motion in the two frames can be calculated. This allows the determination of the pose of the keypoint in the subsequent frame, including motion direction, motion speed, and even acceleration parameters.

[0066] Therefore, the controller 104 and the 3D image sensor 103 can form a "body sensing system" which can be used to detect the position, movement, posture and other information of key points of the trainee in the recognition area, and can realize corresponding action guidance, response and control.

[0067] In some embodiments, the exercise training device 100 may further include a display screen 105, communicatively connected to the controller 104, and controlled by the controller 104 to display images related to rehabilitation training, including guiding patient movements to reach a designated position, etc. Exemplarily, the display screen 105 may be a display screen made of materials such as LCD, LED, or OLED. The display screen 105 may be a flat screen or a curved screen. Exemplarily, the display screen 105 may be as follows: Figure 1 As described above, the 3D image sensor 103 is erected at the rear end of the table body 111 and can be located on the top of the display screen 105.

[0068] In some embodiments, a game animation corresponding to the rehabilitation training can be displayed on the monitor to increase the fun of the training. In the game, trainees can score points by performing specified or unspecified preset movements. The score for correctly and properly performing the preset movements will be higher than the score for compensatory movements, thereby encouraging trainees to perform the preset movements as correctly as possible and avoid compensatory movements that are detrimental to rehabilitation.

[0069] like Figure 2 The diagram shown illustrates the module structure of the controller in one embodiment of this disclosure.

[0070] The controller 200 includes a bus 201, a processor 202, and a memory 203. The processor 202 and the memory 203 can communicate with each other via the bus 201. The memory 203 can store program instructions. The processor 202 implements the program instructions stored in the memory 203. Figure 1 The controller in this embodiment includes the steps of the motion guidance method shown in the following embodiments.

[0071] Bus 201 can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of representation, although only one thick line is used in the diagram, this does not indicate that there is only one bus or one type of bus.

[0072] In some embodiments, processor 202 may be implemented as a central processing unit (CPU), microprocessor unit (MCU), system on chip (System on Chip), or field-programmable array (FPGA). Memory 203 may include volatile memory for temporary data storage during program execution, such as random access memory (RAM).

[0073] The memory 203 may also include non-volatile memory for data storage, such as read-only memory (ROM), flash memory, hard disk drive (HDD), or solid-state disk (SSD).

[0074] In some embodiments, the controller 200 may further include a communicator 204. The communicator 204 is used for communication with external devices. In specific examples, the communicator 204 may include one or more wired and / or wireless communication circuit modules. For example, the communicator 204 may include one or more of, such as a wired network card, a USB module, a serial interface module, etc. The wireless communication protocols followed by the wireless communication module include, for example, Near Field Communication (NFC) technology, Infrared (IR) technology, Global System for Mobile Communications (GSM), General Packet Radio Service (GPRS), Code Division Multiple Access (CDMA), Wideband Code Division Multiple Access (WCDMA), Time-Division Code Division Multiple Access (TD-SCDMA), Long Term Evolution (LTE), Bluetooth (BT), Global Navigation Satellite System (GNSS), etc.

[0075] like Figure 3 The diagram shown illustrates a flowchart of a motion guidance method according to an embodiment of this disclosure.

[0076] The motion guidance method can be as follows: Figure 1 , Figure 2 The controller in the sports training equipment executes program instructions.

[0077] Specifically Figure 3 The motion guidance method specifically includes:

[0078] Step S301: Based on the recognition of target image data captured from the trainee, obtain the location information of multiple key points on the trainee's body related to the training area.

[0079] The trainees can be patients undergoing rehabilitation training or other trainee groups. Since the principles of key point recognition and location information acquisition have been exemplarily given in previous embodiments, they will not be repeated here.

[0080] according to Figure 4 , Figure 5 It is understood that the aforementioned multiple key points actually include: fixed key points defined as fixed in position during the preset action, and motion key points where movement occurs. Taking the upper limb as an example, with the elbow joint as the training area, the fixed key points include the representation points of the elbow joint, shoulder joint, and head, while the motion key points include the representation point of the hand. If, for example, a situation arises... Figure 5 If at least one fixed key point in the data experiences an offset event (that should be fixed but has moved), as shown in the figure, then the detection of whether a "compensatory action" has occurred can be performed based on this.

[0081] Step S302: Determine the guiding position that the key movement point needs to reach in performing the preset action, and guide the trainee according to the guiding position.

[0082] In some embodiments, the preset movements can be pre-stored in the controller. The preset movements can generate corresponding guiding positions based on the purpose of rehabilitation training and the body part to be trained (such as the elbow joint, shoulder joint, etc. in the upper limb). For example, the preset movements can mimic actions of the body part to be trained in a real-world scenario, such as pouring water or swimming. Since the range of motion for each key movement point is basically determined by the preset movements, the guiding positions, including the spatial coordinates of the guiding positions within the recognition area, can be obtained accordingly. For example, if the preset movement trains the elbow joint and requires the right hand to rotate 30° to the left, then the spatial coordinates of the guiding position can be obtained by rotating the elbow joint 30° to the left based on the current spatial coordinates of the hand.

[0083] After obtaining the guided position, guidance can be provided to the trainee. In some embodiments, guidance information can be output to prompt the trainee to perform actions to bring the key movement points to the guided position. As an example, the guidance information includes one of the following: voice, image, text, or assistance from the interactive motion mechanism. For example, by setting up a speaker connected to the controller for communication to play voice instructions about the trainee's right hand moving up, down, left, right, or rotating. For example, by displaying on a monitor, for example... Figure 4 Images can be used to indicate the position the trainee's right hand needs to move to. For example, a text message can be displayed on a monitor to show the trainee's instructions for moving or rotating their right hand up, down, left, or right. Alternatively, a motor-driven interactive motion mechanism can apply force in the direction of the guide position to move or guide the trainee's right hand to that position.

[0084] Step S303: Based on the target image dataset collected in sequence during the trainee's execution of preset actions, obtain the position and motion time sequence information of the multiple key points.

[0085] In some embodiments, when a trainee is performing a preset action, the positional changes of each key point can be captured by a 3D image sensor, and as described in the previous embodiments, posture changes (such as direction, velocity, acceleration, etc.) can be calculated by the trajectory changes of key points between different frames. The target image dataset includes target image data arranged in time sequence. For example, the RGB image and depth information of each key point at each time step.

[0086] Step S304: In response to the position movement timing information indicating that at least one of the fixed key points has shifted, determine that the trainee used a compensatory action when performing the preset action.

[0087] by Figure 4 and Figure 5 As can be seen from the example, in Figure 5 In the process, when the head, shoulders, and elbows, which should not be moving, are found to be deviating, it can be determined that a compensatory action has been performed.

[0088] More specifically, to improve the accuracy of compensatory action judgment, relevant offset conditions can be set. These offset conditions can be related to the rotation angle and / or translation distance of a fixed key point, and can also be combined with the duration of the offset state.

[0089] In some examples, the trainee can be determined to perform a compensatory action when the rotation angle of at least one of the fixed key points reaches a preset angle threshold. Figure 4 For example, when the rotation angle of either the head or shoulder exceeds a preset angle threshold, the trainee is considered to be performing a compensatory action. Alternatively, when both the head and shoulders rotate beyond the preset angle threshold, the trainee is considered to be performing a compensatory action. It should be noted that the preset angle thresholds for head and shoulder rotation can be the same or different, but can at least be set to be greater than 0°.

[0090] In some examples, the trainee can be determined to perform a compensatory action when the rotation angle of at least one of the fixed key points reaches a preset angle threshold and continues for a preset duration. Figure 4 For example, when the rotation angle of either the head or shoulder exceeds a preset angle threshold and remains so for n seconds (n>0, e.g., 1, 2, 3 seconds), the trainee is considered to have performed a compensatory action. Alternatively, when both the head and shoulders rotate more than the preset angle threshold and remain so for n seconds, the trainee is considered to have performed a compensatory action.

[0091] like Figure 6 The diagram shown illustrates an exemplary judgment logic for a compensatory action in one embodiment of this disclosure.

[0092] exist Figure 6The paper presents a judgment logic for determining compensatory actions. Specifically, in this judgment logic, a status identifier can be provided to the trainee to indicate whether the trainee's status is "compensatory state" or "normal state". If the trainee performs a compensatory action, the identifier value can be set to "compensatory state", such as 0; if no compensatory state occurs, the identifier value representing "normal state" can be set, such as "1". Figure 6 In the logical flow, the rotation angle of the key point is used as an example to determine whether compensation is required.

[0093] exist Figure 6 The process includes:

[0094] Step S601: Obtain the trainee's angle information at the current moment.

[0095] Step S602: Calculate the absolute value of the angle difference between the current time and the previous time.

[0096] Step S603: Determine whether the absolute value of the angle difference is greater than the preset angle threshold.

[0097] If yes, proceed to step S604; otherwise, end the judgment process at the current moment.

[0098] Step S604: Calculate whether the absolute value of the angle difference is greater than the duration of the preset angle threshold.

[0099] For example, a timer can be used to keep track of time.

[0100] Step S605: Determine whether the calculated duration has reached the preset duration.

[0101] If yes, proceed to step S606; if no, proceed to step S607.

[0102] Step S606: Determine the user's status as "compensation status".

[0103] Step S607: Reset the timer.

[0104] Step S608: Determine the user's status as "normal".

[0105] A specific calculation example is provided. During the trainee's performance of an action, the controller continuously acquires the position information of each key point, including the current angle, obtained from the target image data captured by the 3D image sensor, and then calculates the angle of the fixed key point. and the angle of key points of movement (i is the joint number, t is the t-th time). The angle of the fixed keypoint at the next time step (t+1) is calculated. Angle relative to the current time (t) at a fixed key point The absolute value of the difference ,Right now .when Duration At that time, the user is determined to be in a compensation status. >0. Afterwards, when Duration When the time is right, the determined user status can be switched to normal status.

[0106] In other embodiments, in addition to the rotation angle, displacement can also be used as a condition for judging the compensatory action, or displacement combined with the duration of maintenance can be used as a condition for judging the compensatory action.

[0107] In some examples, the trainee is determined to perform a compensatory action when the displacement distance of at least one of the fixed key points reaches a preset displacement threshold. Alternatively, the trainee is determined to perform a compensatory action when the displacement distance of at least one of the fixed key points reaches the preset displacement threshold and continues for a preset duration.

[0108] It's important to note that judging compensatory movements based on rotation angle reduces interference, such as external force interference, compared to methods that rely on displacement. For example, when a trainee performs a preset movement, if external force causes translation, and the elbow and shoulder do not rotate or rotate below a preset angle threshold, yet the hand still correctly completes the required movement, judging based on rotation angle would correctly indicate that the trainee did not perform a compensatory movement. However, judging based on displacement would lead to the incorrect conclusion that the trainee performed a compensatory movement. Therefore, judging compensatory movements based on rotation angle is superior to judging based on displacement.

[0109] To make the rehabilitation training process more flexible and free for patients, in some embodiments, the exercise training device can also acquire the trainee's movement intention information of the affected limb through various detection methods, thereby setting a guidance position that matches the trainee's movement intention to meet the trainee's needs and improve the user experience. In some embodiments, the movement intention can be passively captured by the system or actively expressed by the trainee. In some embodiments, the trainee's movement intention can be acquired by running a training mode with predetermined movements assigned to the trainee, or it can be acquired in a training mode where the trainee moves freely according to existing preset movements. After acquiring the movement intention, movement guidance that matches the movement intention can be provided. The method, scenario, and mode of acquiring the movement intention can be very flexible, effectively improving the single and inflexible method of random or fixed movement guidance positions in related technologies, and enhancing the user experience.

[0110] In some embodiments, the first positional motion timing information of each key point of the trainee's healthy limb and the second positional motion timing information of each key point of the affected limb can be obtained based on a target image dataset acquired in a time sequence. The contralateral movement that coordinates with or is symmetrical to the movement of the healthy limb is determined as the target pose of the affected limb, and the guiding position of the motion key points on the affected limb is determined based on the target pose. Specifically, the exercise training device can provide a pre-specified preset movement guidance mode, which trains the trainee according to the pre-specified training movements. This mode can also be called a "daily movement mode." In this mode, the 3D image sensor continuously captures the positional motion timing information of the trainee's healthy and affected limbs, and then uses the movement of the healthy limb at the current moment as a reference to determine the guiding position of the next movement of the affected limb. For example, if the preset movement is pouring water from a cup, the left arm is the healthy limb, and the right arm is the affected limb. The right arm imitates pouring water from a teapot, and the left arm imitates catching water in a cup; therefore, the position of the right hand must coordinate with the left hand. For example, if the left and right arms mimic swimming movements, say in the breaststroke, then the movements of the left and right arms are symmetrical.

[0111] As an example, you can refer to Figure 7 As shown, when the trainee performs a designated action (symmetrical action such as swimming or skiing, or ADL action involving both sides such as pouring water or opening a door) with both upper limbs, a 3D image sensor acquires the three-dimensional information of the healthy side in spatial coordinates to determine the trainee's movement intention within the current movement sequence. The position of the healthy side indicates the temporal position of the current action, thereby determining the guiding position for the affected side's current movement intention. Based on the guiding position of the affected side's movement intention, the affected limb is then guided.

[0112] exist Figure 7 In this training, the healthy and affected limbs perform symmetrical movements. The position of the affected (right) limb is determined by the healthy (left) limb reaching the desired position. Specifically, the symmetrical position of the healthy limb's hand can be selected as the guiding position (indicated by the dotted line) for the affected right hand, thus guiding the trainee to move their right hand to that guiding position in the next moment. As described in previous embodiments, the guidance can be provided through voice, images, text, or interactive motion mechanisms. This allows the movement of the healthy limb to determine the "movement intention," thereby inducing movement of the affected limb.

[0113] In some embodiments, the trainee's intention can also be passively captured during free movement training, such as in "free movement mode". For example, according to the above scheme, a preset movement can be matched based on the position reached by the unaffected limbs through free movement. After matching the preset movement, the trainee's movement intention can be obtained. The guiding position can be determined by referring to the preset movement position and the corresponding or symmetrical target position on the opposite side.

[0114] In some embodiments, preset movements can be predicted based on the movement of key points on the healthy side of the limb, and then the movements of the affected side of the limb can be predicted based on the predicted movements of the healthy side of the limb.

[0115] As an example, a first preset movement of the healthy limb can be predicted based on the movement timing information of the first position. Specifically, for example, the posture of the healthy limb at the next moment can be obtained by analyzing the trajectory of the pose changes of various key points in the previous few moments. The poses of various key points at the next moment can be combined and matched with the first preset movement in a pre-stored preset movement library. This expresses the trainee's movement intention. A second preset movement on the opposite side that matches or is symmetrical to the first preset movement posture can then be obtained. For example, in a water-pouring scenario, the action of filling a cup matches the action of pouring water from a teapot, or in a breaststroke scenario, the movements of both limbs are symmetrical. Therefore, by using the second predicted movement as the target movement of the affected limb, the guiding position of the affected limb at the next moment can be determined according to the requirements of the second preset movement.

[0116] In another embodiment, the guiding position that matches the trainee's movement intention can also be determined based on the intention actively expressed by the trainee. For example, the guiding position that matches the movement intention can be determined and guidance formed in response to instruction information representing the trainee's movement intention on the affected side of their limb. As an example, the instruction information representing the trainee's movement intention can be obtained from biometric information output by the trainee. Further exemplarily, the biometric information includes, but is not limited to, one or more of, such as eye movements, gestures, and partial or overall body postures (e.g., head shaking, nodding, heart-shaped hand gestures). The guiding position that matches the movement intention is determined based on the movement intention corresponding to the biometric information, and guidance is formed. For example, if the trainee's eyes turn to the left, the eye direction is captured by a 3D image sensor, and the controller can determine the direction of the gaze based on the eye direction, extending the gaze to the intersection with the table as the guiding position.

[0117] When a user moves their upper limbs within the rehabilitation robot's activity space using the end effector, the motion-sensing camera detects whether the user engages in compensatory movements. In this free movement mode, it determines if the user intends to move but is unable to move the end effector. By detecting compensatory movements based on the user's intention, the system identifies the user's movement intent and provides assistance.

[0118] It should be noted that compensatory movements indicate that the trainee attempts a preset movement but fails to complete it due to insufficient motor ability. Therefore, in some embodiments, the trainee's movement intention can be captured based on the compensatory movements of the affected limb to determine the guiding position.

[0119] Specifically, in response to detecting that the trainee is using a compensatory movement, a guiding position that conforms to the movement intention can be determined based on the expected movement corresponding to the compensatory movement, and guidance can be formed. The expected movement includes at least one of the following: a preset movement; a reference movement of the unaffected limb.

[0120] As an example, in free training mode, the preset action corresponding to the compensatory movement can be predicted directly based on the recorded second-position motion sequence information of each key point on the affected limb. For instance, if the affected right hand makes a paddling motion but cannot complete the compensatory movement, the predicted paddling motion can be used as the next action to be performed by the affected right hand, and guiding positions can be set for its key points accordingly. In daily training mode, since the previously specified preset action is known, if a compensatory movement is detected, the guiding positions corresponding to the preset action can be repeatedly specified.

[0121] As another example, the guiding position can also be set by referring to the reference movement of the contralateral healthy limb. In free training mode, the preset movement of the first healthy limb can be used as a reference movement to obtain the target pose for contralateral coordination or symmetry. Therefore, when a compensatory movement on the affected side is detected, the guiding position of the affected limb is set according to the target pose on the contralateral side corresponding to the reference movement. Alternatively, as... Figure 7 As shown, the healthy limb reaches the current position, which can be understood as the trainee's expected position and is used as the movement intention. The affected limb reaches the current position only after making a compensatory movement, which is different from the expected symmetrical position of the healthy limb represented by the dotted line. Therefore, the position of the dotted line can be set as the guiding position.

[0122] In some embodiments, when a trainee exhibits compensatory movements and / or insufficient motor ability, the interactive motion mechanism (e.g., an end effector or exoskeleton) can apply reverse force to assist the trainee, providing guidance and support. Specifically, when it is detected that the trainee is using compensatory movements and / or meets the condition of insufficient movement completion ability, the interactive motion mechanism is controlled to move the trainee's affected limb to guide the key movement point to the guided position.

[0123] In some embodiments, the insufficient ability to complete the action may include one of the following: for example, the static state of the interactive motion mechanism lasts for a preset duration; for example, the movement distance of the interactive motion mechanism within the preset duration is less than a preset distance threshold; for example, the movement speed of the interactive motion mechanism is lower than a preset speed threshold. Specifically, when the interactive motion mechanism held by the trainee is stationary for a long time, moves a short distance within a certain period of time, or moves at a very slow speed, it indicates that the trainee may be unable to move the interactive motion mechanism or finds it very difficult to move it. Therefore, the active movement of the interactive motion mechanism can assist the trainee.

[0124] Since the exercise training equipment can be equipped with audio-visual devices such as displays and speakers, it can interact with the trainee through multimedia information exchange in the form of images, voice, and text. It can also interact with the trainee through the interactive motion mechanism to exchange information about the application of force. In addition to the methods mentioned in previous embodiments, these interactive methods can guide the trainee's movements. To enhance engagement, in the multimedia information interaction method, the display screen can show the movement of virtual objects in a mapped image that is correlated with the actions performed by the trainee. For example, the mapped image can display a flat screen corresponding to a desktop, showing virtual limbs moving synchronously with the trainee's limbs, with the range of motion of the virtual limbs related to that of the actual limbs. Alternatively, a game program corresponding to the training process can be pre-configured. The controller can run the game program during training to display game animations as mapped images, and the trainee's limb movements are related to the movements of virtual targets in the mapped images. For example, when the trainee swings their affected limb, a virtual character in the mapped image punches a target, or a virtual character jumps to a target, etc. The punching and jumping amplitudes can be related to the range of motion of the affected limb.

[0125] Considering that 3D image sensors may exhibit errors, such as inaccurate calibration parameters due to movement, this can result in errors in the position and motion parameters of key points calculated based on target image data. Therefore, in some embodiments, the drive stroke data of the drive motor of the interactive motion mechanism can be used to calibrate these errors.

[0126] like Figure 8 The diagram illustrates a process for calibrating key point location information identified from target image data in one embodiment of this disclosure.

[0127] exist Figure 8 The process specifically includes:

[0128] Step S801: Estimate the first current position of the interactive motion mechanism based on the drive stroke data of the interactive motion mechanism.

[0129] In some embodiments, the drive stroke data may include encoder data of the drive motor of the interactive motion mechanism, the encoder data indicating the rotational stroke of the drive motor corresponding to the motion stroke of the interactive motion mechanism. Based on the initial position of the interactive motion mechanism combined with the stroke indicated by the encoder data, the first current position of the terminal effector can be calculated.

[0130] Step S802: Determine the second current position of the interactive motion mechanism based on the currently captured target image data.

[0131] The principle of determining the spatial coordinates of each point in an image by performing coordinate system transformation calculations to obtain, for example, the position of a terminal effector, has been described in previous embodiments and will not be repeated here.

[0132] Step S803: Compare whether the deviation between the first current position and the second current position is large enough to meet the preset deviation condition.

[0133] In step S803, it is determined whether the deviation is large. If it is only a small deviation within the allowable range, calibration is not required.

[0134] If yes, proceed to step S804; if no, proceed to step S805.

[0135] Step S804: Correct the first current position of the interactive motion mechanism to the second current position.

[0136] That is, the first position coordinates based on image recognition can be replaced by the second position coordinates obtained from the encoder data of the drive motor.

[0137] Step S806: Based on the second current position, correct the positions of the multiple key points of the trainee identified in the target image data at the current and / or previous times.

[0138] Specifically, after the correction, the second current position of the interactive motion mechanism is the position of the trainee's affected limb hand. Accordingly, for other key points, the positional changes from the "first current position to the second current position" also need to be corrected.

[0139] Step S805: End the process.

[0140] You can refer to this. Figure 9 As shown, this is illustrated in a specific embodiment. Figure 8 A simplified diagram illustrating the principle of the process flow. Figure 9 The example demonstrates an interactive motion mechanism as an end effector, with a 3D image sensor and controller constituting a motion sensing system.

[0141] exist Figure 9 In this process, based on the current encoder reading of the drive motor and the structural parameters of the exercise training equipment, forward kinematics calculations are performed to determine the first current position of the terminal effector. Furthermore, the second current position of the terminal effector can be calculated using a 3D image sensor (or controller). If there is a significant error between the first and second current positions, the end position of the terminal effector is corrected based on the encoder data, and the positions of the remaining key points on the affected upper limb (excluding the hand) are corrected by combining the data from the previous frame (including the positions of other key points).

[0142] like Figure 10The diagram shows a schematic of a motion guidance device in one embodiment of this disclosure. The motion guidance device 1000 is applied to a sports training device, which includes an interactive motion mechanism capable of force interaction and movement with at least the affected limb of the trainee. It should be noted that the principle and technical implementation of the motion guidance device 1000 can refer to the motion guidance methods in previous embodiments, therefore, they will not be repeated in this embodiment.

[0143] The motion guiding device 1000 includes:

[0144] The key point recognition module 1001 is used to obtain the position information of multiple key points on the trainee’s body related to the training part based on the recognition of target image data captured from the trainee; wherein, the multiple key points include: fixed key points that are defined as fixed in position in preset actions, and motion key points that are in motion.

[0145] The guidance position generation module 1002 is used to determine the guidance position that the motion key point needs to reach in performing the preset action, and to guide the trainee according to the guidance position.

[0146] The motion detection module 1003 is used to obtain the position and motion time sequence information of the multiple key points based on the target image dataset collected in sequence during the training of the trainee performing the preset action;

[0147] The compensation detection module 1004 is used to determine that the trainee uses a compensation action when performing the preset action in response to the position movement timing information indicating that at least one of the fixed key points has shifted.

[0148] It should be noted that, in Figure 10 The various functional modules in the embodiments can be implemented, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, they can be implemented, in whole or in part, in the form of a program instruction product. A program instruction product includes one or a set of program instructions. When the program instructions are loaded and executed on a computer, all or part of the flow or function according to this disclosure is generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The program instructions can be stored in a computer-readable storage medium or transferred from one computer-readable storage medium to another.

[0149] and, Figure 10The apparatus disclosed in the embodiments can be implemented through other modular division methods. The apparatus embodiments shown above are merely illustrative. For example, the module division is only a logical functional division, and in actual implementation, there may be other division methods. For example, a group of modules or modules may be combined or dynamically integrated into another system, or some features may be ignored or not executed. Furthermore, the shown or discussed mutual coupling, direct coupling, or communication connection may be through some interfaces, and the indirect coupling or communication connection between devices or modules may be electrical or other forms.

[0150] in addition, Figure 10 The functional modules and sub-modules in the embodiments can be dynamically integrated within a single processing unit, or each module can exist physically independently, or two or more modules can be dynamically integrated within a single unit. These dynamic units can be implemented in hardware or as software functional modules. If these dynamic units are implemented as software functional modules and sold or used as independent products, they can also be stored in a computer-readable storage medium. This storage medium can be a read-only memory, a hard disk, or an optical disk, etc.

[0151] It should be specifically noted that the flowchart representations of the embodiments described above in this disclosure can be understood as representing modules, segments, or portions of code comprising one or more sets of executable instructions configured to implement specific logical functions or processes. Furthermore, the scope of the preferred embodiments of this disclosure includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functions involved.

[0152] For example, Figure 3 The order of the steps in the method embodiment may vary in specific scenarios and is not limited to the above representation.

[0153] This disclosure also provides a computer program product, including: program instructions for executing the motion guidance method as described. For example, executing... Figure 3 The steps in the motion guidance method.

[0154] This disclosure also provides a computer-readable storage medium storing program instructions that, when executed, implement the motion guidance method of any of the previous embodiments.

[0155] That is, the method steps in the above embodiments are implemented as software or computer code that can be stored in a recording medium (such as CD ROM, RAM, floppy disk, hard disk or magneto-optical disk), or implemented as computer code that is originally stored in a remote recording medium or a non-transitory machine-readable medium and will be stored in a local recording medium after being downloaded via a network, so that the method represented herein can be stored in such software processing on a recording medium using a general-purpose computer, a special processor or programmable or special hardware (such as ASIC or FPGA).

[0156] In summary, the embodiments of this disclosure provide a motion guidance method, controller, training device, program product, and medium. The method includes: identifying the positional information of multiple key points on the trainee's body related to the training area based on the recognition of target image data captured from the trainee; determining the guidance position that the motion key points need to reach in performing a preset action, and guiding the trainee according to the guidance position; obtaining the positional motion temporal information of multiple key points based on the target image dataset collected sequentially during the trainee's performance of the preset action; and determining that the trainee uses a compensatory action when performing the preset action in response to a displacement event indicated by the positional motion temporal information at least one fixed key point. This disclosure is based on image recognition of the positional information of the trainee's body key points and the motion process, accurately determines the trainee's motion state, provides accurate guidance positions for correct guidance, and can detect undesirable compensatory actions, thereby improving the rehabilitation training effect.

[0157] Therefore, on the one hand, when faced with complex symptoms, it can also overcome the limitations of motor encoders and sensors, accurately record the movement trajectory of the user's upper limbs in all aspects, and provide feedback to the therapist and the user.

[0158] On the other hand, because it uses non-contact detection based on image recognition, it can facilitate and speed up rehabilitation training, take into account patients with mobility impairments, avoid the inconvenience of wearable detection, and effectively and accurately record data during exercise.

[0159] On the other hand, this disclosed solution can eliminate the uncertainty in estimating the user's joint movements caused by current terminal or exoskeleton rehabilitation robots, and meet the data recording (image recording) requirements of patients' rehabilitation exercises in the visual dimension. Based on this, it can effectively improve the user's correct training posture and provide patients and therapists with multi-dimensional recording and analysis channels.

[0160] The above embodiments are merely illustrative of the principles and effects of this disclosure and are not intended to limit this disclosure. Any person skilled in the art can modify or alter the above embodiments without departing from the spirit and scope of this disclosure. Therefore, all equivalent modifications or alterations made by those skilled in the art without departing from the spirit and technical concept disclosed in this disclosure should still be covered by the protection scope of this disclosure.

Claims

1. A controller, characterized in that, An interactive motion mechanism is used in sports training equipment, the sports training equipment including a movable interactive motion mechanism capable of force interaction with at least the affected limb of the trainee; the controller includes: Processor and memory; The memory stores program instructions; The processor is configured to execute the program instructions to perform a motion guidance method, the motion guidance method comprising: Based on the recognition of target image data captured from the trainee, the positional information of multiple key points on the trainee's body related to the training part is obtained; wherein, the multiple key points include: fixed key points that are defined as fixed in position in the preset action, and motion key points that are in motion. Determine the guiding position that the key points of the movement need to reach in the execution of the preset action, and guide the trainee according to the guiding position; Based on the target image dataset collected sequentially during the trainee's execution of preset actions, the position and motion temporal information of the multiple key points is obtained. In response to the positional motion timing information indicating that at least one of the fixed key points has shifted, it is determined that the trainee used a compensatory action when performing the preset action.

2. The controller according to claim 1, characterized in that, The method of determining a compensatory action for the trainee to perform the preset action in response to the positional motion timing information indicating that at least one of the fixed key points has shifted includes at least one of the following: When the rotation angle of at least one of the fixed key points reaches a preset angle threshold, it is determined that the trainee performs a compensatory action; When the rotation angle of at least one of the fixed key points reaches a preset angle threshold and continues for a preset duration, it is determined that the trainee performs a compensatory action.

3. The controller according to claim 1, characterized in that, The motion guidance method further includes: The first current position of the interactive motion mechanism is estimated based on the drive stroke data of the interactive motion mechanism; The second current position of the interactive motion mechanism is determined based on the currently captured target image data; Compare the deviation between the first current position and the second current position; If the comparison result indicates that the deviation is large enough to meet the preset deviation condition, then the current position of the interactive motion mechanism is corrected to the second current position; Based on the second current position, the positions of the multiple key points of the trainee identified in the target image data at the current and / or previous times are corrected.

4. The controller according to claim 1, characterized in that, The process of determining the guiding position that the key movement points need to reach during the execution of the preset action, and guiding the trainee according to the guiding position, includes: To obtain information on the trainee's motor intentions in the affected limb; Determine a guiding position that conforms to the stated motion intention, and form a guide.

5. The controller according to claim 4, characterized in that, The determination of the guiding position that conforms to the intended movement, and the formation of the guide, includes at least one of the following: 1) Based on the target image dataset acquired in a time sequence, obtain the first position movement time sequence information of each key point of the healthy side limb of the trainee, and the second position movement time sequence information of each key point of the affected side limb paired with the healthy side limb. Then, determine the contralateral movement that is coordinated with or symmetrical to the movement of the healthy side limb as the target pose of the affected side limb, and determine the guiding position of the movement key points on the affected side limb according to the target pose. 2) Based on the target image dataset acquired in a time sequence, obtain the first position movement time sequence information of each key point of the trainee's healthy limb and the second position movement time sequence information of each key point of the affected limb paired with the healthy limb; predict the first preset movement of the healthy limb based on the first position movement time sequence information, and determine the second preset movement of the opposite side that is in symmetry or positional coordination with the first preset movement; determine the guiding position of the affected limb that conforms to the trainee's movement intention based on the second preset movement. 3) In response to instruction information indicating the trainee's intention to move the affected limb, determine a guiding position that conforms to the intention to move and form a guide; 4) In response to detecting that the trainee is using a compensatory movement, determine a guiding position that conforms to the movement intention based on the expected movement corresponding to the compensatory movement, and form a guide; the expected movement includes at least one of the following: a preset movement; a reference movement of the healthy limb.

6. The controller according to claim 5, characterized in that, In response to instruction information indicating the trainee's intention to move the affected limb, a guiding position consistent with the intended movement is determined, and guidance is formed, including: In response to the biometric information output by the trainee, the movement intention corresponding to the biometric information is determined, a guiding position that conforms to the movement intention is determined, and guidance is formed.

7. The controller according to claim 1, 5, or 6, characterized in that, The processor is further configured to, in response to detecting that the trainee is using compensatory movements and / or meets the condition of insufficient movement completion ability, control the interactive motion mechanism to move the trainee's affected limb to the guide position so that the key movement point reaches the guide position.

8. The controller according to claim 7, characterized in that, The conditions for insufficient action completion capability include at least one of the following: the static state of the interactive motion mechanism lasts for a preset duration; the movement distance of the interactive motion mechanism within the preset duration is less than a preset distance threshold; the movement speed of the interactive motion mechanism is lower than a preset speed threshold.

9. The controller according to claim 1, characterized in that, The method of guiding the trainee according to the guidance position includes: outputting guidance information prompting the trainee to perform an action so that the key points of movement reach the guidance position, the guidance information including one of the following: voice, image, text or assistance from the interactive motion mechanism; and / or, the method further includes: displaying a mapping image on a display, wherein the movement of a virtual object in the mapping image is correlated with the action performed by the trainee.

10. A sports training device, comprising: The device itself forms a recognition area; An interactive motion mechanism is movably disposed in the recognition area; A 3D image sensor, disposed on the device body, is used to capture target image data; the field of view of the 3D image sensor covers the recognition area; The controller as described in any one of claims 1 to 9 is communicatively connected to the 3D image sensor.

11. The sports training equipment according to claim 10, characterized in that, The device body includes: a height-adjustable table, the tabletop of which forms the recognition area; the interactive motion mechanism is disposed on the tabletop; and / or, the motion training device further includes: a display, disposed on the device body and communicatively connected to the controller; and / or, the 3D image sensor is disposed above the display.

12. A computer program product, characterized in that, include: The program instructions in the memory of the controller as described in any one of claims 1 to 9.

13. A computer-readable storage medium, characterized in that, The memory stores program instructions in the controller as described in any one of claims 1 to 9.

Images