Acupuncture point targeted dynamic stimulation device

Abstract

The invention provides an acupoint targeted dynamic stimulation device, and relates to the technical field of medical rehabilitation. The device comprises an acupoint stimulation assembly, an action monitoring assembly, a display device and a controller. The acupoint stimulation assembly comprises a fixing assembly and a bionic stimulation assembly; the action monitoring assembly comprises a video monitoring assembly and a wearable monitoring assembly which are respectively used for shooting action videos of a user and monitoring position signals of a plurality of parts of the user; the controller is used for playing a preset action video; generating a first control instruction; obtaining a plurality of preset video clips; determining an action monitoring index, and generating a second control instruction; displaying the action monitoring index, and displaying a prompt message about whether to continue the next stimulation period; and continuing the next stimulation period or stopping stimulation. According to the invention, the action monitoring index can be determined in real time based on the video and the position signal, the accuracy of evaluating the standard degree of the action of the user is improved, and the effect of acupoint stimulation is improved.

Description

Technical Field

[0001] This invention relates to the field of medical rehabilitation technology, and in particular to a dynamic stimulation device for acupoint targeting. Background Technology

[0002] Currently, acupoint stimulation therapy is widely used in rehabilitation medicine, sports training, and daily health care. Common acupoint stimulation methods include manual pressure, electrical stimulation, and hot compresses. These techniques typically employ static or fixed stimulation patterns, leading to a disconnect between the stimulation effect and the user's movements. It is difficult to assess the user's movement status in real time by considering movement trends, and it is also difficult to dynamically adjust the stimulation method, resulting in poor acupoint stimulation effects.

[0003] The information disclosed in the background section of this application is intended only to enhance the understanding of the general background of this application and should not be construed as an admission or in any way implying that the information constitutes prior art known to those skilled in the art. Summary of the Invention

[0004] This invention provides a dynamic stimulation device for acupoint targeting, which can solve the technical problems of related technologies that make it difficult to evaluate the user's movement status in real time by combining movement trends, and make it difficult to dynamically adjust the stimulation method.

[0005] According to a first aspect of the present invention, an acupoint-targeted dynamic stimulation device is provided, comprising: an acupoint stimulation component, a motion monitoring component, a display device, and a controller; the acupoint stimulation component includes a fixing component and a bionic stimulation component, wherein the fixing component is used to fix the bionic stimulation component at a preset acupoint location, and the bionic stimulation component is used to output pressure stimulation according to control commands from the controller; the motion monitoring component includes a video monitoring component and a wearable monitoring component, the video monitoring component is used to capture a video of the user's motion, and the wearable monitoring component is used to monitor position signals of multiple body parts of the user; the controller is used to: play a preset motion video through the display device at the start of a stimulation cycle; and, according to the preset acupoint stimulation component, play a video of the user's motion at the start of a stimulation cycle; and, according to the preset acupoint stimulation component, play a video of the user's motion at the start of a stimulation cycle. The system generates a first control command from the motion video and sends it to the acupoint stimulation component. The control command includes a first control command and a second control command. The preset motion video is split into multiple preset video segments. Based on each preset video segment, the motion video, and the position signal, motion monitoring indicators are determined, and a second control command is generated based on the preset video segments and sent to the acupoint stimulation component. At the end of the stimulation cycle, the motion monitoring indicators are displayed on a display device, along with a prompt message indicating whether to continue to the next stimulation cycle. Upon receiving feedback indicating continued stimulation, the next stimulation cycle continues; upon receiving feedback indicating no further stimulation, stimulation is stopped.

[0006] According to the present invention, multiple key points of a person in multiple preset action video frames are detected by a key point detection model; based on the position coordinates of the key points in the multiple preset action video frames, it is determined whether each preset action video frame is a segmented video frame; each segmented video frame is used as the first video frame in a preset video segment to obtain multiple preset video segments.

[0007] According to the present invention, determining whether each preset action video frame is a segmented video frame includes: according to the formula: Obtain the segmentation criteria for the i-th preset action video frame. ,in, Let J be the coordinates of the j-th keypoint in the i-th preset motion video frame. Let J be the coordinates of the j-th keypoint in the (i-1)-th preset motion video frame. Let J be the coordinates of the j-th keypoint in the (i+1)-th preset action video frame. For preset angle, For vectors with vector The angle between the points, n is the number of key points, j≤n, and j and n are both positive integers, max is the function that takes the maximum value; in the segmentation judgment condition If the condition is met, the i-th preset action video frame is determined to be a segmented video frame.

[0008] According to the present invention, determining motion monitoring indicators includes: acquiring a first timestamp of the first video frame and a second timestamp of the last video frame of each preset video segment; determining the segmentation time of the motion video based on the first timestamp, the second timestamp, and a preset reaction time; segmenting the motion video according to the segmentation time to obtain multiple motion video segments; acquiring the relative positional relationship between various wearable monitoring components that are in the same time period as each motion video segment based on the segmentation time and the position signal; and determining motion monitoring indicators based on the preset video segments, the motion video segments, and the relative positional relationship.

[0009] According to the present invention, determining the segmentation time of an action video includes: taking the time corresponding to a first timestamp as the segmentation start time of the action video; accumulating a second timestamp with a preset reaction time to obtain the segmentation end time; and obtaining the segmentation time based on the segmentation start time and the segmentation end time.

[0010] According to the present invention, determining motion monitoring indicators includes: acquiring the first coordinates of key points in each motion video frame of a motion video segment, and acquiring the second coordinates of key points in each preset motion video frame of a preset video segment; combining the first coordinates of multiple key points in the motion video frames to obtain a first coordinate vector, and processing it through a first feature extraction layer to obtain first coordinate feature information; combining the second coordinates of multiple key points in the preset motion video frames to obtain a second coordinate vector, and processing it through a first feature extraction layer to obtain second coordinate feature information; combining the relative positional relationships of multiple wearable monitoring components to obtain a relative positional relationship vector, and processing it through a second feature extraction layer to obtain second coordinate feature information. The process involves: processing the coordinate vectors of adjacent action video frames to obtain relative position feature information; subtracting the first coordinate vectors of adjacent action video frames to obtain a first motion trend vector, and processing this vector through a first feature extraction layer to obtain first motion trend feature information; subtracting the second coordinate vectors of adjacent preset action video frames to obtain a second motion trend vector, and processing this vector through a first feature extraction layer to obtain second motion trend feature information; subtracting the relative position relationship vectors of two adjacent sets to obtain a relative position trend vector, and processing this vector through a second feature extraction layer to obtain relative position trend feature information; and processing the first and second coordinate feature information using a first cross-attention mechanism to obtain second coordinate feature information. The system firstly determines the action feature information; then, through a second cross-attention mechanism, it processes the relative position feature information and the second coordinate feature information to obtain the second action feature information corresponding to the second coordinate feature information; through a third cross-attention mechanism, it processes the second motion trend feature information and the first motion trend feature information to obtain the first trend determination feature information corresponding to the second motion trend feature information; through a fourth cross-attention mechanism, it processes the second motion trend feature information and the relative position trend feature information to obtain the second trend determination feature information corresponding to the second motion trend feature information; finally, it combines the first action determination feature information and the second action determination feature information corresponding to the same preset action video frame. The process involves: stitching together lines to obtain motion determination feature information; stitching together the first and second trend determination feature information corresponding to the same preset motion video frames to obtain trend determination feature information; combining the motion determination feature information to obtain a motion determination matrix; inputting the matrix into a dynamic weight layer for processing to obtain weight allocation data; combining the trend determination feature information to obtain a trend determination matrix; processing the motion determination matrix through a first fully connected layer and a first activation layer to obtain motion determination indicators; processing the trend determination matrix through a second fully connected layer and a second activation layer to obtain trend determination indicators; and determining motion monitoring indicators based on the weight allocation data, motion determination indicators, and trend determination indicators.

[0011] According to the present invention, the controller is further configured to: acquire sample coordinate feature information and sample motion trend feature information of sample action video frames, and acquire sample relative position relationship, as well as sample relative position feature information and relative position trend feature information; obtain sample action judgment matrix and sample trend judgment matrix based on sample coordinate feature information, sample motion trend feature information, sample relative position feature information and relative position trend feature information, and second coordinate feature information and second motion trend feature information of preset action video frames; process the sample action judgment matrix through dynamic weight hierarchy to obtain weight allocation sample data; obtain weight loss function based on weight allocation sample data, sample action video frames and preset action video frames; and process the sample action judgment matrix based on the first fully connected layer and the first activation layer. The sample trend determination matrix is ​​processed to obtain sample action determination indicators; based on the sample action determination indicators and the annotation information of the sample action video frames, an action determination loss function is obtained; the sample trend determination matrix is ​​processed through the second fully connected layer and the second activation layer to obtain sample trend determination indicators; based on the sample trend determination indicators and the annotation information of the sample action video frames, a trend determination loss function is obtained; based on the weight loss function, action determination loss function, and trend determination loss function, a training loss function is determined; the first feature extraction layer, the second feature extraction layer, the first cross-attention mechanism, the second cross-attention mechanism, the third cross-attention mechanism, the fourth cross-attention mechanism, the dynamic weight layer, the first fully connected layer and the first activation layer, as well as the second fully connected layer and the second activation layer are trained using the training loss function.

[0012] According to the present invention, obtaining the weight loss function includes: performing feature extraction processing on multiple sample action video frames using a 3D convolutional neural network model to obtain first action video feature information, and performing feature extraction processing on multiple preset action video frames using a 3D convolutional neural network model to obtain second action video feature information; according to the formula: Obtain the weight loss function ,in, Assign sample data to the weights of the sample action judgment index, where T is the duration of the preset action video segment. This is the first action video feature information. For the second action video feature information, sim is the similarity function, and the weight allocation of the sample trend judgment index is 1- .

[0013] According to a second aspect of the present invention, a method for targeted dynamic stimulation of acupoints is provided, comprising: at the beginning of a stimulation cycle, playing a preset action video via a display device; generating a first control command based on the preset action video and sending it to an acupoint stimulation component; splitting the preset action video to obtain multiple preset video segments; determining action monitoring indicators based on each preset video segment, a user's action video captured by a video monitoring component, and position signals of multiple parts of the user obtained by a wearable monitoring component, and generating a second control command based on the preset video segments and sending it to the acupoint stimulation component; at the end of the stimulation cycle, displaying the action monitoring indicators via a display device and displaying a prompt message indicating whether to continue to the next stimulation cycle; continuing to the next stimulation cycle after receiving feedback information indicating that to continue to the next stimulation cycle, and stopping stimulation after receiving feedback information indicating that to no longer continue to the next stimulation cycle.

[0014] By adopting the above technical solution, the present invention can achieve the following technical effects: According to the present invention, at the beginning of the stimulation cycle, a preset action video can be played, and a first control command can be generated and sent to the acupoint stimulation component. The preset action video can be split into multiple preset video segments to determine the action monitoring index, and a second control command can be generated and sent to the acupoint stimulation component. At the end of the stimulation cycle, the action monitoring index is displayed, along with a prompt message indicating whether to continue the next stimulation cycle. After receiving feedback information, it is determined whether to continue the next stimulation cycle or stop stimulation. The user's action state can be evaluated in real time based on the action trend, and the stimulation method can be dynamically adjusted, improving the accuracy of the assessment of the user's action standard and enhancing the effect of acupoint stimulation. Furthermore, based on the position coordinates of key points in multiple preset action video frames, it can be determined whether each preset action video frame is a segmented video frame, and the preset action video can be split to obtain multiple preset video segments. Each preset video segment is a video segment of a person performing a complete action in the preset action video, providing basic data for determining the action monitoring index. When segmenting the action video, the segmentation time of the action video can be determined based on a first timestamp, a second timestamp, and a preset reaction time, thereby obtaining the action video segments. Considering that users need a certain reaction time before they can begin imitating the actions of characters in preset action video frames, the second timestamp is accumulated with the preset reaction time to obtain the segmentation end time. This allows for partial overlap between adjacent action video segments, ensuring that each action video segment can include the complete action performed by the user imitating the corresponding preset video segment, thus improving the accuracy, objectivity, and comprehensiveness of action video segment segmentation. When determining the action monitoring index, weight allocation data, action judgment index, and trend judgment index are determined based on preset video segments, action video segments, and their relative positional relationships, thereby determining the action monitoring index. Considering that users are in the rehabilitation phase and have difficulty performing standard actions, weight allocation data is determined, and the action judgment index and trend judgment index are weighted and summed based on the weight allocation data to determine the action monitoring index. Higher action monitoring indexes are given when users perform poorly but make an effort to perform correct actions, improving the accuracy, objectivity, and comprehensiveness of the action monitoring index determination. During training at each level, a weighted loss function can be obtained based on weighted sample data, sample action video frames, and preset action video frames.Considering that longer preset action videos make it harder for users to maintain standard movements, the duration of the preset action video is used as a weight for allocating sample data to the accuracy of the sample action judgment index. This means that the longer the preset action video, the faster the weight allocation of the sample action judgment index decreases, resulting in lower weight allocation sample data for the sample action judgment index. This makes it harder for the weight allocation sample data for the sample action judgment index corresponding to longer preset action videos to approach accurate weight allocation sample data, further improving the accuracy of determining the weight allocation sample data for the sample action judgment index and increasing the training value of longer preset action videos. This allows the simple dynamic weight layer after training to simulate the operation of complex 3D convolutional neural network models. Furthermore, action judgment loss functions and trend judgment loss functions are determined, improving the accuracy of obtaining the sample action judgment matrix and sample trend judgment matrix, thereby assisting the training of each layer, improving the accuracy and targeting of training, and enhancing the performance of each layer. Furthermore, at the end of the stimulation cycle, the device can display motion monitoring indicators and a prompt message indicating whether to continue to the next stimulation cycle. Upon receiving feedback, it can determine whether to continue to the next stimulation cycle or stop stimulation. This allows for real-time evaluation of the user's motion state in conjunction with motion trends, and dynamic adjustments to the stimulation method, improving the accuracy of assessing the user's motion standard and enhancing the effectiveness of acupoint stimulation.

[0015] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only, and are not intended to limit the invention. Other features and aspects of the invention will become clearer from the following detailed description of exemplary embodiments with reference to the accompanying drawings. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other embodiments can be obtained based on these drawings without creative effort.

[0017] Figure 1 A schematic diagram of an acupoint-targeting dynamic stimulation device according to an embodiment of the present invention is shown as an example.

[0018] Figure 2 A flowchart for determining motion monitoring indicators according to an embodiment of the present invention is shown as an example;

[0019] Figure 3 An exemplary flowchart of a dynamic stimulation method for acupoint targeting according to an embodiment of the present invention is shown. Detailed Implementation

[0020] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0021] The technical solution of the present invention will be described in detail below with reference to specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments.

[0022] Figure 1 An exemplary schematic diagram of an acupoint-targeted dynamic stimulation device according to an embodiment of the present invention is shown. The device includes: an acupoint stimulation component, a motion monitoring component, a display device, and a controller. The acupoint stimulation component includes a fixing component and a bionic stimulation component, wherein the fixing component is used to fix the bionic stimulation component at a preset acupoint location, and the bionic stimulation component is used to output pressure stimulation according to the control command of the controller. The motion monitoring component includes a video monitoring component and a wearable monitoring component. The video monitoring component is used to capture a video of the user's motion, and the wearable monitoring component is used to monitor the position signals of multiple parts of the user's body. The controller is used to: at the beginning of the stimulation cycle, play a preset motion video through the display device. The system generates a first control command based on a preset action video and sends it to the acupoint stimulation component. The control command includes a first control command and a second control command. The preset action video is split into multiple preset video segments. Based on each preset video segment, the action video, and the position signal, an action monitoring index is determined, and a second control command is generated based on the preset video segment and sent to the acupoint stimulation component. At the end of the stimulation cycle, the action monitoring index is displayed on a display device, along with a prompt message indicating whether to continue to the next stimulation cycle. Upon receiving feedback indicating that the next stimulation cycle should continue, the next stimulation cycle continues. Upon receiving feedback indicating that the next stimulation cycle should not continue, the stimulation stops.

[0023] According to an embodiment of the present invention, the acupoint-targeted dynamic stimulation device can play a preset action video at the beginning of a stimulation cycle and generate a first control command, which is sent to the acupoint stimulation component. The preset action video can be split into multiple preset video segments to determine action monitoring indicators, and a second control command can be generated and sent to the acupoint stimulation component. At the end of the stimulation cycle, the action monitoring indicators are displayed, along with a prompt message indicating whether to continue to the next stimulation cycle. Upon receiving feedback, the device determines whether to continue to the next stimulation cycle or stop stimulation. The device can evaluate the user's action status in real time based on action trends and dynamically adjust the stimulation method, improving the accuracy of assessing the user's action standard and enhancing the effect of acupoint stimulation.

[0024] According to an embodiment of the present invention, the acupoint stimulation component includes a fixing component and a bionic stimulation component. The fixing component is used to fix the bionic stimulation component at a preset acupoint location, and the bionic stimulation component is used to output pressure stimulation according to the control command of the controller. The fixing component can be a lightweight rigid ring-shaped body with an adjustable size and a movable rotating pushing device, and the bionic stimulation component is built into the pushing device. It can be connected to the main body of the device through an adjustable fixing strap, with the end wrapped around the back of the knee for fixation, to ensure that it will not fall off or shift during the user's movement. For example, the strap supports the thigh to ensure that the bionic stimulation component will not fall off or shift during leg flexion, extension, or swinging. The preset acupoint can be 2cm below and 2cm inside the midpoint of the popliteal crease, between the medial head tendon of the gastrocnemius muscle, below and inside the Weizhong acupoint, or inside the Heyang acupoint, etc. The part of the fixing component that contacts the skin is a soft pad, which can improve the user's comfort and is suitable for various skin types. The biomimetic stimulation component can be a biomimetic finger pressure head, that is, a hemispherical medical silicone head with a finely textured surface, which can receive control commands from the controller and adjust the stimulation pressure (e.g., 0.5-3 kg) according to the control commands through a rotating push device to simulate finger pressure from a human thumb and output pressure stimulation.

[0025] According to an embodiment of the present invention, the motion monitoring component includes a video monitoring component and a wearable monitoring component. The video monitoring component is used to capture video of the user's movements, and the wearable monitoring component is used to monitor position signals of multiple parts of the user's body. The video monitoring component can be a camera that captures the user's movements while the user is moving, thus creating the motion video. The wearable monitoring component can be a wearable gyroscope (e.g., an inertial measurement unit), worn on the user's neck, waist, knees, ankles, elbows, shoulders, etc. It can capture the user's motion video through video monitoring components and monitor the position signals (i.e., the three-dimensional coordinates of each part of the user) of multiple parts of the user through wearable monitoring components, and identify important motion nodes. When an important motion node is reached, a trigger signal is generated. Then, through a trained first feature extraction layer, second feature extraction layer, first cross-attention mechanism, second cross-attention mechanism, third cross-attention mechanism, fourth cross-attention mechanism, dynamic weight layer, first fully connected layer and first activation layer, as well as second fully connected layer and second activation layer, the position signal of the motion detected by the wearable gyroscope is compared with the key points of the motion. The user's motion video captured by the video monitoring component is compared with the preset motion video (i.e., rehabilitation exercise teaching video) played on the display device, eliminating the influence of body shape differences and more objectively assessing the standard of the user's motion and motion trend.

[0026] According to an embodiment of the present invention, the display device can play preset action videos (i.e., rehabilitation exercise instruction videos) for users to imitate the preset action videos and perform exercises. At the end of the stimulation cycle, the display device can display action monitoring indicators and display a prompt message indicating whether to continue the next stimulation cycle, prompting the user whether to continue acupoint stimulation.

[0027] According to an embodiment of the present invention, the controller is configured to: play a preset action video via a display device at the start of a stimulation cycle. The duration of the stimulation cycle is the duration of the preset action video. At the start of the stimulation cycle, the preset action video is played via the display device for the user to imitate the preset action video for rehabilitation exercises. The preset action video may be an instructional video of the "three-step sequential exercise method," namely, instructional videos of four-point kneeling hip flexion training, four-point kneeling hip swing training, and prone-kneeling dynamic transition training. These three videos constitute a perfect rehabilitation sequence of "stability-loosening-integration," which works synchronously with the analgesic effect brought about by acupoint stimulation, achieving a unity of pain relief and therapeutic function.

[0028] According to an embodiment of the present invention, a first control command is generated based on a preset action video and sent to an acupoint stimulation component. The control command includes a first control command and a second control command. When the preset action video begins playing, the first control command is generated and sent to the acupoint stimulation component to stimulate the user's acupoints. The first control command provides basic intensity acupoint stimulation; for example, the first control command is a stimulation pressure of 0.5 kg to control the acupoint stimulation component to provide basic intensity acupoint stimulation to the user. The second control command is generated based on the preset action video and can provide different intensities of acupoint stimulation according to the action the user is imitating. For example, if the preset action video shows a person rotating their waist to its maximum extent, and the user is imitating this maximum waist rotation, the generated second control command would be a stimulation pressure of 2 kg to control the acupoint stimulation component to provide higher intensity acupoint stimulation to the user, thereby enhancing the effect of acupoint stimulation.

[0029] According to an embodiment of the present invention, a preset action video is split to obtain multiple preset video segments, including: detecting multiple key points of a person in multiple preset action video frames of the preset action video using a key point detection model; determining whether each preset action video frame is a segmented video frame based on the position coordinates of the key points in the multiple preset action video frames; and using each segmented video frame as the first video frame in the preset video segments to obtain multiple preset video segments.

[0030] According to an embodiment of the present invention, a preset action video frame can be acquired every second, and multiple key points of the person in the multiple preset action video frames can be detected using a key point detection model (e.g., OpenPose, HRnet, etc.), as well as the two-dimensional coordinates of each key point in the image coordinate system, such as key points of the neck, shoulder, hip, elbow, and knee. The two-dimensional coordinates of each key point in the image coordinate system are the position coordinates of the key points.

[0031] According to an embodiment of the present invention, determining whether each preset action video frame is a segmented video frame based on the position coordinates of key points in multiple preset action video frames includes: obtaining the segmentation determination condition of the i-th preset action video frame according to formula (1). ,

[0032] (1)

[0033] in, Let J be the coordinates of the j-th keypoint in the i-th preset motion video frame. Let J be the coordinates of the j-th keypoint in the (i-1)-th preset motion video frame. Let J be the coordinates of the j-th keypoint in the (i+1)-th preset action video frame. For preset angle, For vectors with vector The angle between the points, n is the number of key points, j≤n, and j and n are both positive integers, max is the function that takes the maximum value; in the segmentation judgment condition If the condition is met, the i-th preset action video frame is determined to be a segmented video frame.

[0034] According to an embodiment of the present invention, in formula (1), Vectors representing each key point with vector The maximum value among the included angles, when the above maximum value is greater than or equal to a preset angle (e.g., 90°, 100°, etc.). In this case, it can be assumed that in the (i-1), i, and (i+1)th preset action video frames, the movement direction of the j-th key point of the character's body undergoes a significant change, meaning the character's movement changes drastically, with a change in movement direction. Therefore, the i-th preset action video frame can be considered the key frame where the character's movement changes. For example, after the i-th preset action video frame, the character's arm changes from contracting in front of the chest to extending overhead, or after the i-th preset action video frame, the character's arm returns from extending overhead to the sides of the body. At this time, the position of the key point corresponding to the elbow will change significantly, indicating a change in movement direction. Therefore, the i-th preset action video frame can be considered the key frame where the character's movement changes. Furthermore, it can be assumed that... As a criterion for judgment In the segmentation determination condition If the condition is met, the i-th preset action video frame can be considered as the key frame of the change in the character's action, and the i-th preset action video frame can be determined as a segmented video frame, thereby segmenting the preset action video.

[0035] According to an embodiment of the present invention, each segmented video frame is used as the first video frame in each preset video segment, and the preceding preset action video frame of each segmented video frame is used as the last video frame of the previous preset video segment. After segmentation, multiple preset video segments can be obtained. Through the above processing, each change in the character's movement in the preset action video allows for segmentation of the preset action video, thus making each preset video segment a video clip of the character performing a complete action in the preset action video, facilitating comparison with the user's action video. For example, a video clip in the preset action video where the character's arm changes from retracting in front of the chest to extending overhead can be considered one preset video segment, and a video clip where the character's arm returns from extending overhead to retracting in front of the chest can be considered another preset video segment.

[0036] In this way, based on the position coordinates of key points in multiple preset action video frames, it can be determined whether each preset action video frame is a segmented video frame, and the preset action video can be split into multiple preset video segments. Each preset video segment is a video clip of a person performing a complete action in the preset action video, providing basic data for determining motion monitoring indicators.

[0037] According to an embodiment of the present invention, determining motion monitoring indicators based on each preset video segment, the motion video, and the position signal includes: acquiring a first timestamp of the first video frame and a second timestamp of the last video frame of each preset video segment; determining a segmentation time for the motion video based on the first timestamp, the second timestamp, and a preset reaction time; segmenting the motion video according to the segmentation time to obtain multiple motion video segments; acquiring the relative positional relationship between various wearable monitoring components that are in the same time period as each motion video segment based on the segmentation time and the position signal; and determining motion monitoring indicators based on the preset video segments, the motion video segments, and the relative positional relationship.

[0038] According to an embodiment of the present invention, a first timestamp of the first video frame and a second timestamp of the last video frame of each preset video segment are obtained. For example, if the timestamp of the first video frame of a preset video segment is 5 seconds and the timestamp of the last video frame is 10 seconds, then 5 seconds is the first timestamp of the preset video segment and 10 seconds is the second timestamp of the preset video segment.

[0039] According to an embodiment of the present invention, determining the segmentation time of an action video based on a first timestamp, a second timestamp, and a preset reaction time includes: using the time corresponding to the first timestamp as the start time of segmentation of the action video; accumulating the second timestamp with the preset reaction time to obtain the end time of segmentation; and obtaining the segmentation time based on the start time and the end time of segmentation. For example, if the time corresponding to the first timestamp is the 5th second, then the 5th second can be used as the start time of segmentation of the action video. The reaction time required for a user to imitate the actions of a person in a preset video clip is the preset reaction time, for example, 0.3 seconds, 0.5 seconds, etc. Since users need a certain reaction time to imitate the actions of a character in a preset action video frame, the user's actions in the action video frame may lag behind the actions of the character in the preset action video frame at the same timestamp. For example, the actions of the character in the preset action video frame corresponding to the 10-second timestamp are similar to the user's actions in the action video frame corresponding to the 10.3-second timestamp. Therefore, the second timestamp can be added to the preset reaction time to obtain the segmentation end time. For example, if the second timestamp corresponds to the 10-second time and the preset reaction time is 0.3 seconds, then the 10.3-second time can be used as the segmentation end time of the action video. Then, based on the segmentation start time and segmentation end time, the segmentation time can be obtained.

[0040] According to an embodiment of the present invention, the motion video is segmented according to the segmentation time to obtain multiple motion video segments. For example, if the segmentation start time is 5 seconds and the segmentation end time is 10.3 seconds, the motion video frame with the timestamp of 5 seconds can be used as the start frame of the motion video segment, and the motion video frame with the timestamp of 10.3 seconds can be used as the end frame of the motion video segment to segment the motion video, thereby obtaining the corresponding motion video segments. Based on the same processing method, multiple motion video segments can be obtained. Because the end time of each action video segment is delayed relative to the end time of the corresponding preset video segment (i.e., the preset video segment that the user is imitating in the action video segment) by using a preset reaction time, and the start time of the segment is not delayed to ensure the integrity of the action, adjacent action video segments will have partially overlapping video segments. For example, if the current action video segment is from 10 seconds to 15.3 seconds, and the previous action video segment was from 5 seconds to 10.3 seconds, the beginning of the current action video segment will overlap with the end of the previous action video segment. That is, the two action video segments will have overlapping action video segments from 10 seconds to 10.3 seconds. In other words, the two action video segments have a small portion of the same user's actions. Since the duration of the overlapping action video segments is short (i.e., only the preset reaction time), it has little impact on determining the action monitoring indicators, and it ensures that the user's imitation of the corresponding preset video segment in each action video segment is complete.

[0041] In this way, the segmentation time of the action video can be determined based on the first timestamp, the second timestamp, and the preset reaction time, thereby obtaining action video segments. Considering that users need a certain reaction time before they can begin imitating the actions of a character in the preset action video frame, the second timestamp is added to the preset reaction time to obtain the segmentation end time. This allows for partial overlap between adjacent action video segments, ensuring that each action video segment can include the complete action performed by the user imitating the corresponding preset video segment, thus improving the accuracy, objectivity, and comprehensiveness of action video segment segmentation.

[0042] According to an embodiment of the present invention, the relative positional relationship between various wearable monitoring components within the same time period as each action video segment is obtained based on the segmentation time and the position signal. For example, the position signal of the wearable monitoring component on the neck can be selected as the reference coordinate. The position signals (i.e., three-dimensional coordinates) of the wearable monitoring components on other parts of the body are subtracted from the reference coordinate to obtain the position signals of each wearable monitoring component. The vector between the position signal of the wearable monitoring component on the neck is the relative positional relationship. The above relative positional relationship can reflect the user's action posture within the same time period as each action video segment.

[0043] Figure 2 A flowchart for determining motion monitoring indicators according to an embodiment of the present invention is shown as an example.

[0044] According to an embodiment of the present invention, determining motion monitoring indicators based on the preset video segment, the motion video segment, and the relative positional relationship includes: acquiring the first coordinates of key points in each motion video frame of the motion video segment, and acquiring the second coordinates of key points in each preset motion video frame of the preset video segment; combining the first coordinates of multiple key points in the motion video frame to obtain a first coordinate vector, and processing it through a first feature extraction layer to obtain first coordinate feature information; combining the second coordinates of multiple key points in the preset motion video frame to obtain a second coordinate vector, and processing it through a first feature extraction layer to obtain second coordinate feature information; and combining the relative positional relationships of multiple wearable monitoring components. The process involves: obtaining a relative positional relationship vector and processing it through a second feature extraction layer to obtain relative positional feature information; subtracting the first coordinate vectors of adjacent action video frames to obtain a first motion trend vector, and processing it through a first feature extraction layer to obtain first motion trend feature information; subtracting the second coordinate vectors of adjacent preset action video frames to obtain a second motion trend vector, and processing it through a first feature extraction layer to obtain second motion trend feature information; subtracting two adjacent sets of relative positional relationship vectors to obtain a relative positional trend vector, and processing it through a second feature extraction layer to obtain relative positional trend feature information; and finally, using a first cross-attention mechanism to process the first coordinate feature information and the second coordinate feature information. The process involves: processing the relative position feature information and the second coordinate feature information to obtain the first action determination feature information corresponding to the second coordinate feature information; processing the relative position feature information and the second coordinate feature information using a second cross-attention mechanism to obtain the second action determination feature information corresponding to the second coordinate feature information; processing the second motion trend feature information and the first motion trend feature information using a third cross-attention mechanism to obtain the first trend determination feature information corresponding to the second motion trend feature information; processing the second motion trend feature information and the relative position trend feature information using a fourth cross-attention mechanism to obtain the second trend determination feature information corresponding to the second motion trend feature information; and then combining the first action determination feature information and the second motion trend feature information corresponding to the same preset action video frame. The motion determination feature information is obtained by concatenating the motion determination feature information and concatenating the first trend determination feature information and the second trend determination feature information corresponding to the same preset motion video frames to obtain trend determination feature information. The motion determination feature information is combined to obtain a motion determination matrix, which is then processed by a dynamic weighting layer to obtain weight allocation data. The trend determination feature information is combined to obtain a trend determination matrix. The motion determination matrix is ​​processed through a first fully connected layer and a first activation layer to obtain motion determination indicators. The trend determination matrix is ​​processed through a second fully connected layer and a second activation layer to obtain trend determination indicators. Based on the weight allocation data, motion determination indicators, and trend determination indicators, motion monitoring indicators are determined.

[0045] According to an embodiment of the present invention, the position coordinates of the key points in each action video frame of the action video segment are the first coordinates. Based on the same processing method, the second coordinates of the key points in each preset action video frame of the preset video segment can be obtained. Since the number of key points in each action video frame is fixed, for example, eight key points in total (neck, shoulder, hip, elbow, knee), the first coordinates of multiple key points in the action video frame can be combined to obtain a first coordinate vector (e.g., eight first coordinates combined to obtain a 16-dimensional first coordinate vector). This vector is then subjected to dimensionality upscaling (e.g., from 16 dimensions to 64 dimensions) through a first feature extraction layer (e.g., including fully connected layers and activation layers, where the activation layer uses the ReLU activation function). This allows the acquisition of the first coordinate feature information, which describes the positional features of the key points in each action video frame. Similarly, based on the same processing method, the second coordinates of multiple key points in the preset action video frame can be combined to obtain a second coordinate vector. This vector is then processed through a first feature extraction layer to obtain second coordinate feature information, which describes the positional features of the key points in each preset action video frame. The relative positional relationships of multiple wearable monitoring components can be combined to obtain a relative positional relationship vector. This vector can then be upgraded through a second feature extraction layer (e.g., including fully connected layers and activation layers, where the activation layer is a feature extraction layer processed using the ReLU activation function) to obtain relative positional feature information that can describe the relative positional relationships of various parts of the user's body.

[0046] According to an embodiment of the present invention, subtracting the first coordinate vectors of adjacent action video frames yields a new vector, which is the first motion trend vector. Further, by performing dimensionality upscaling through a first feature extraction layer, first motion trend feature information can be obtained, which describes the user's motion trend characteristics between adjacent action video frames. Based on the same processing method, subtracting the second coordinate vectors of adjacent preset action video frames yields a second motion trend vector, which, after processing through the first feature extraction layer, yields second motion trend feature information, describing the user's motion trend characteristics between adjacent preset action video frames. Furthermore, subtracting two adjacent sets of relative positional relationship vectors yields a relative positional trend vector, which, after processing through the second feature extraction layer, yields relative positional trend feature information, describing the changing trends of the relative positional relationships of various parts of the user. The first feature extraction layer is used to process two-dimensional coordinate information, and the second feature extraction layer is used to process three-dimensional coordinate information.

[0047] According to an embodiment of the present invention, a first cross-attention mechanism is used to process the first coordinate feature information and the second coordinate feature information to obtain the first action determination feature information corresponding to the second coordinate feature information. The Q matrix in the first cross-attention mechanism is the query matrix. Multiplying the second coordinate feature information by the query matrix projects the second coordinate feature information into a more suitable attention space for computation, thereby obtaining a transformed vector, which is the query vector. The K matrix in the first cross-attention mechanism is the key-value matrix. Multiplying the first coordinate feature information by the key-value matrix projects the first coordinate feature information into a more suitable attention space for computation, thereby obtaining a transformed vector, which is the key-value vector. Based on the same processing method, the key-value vector corresponding to each first coordinate feature information can be obtained. Since the query matrix and the key-value matrix have the same shape, the query vector and the key-value vector also have the same dimension, thus allowing the dot product operation to be performed on the query vector and the key-value vector. Furthermore, the query vector is multiplied by the key-value vector corresponding to each first coordinate feature (i.e., a dot product operation), then divided by the square root of the dimension of the key-value vector, and then processed through the softmax activation function to obtain the relevant weight of each first coordinate feature. This weight describes the degree of correlation between each first coordinate feature and the second coordinate feature, that is, the correlation between the position of the key point in each action video frame and the position of the key point in the preset action video frame. The larger the relevant weight, the higher the correlation between the position of the key point in the action video frame and the position of the key point in the preset action video frame, and the closer the positions of the key points in the action video frame and the preset action video frame are. Of course, both the query matrix and the key-value matrix are learnable weight matrices, and their parameters can be determined through training. Since the user is in the rehabilitation stage, theoretically, the user's movement standard should be lower. Therefore, the aforementioned relevant weights should be lower, that is, the position of the key point in the action video frame differs significantly from the position of the key point in the corresponding preset action video frame.

[0048] According to an embodiment of the present invention, by multiplying the first coordinate feature information by the weight matrix of the first cross-attention mechanism (i.e., the V matrix of the first cross-attention mechanism), a weight vector corresponding to each first coordinate feature information can be obtained. The weight matrix is ​​a learnable weight matrix, and its parameters can be determined through training. Further, by weighting and summing multiple weight vectors using the relevant weights corresponding to each first coordinate feature information, a new vector is obtained, which is the first action judgment feature information. This feature information can fuse the features of the key points in the action video frame related to the key point positions of the preset action video frame, describing the features of the key point positions of the preset action video frame. It is used to determine, from the perspective of the video, whether the user's action in each action video frame matches the standard action of the person in the corresponding preset action video frame (the preset action video frame in which the user is imitating), thereby improving the accuracy and specificity of the feature representation of the key point positions in the preset action video frame. Based on the same processing method, the first action judgment feature information corresponding to each second coordinate feature information can be obtained.

[0049] According to an embodiment of the present invention, similar to obtaining the first action determination feature information, a second cross-attention mechanism can be used to process the relative position feature information and the second coordinate feature information to obtain the second action determination feature information corresponding to the second coordinate feature information. This can fuse features of the relative positional relationships of various parts of the user related to the positions of key points in the preset action video frame, describing the positional features of the key points in the preset action video frame, and used to determine from a spatial perspective whether the user's actions in each action video frame match the standard actions of the person in the corresponding preset action video frame. Furthermore, a third cross-attention mechanism can be used to process the second motion trend feature information and the first motion trend feature information to obtain the first trend determination feature information corresponding to the second motion trend feature information. This can fuse features of the user's motion trend in the action video related to the standard motion trend of the person in the preset action video frame, describing the standard motion trend of the person in the preset action video frame, and used to determine from a video perspective whether the user's motion trend in each action video frame matches the standard motion trend of the person in the corresponding preset action video frame. Furthermore, the second motion trend feature information and the relative position trend feature information can be processed through the fourth cross-attention mechanism to obtain the second trend judgment feature information corresponding to the second motion trend feature information. It can integrate the features of the changing trend of the relative position relationship of various parts of the user related to the standard motion trend of the person in the preset motion video, describe the features of the standard motion trend of the person in the preset motion video, and use it to determine from a spatial perspective whether the user's motion trend in each motion video frame matches the standard motion trend of the person in the corresponding preset motion video frame.

[0050] According to embodiments of the present invention, by splicing together the first action determination feature information and the second action determination feature information corresponding to the same preset action video frame, action determination feature information can be obtained. This can comprehensively describe the degree of matching between the user's action in the action video frame and the standard action of the person in the corresponding preset action video frame from both video and spatial perspectives. Similarly, by splicing together the first trend determination feature information and the second trend determination feature information corresponding to the same preset action video frame, trend determination feature information can be obtained. This can comprehensively describe the degree of matching between the user's action trend in the action video frame and the standard action trend of the person in the corresponding preset action video frame from both video and spatial perspectives. For example, if the waist of the person in the preset action video frame has a tendency to rotate to the right, does the user's waist in the corresponding action video frame also have a tendency to rotate to the right?

[0051] According to an embodiment of the present invention, action determination feature information is combined to obtain an action determination matrix, which is then processed by a dynamic weight hierarchy to obtain weight allocation data. Since each preset video segment and action video segment can acquire the same number of preset action video frames when acquiring action video frames and preset action video frames—for example, if a preset video segment is 10 seconds long, 10 preset action video frames can be acquired, and another preset video segment is 5 seconds long, also acquiring 10 preset action video frames—each action video segment also acquires 10 action video frames. Because the number of action video frames and preset action video frames is fixed, all action determination feature information can be combined to obtain a matrix, namely the action determination matrix (for example, 100 128-dimensional action determination feature information pieces combined to form a 100×128-dimensional action determination matrix), which can describe the characteristics of the user's actions. Furthermore, the aforementioned action determination matrix is ​​input into a dynamic weighting layer (e.g., a GRU neural network including convolutional layers, pooling layers, fully connected layers, and activation layers, with the activation layers processed using the Sigmoid activation function) for processing, which outputs a weight, i.e., the weight allocation data for the action determination matrix. During processing by the aforementioned dynamic weighting layer, the degree of matching between the user's action at each moment and the standard action of a person in a preset action video can be determined first, i.e., the standardization of the user's action. When the standardization is high, a higher weight is assigned to the action determination matrix, i.e., a larger weight allocation data is output for the action determination matrix; when the standardization is low, a lower weight is assigned to the action determination matrix, i.e., a smaller weight allocation data is output for the action determination matrix. The difference between 1 and the weight allocation data of the action determination matrix can then be used as the weight allocation data for the trend determination matrix. The aforementioned weight allocation data of the action determination matrix and the weight allocation data of the trend determination matrix constitute the weight allocation data. Furthermore, since the user is in the rehabilitation phase, the standardization of their movements should theoretically be lower. Therefore, the movement judgment matrix will be assigned a lower weight in the data. When determining movement monitoring indicators based on the movement judgment matrix and the trend judgment matrix, the importance of the movement judgment matrix is ​​reduced, while the importance of the trend judgment matrix is ​​increased, thereby increasing the focus on the standardization of the user's movement trends. Similar to obtaining the movement judgment matrix, combining all trend judgment feature information yields the trend judgment matrix, which describes the characteristics of the user's movement trends.

[0052] According to an embodiment of the present invention, by processing the action judgment matrix through a first fully connected layer and a first activation layer (e.g., a sigmoid activation function), the action judgment matrix can be mapped to the probability of the user's action standard (e.g., 0.3, 0.4, etc.), which is the action judgment index. Similarly, by processing the trend judgment matrix through a second fully connected layer and a second activation layer (e.g., a sigmoid activation function), the trend judgment matrix can be mapped to the probability of the user's action trend standard, which is the trend judgment index. Further, by weighted summing the action judgment index and the trend judgment index according to the weight allocation data, the action monitoring index can be obtained. For example, if the action judgment index is 0.4, the trend judgment index is 0.8, the weight allocation data for the action judgment index is 0.3, and the weight allocation data for the trend judgment index is 1-0.3=0.7, then the action judgment index and the trend judgment index can be weighted and summed using the weight allocation data, i.e., 0.4×0.3+0.8×0.7=0.68, and the action monitoring index is 0.68. The higher the movement monitoring index, the more standard the user's movements, the more standard the movement trend, the better the movement completion, and the better the rehabilitation effect. Furthermore, determining the movement monitoring index based on the aforementioned weighted data allows for a higher index even when the user's movement standard is low but the movement trend is high. In other words, even if the user's movement completion is poor, as long as they are striving to complete the correct movement and exhibit the same movement trend as the standard movement, a higher movement monitoring index can be obtained. This is more suitable for users in the rehabilitation phase who have difficulty completing standard movements.

[0053] In this way, weighted data, action judgment indicators, and trend judgment indicators can be determined based on preset video clips, action video clips, and relative positional relationships, thereby determining the action monitoring indicators. Considering that users are in the rehabilitation phase and have difficulty performing standard movements, weighted data was determined, and the action judgment indicators and trend judgment indicators were weighted and summed based on this data to determine the action monitoring indicators. This allows for assigning higher action monitoring indicators when users perform poorly but are making an effort to perform the correct movements, improving the accuracy, objectivity, and comprehensiveness of the determined action monitoring indicators.

[0054] According to an embodiment of the present invention, the controller is further configured to: acquire sample coordinate feature information and sample motion trend feature information of sample action video frames, and acquire sample relative position relationship, as well as sample relative position feature information and relative position trend feature information; obtain a sample action judgment matrix and a sample trend judgment matrix based on the sample coordinate feature information, sample motion trend feature information, sample relative position feature information and relative position trend feature information, and the second coordinate feature information and second motion trend feature information of the preset action video frame; process the sample action judgment matrix through a dynamic weight hierarchy to obtain weight allocation sample data; obtain a weight loss function based on the weight allocation sample data, sample action video frames and preset action video frames; and process the sample action judgment matrix based on the first fully connected layer and the first activation layer. The sample action judgment index is obtained by processing the fixed matrix; the action judgment loss function is obtained based on the sample action judgment index and the annotation information of the sample action video frame; the sample trend judgment matrix is ​​processed through the second fully connected layer and the second activation layer to obtain the sample trend judgment index; the trend judgment loss function is obtained based on the sample trend judgment index and the annotation information of the sample action video frame; the training loss function is determined based on the weight loss function, the action judgment loss function, and the trend judgment loss function; the first feature extraction layer, the second feature extraction layer, the first cross-attention mechanism, the second cross-attention mechanism, the third cross-attention mechanism, the fourth cross-attention mechanism, the dynamic weight layer, the first fully connected layer and the first activation layer, and the second fully connected layer and the second activation layer are trained using the training loss function.

[0055] According to embodiments of the present invention, multiple user action videos can be acquired as sample action videos. Similar to acquiring weight allocation data, sample coordinate feature information and sample motion trend feature information of sample action video frames are acquired, as well as sample relative position relationships, sample relative position feature information, and relative position trend feature information. Based on the sample coordinate feature information, sample motion trend feature information, sample relative position feature information, and relative position trend feature information, and the second coordinate feature information and second motion trend feature information of preset action video frames, a sample action judgment matrix and a sample trend judgment matrix are obtained. Then, the sample action judgment matrix is ​​processed through a dynamic weight hierarchy to obtain weight allocation sample data.

[0056] According to an embodiment of the present invention, a weight loss function is obtained by allocating sample data, sample action video frames, and preset action video frames based on weights, including: performing feature extraction processing on multiple sample action video frames using a 3D convolutional neural network model to obtain first action video feature information, and performing feature extraction processing on multiple preset action video frames using a 3D convolutional neural network model to obtain second action video feature information; and obtaining the weight loss function according to formula (2). ,

[0057] (2)

[0058] in, Assign sample data to the weights of the sample action judgment index, where T is the duration of the preset action video segment. This is the first action video feature information. For the second action video feature information, sim is the similarity function, and the weight allocation of the sample trend judgment index is 1- .

[0059] According to embodiments of the present invention, multiple sample action video frames can be considered to form a three-dimensional input data, namely, two spatial dimensions (length and width) and one temporal dimension (i.e., number of frames). Then, a 3D convolutional neural network model (e.g., R(2+1)D, I3D model) can be used to perform feature extraction processing on the multiple sample action video frames to obtain a high-dimensional (e.g., 1024-dimensional) feature vector, which is the first action video feature information, accurately describing the user's action characteristics in the sample action video. Similarly, by performing feature extraction processing on multiple preset action video frames using a 3D convolutional neural network model, a second action video feature information can be obtained, accurately describing the action characteristics of a person in the preset action video.

[0060] According to an embodiment of the present invention, This can represent the similarity (e.g., cosine similarity) between the first and second action video feature information. Since the first and second action video feature information are obtained based on a 3D convolutional neural network model, this similarity can accurately describe the degree of similarity between the user's actions in the sample action video and the standard actions of a person in a preset action video. It can represent the degree of deviation between the user's actions in a sample motion video and the standard actions of a person in a preset motion video. Furthermore, it can obtain sample data of weight allocation for the sample action judgment index determined based on a 3D convolutional neural network model. Since the weight allocation sample data for the aforementioned action judgment index is determined based on a 3D convolutional neural network model, it can be considered that the weight allocation sample data for this action judgment index is the accurate weight allocation sample data for the action judgment index. The smaller the accurate weight allocation sample data for the aforementioned action judgment index, the greater the deviation between the user's action and the standard action of the person in the preset action video, and the lower the importance of the action judgment index. Furthermore, since the longer the preset action video clip, the more difficult it is for the user to maintain the standard action, the duration T of the preset action video clip (e.g., 5 seconds) can be used as the weight of the accurate weight allocation sample data for the action judgment index corresponding to the preset action video (ignoring the dimension, only the value of the duration T is used as the weight), so that the longer the duration of the preset action video clip (i.e., the more difficult it is for the user to maintain the standard action), the more accurate the weight allocation sample data for the action judgment index. The faster the rate of decline, the lower the weight allocation sample data for the sample action judgment index, thus further improving the accuracy of determining the weight allocation sample data for the sample action judgment index.

[0061] According to an embodiment of the present invention, the absolute value of the difference between the weight allocation sample data of the sample action judgment index and the accurate weight allocation sample data of the sample action judgment index can be used. As the weight loss function, its value is reduced during training to make the weight allocation sample data of the sample action judgment index closer to the accurate weight allocation sample data. This makes the output of the simple dynamic weight layer closer to the output of the complex 3D convolutional neural network model, thereby enabling the simple dynamic weight layer to simulate the operation of the complex 3D convolutional neural network model and improving its performance. Furthermore, using the duration T of the preset action video segment as the weight of the accurate weight allocation sample data of the sample action judgment index corresponding to the preset action video allows the longer the preset action video segment, the faster the accurate weight allocation sample data of the sample action judgment index decreases, resulting in lower accurate weight allocation sample data of the sample action judgment index. This makes it more difficult for the sample action judgment index weight allocation sample data to approach the accurate weight allocation sample data, thus improving the training value of longer preset action videos.

[0062] According to an embodiment of the present invention, similar to obtaining the action judgment index, the sample action judgment matrix is ​​processed according to the first fully connected layer and the first activation layer to obtain the sample action judgment index. Further, the correct sample action judgment index of the sample action video frame can be manually labeled (i.e., when the user's action is judged to be standard, the sample action judgment index is labeled as 1; otherwise, it is labeled as 0). This is the labeling information of the sample action video frame. Then, the cross-entropy loss function based on the sample action judgment index and the labeling information of the sample action video frame can be determined, which is the action judgment loss function. During training, the action judgment loss function is reduced, thereby making the sample action judgment index closer to the labeling information of the sample action video frame. This improves the accuracy of the sample action judgment index determination, thereby improving the accuracy of the action judgment matrix acquisition, and enabling the action judgment matrix to accurately describe the characteristics of the user's actions.

[0063] According to an embodiment of the present invention, similar to obtaining the trend determination index, the sample trend determination matrix can be processed through a second fully connected layer and a second activation layer to obtain the sample trend determination index. Similar to obtaining the action determination loss function, a cross-entropy loss function based on the sample trend determination index and the annotation information of the sample action video frames (i.e., when the user's action trend is judged to be standard, the sample trend determination index is labeled as 1; otherwise, it is labeled as 0) can be determined. This is the trend determination loss function. During training, the trend determination loss function is reduced, thereby making the sample trend determination index closer to the annotation information of the sample action video frames. This improves the accuracy of determining the sample trend determination index, and consequently improves the accuracy of obtaining the trend determination matrix, enabling the trend determination matrix to accurately describe the characteristics of the user's action trend.

[0064] According to an embodiment of the present invention, a training loss function is determined based on a weight loss function, an action determination loss function, and a trend determination loss function. The training loss function is obtained by summing the aforementioned weight loss function, action determination loss function, and trend determination loss function. Furthermore, by training the loss function and adjusting the parameters of the first feature extraction layer, the second feature extraction layer, the first cross-attention mechanism, the second cross-attention mechanism, the third cross-attention mechanism, the fourth cross-attention mechanism, the dynamic weight layer, the first fully connected layer and the first activation layer, as well as the second fully connected layer and the second activation layer using gradient descent, training is completed after multiple training sessions (i.e., training using multiple preset action videos, weighted sample data, sample action video frames, preset action video frames, sample action judgment indicators, annotation information of sample action video frames, sample trend judgment indicators, and annotation information of sample action video frames). This yields the trained first feature extraction layer, the second feature extraction layer, the first cross-attention mechanism, the second cross-attention mechanism, the third cross-attention mechanism, the fourth cross-attention mechanism, the dynamic weight layer, the first fully connected layer and the first activation layer, as well as the second fully connected layer and the second activation layer.

[0065] In this way, a weighted loss function can be obtained based on weighted sample data, sample action video frames, and preset action video frames during the training of each level. Considering that the longer the preset action video is, the more difficult it is for users to maintain standard movements, the duration of the preset action video is used as the weight for the weighted sample data of the sample action judgment index. This means that the longer the preset action video is, the faster the weighted sample data of the sample action judgment index decreases, resulting in lower weighted sample data of the sample action judgment index. This makes it more difficult for the weighted sample data of the sample action judgment index corresponding to longer preset action videos to approach accurate weighted sample data, further improving the accuracy of determining the weighted sample data of the sample action judgment index and increasing the training value of longer preset action videos. This allows the simple dynamic weighted levels after training to simulate the operations of complex 3D convolutional neural network models. Furthermore, the action judgment loss function and trend judgment loss function are determined, improving the accuracy of obtaining the sample action judgment matrix and sample trend judgment matrix, thereby assisting the training of each level, improving the accuracy and targeting of training, and enhancing the performance of each level.

[0066] According to an embodiment of the present invention, a second control command is generated based on a preset video clip and sent to the acupoint stimulation component. For example, if the waist of a person in the preset video clip rotates to its maximum extent, the user should also imitate this rotation to its maximum extent. In this case, a second control command with a greater stimulation pressure than the first control command (e.g., increasing the stimulation pressure to 2 kg) can be generated and sent to the acupoint stimulation component.

[0067] According to an embodiment of the present invention, at the end of the stimulation cycle, a motion monitoring index is displayed on a display device, along with a prompt message indicating whether to continue to the next stimulation cycle. Displaying the motion monitoring index (e.g., a motion monitoring index of 0.68) at the end of the stimulation cycle helps the user or doctor understand the standardity of the user's movements and movement trends, and displays a prompt message indicating whether to continue to the next stimulation cycle (e.g., a prompt message including both "continue" and "stop" options) for the user or doctor to choose whether to continue stimulation.

[0068] According to an embodiment of the present invention, upon receiving feedback indicating that the next stimulation cycle should continue, the next stimulation cycle continues; upon receiving feedback indicating that the next stimulation cycle should not continue, stimulation is stopped. Upon receiving feedback indicating that the next stimulation cycle should continue, it can be assumed that the user or doctor believes that stimulation needs to continue, and thus the next stimulation cycle can continue. Upon receiving feedback indicating that the next stimulation cycle should not continue, it can be assumed that the user or doctor believes that the stimulation effect has been achieved and that further stimulation is unnecessary, and thus stimulation can be stopped.

[0069] In this way, at the end of the stimulation cycle, the display device shows the motion monitoring indicators and a prompt message indicating whether to continue to the next stimulation cycle. Upon receiving feedback, it determines whether to continue to the next stimulation cycle or stop stimulation. It can combine motion trends to evaluate the user's motion status in real time and dynamically adjust the stimulation method, improving the accuracy of assessing the user's motion standard and enhancing the effect of acupoint stimulation.

[0070] According to an embodiment of the present invention, the acupoint-targeted dynamic stimulation device can play a preset action video at the beginning of a stimulation cycle, generate a first control command, send it to the acupoint stimulation component, and split the preset action video to obtain multiple preset video segments, thereby determining action monitoring indicators. It can also generate a second control command and send it to the acupoint stimulation component. At the end of the stimulation cycle, the action monitoring indicators are displayed, along with a prompt message indicating whether to continue to the next stimulation cycle. Upon receiving feedback, it determines whether to continue to the next stimulation cycle or stop stimulation. It can evaluate the user's action state in real time based on action trends and dynamically adjust the stimulation method, improving the accuracy of assessing the user's action standard and enhancing the effect of acupoint stimulation. Furthermore, based on the position coordinates of key points in multiple preset action video frames, it can determine whether each preset action video frame is a segmented video frame and split the preset action video to obtain multiple preset video segments. Each preset video segment is a video clip of a person performing a complete action in the preset action video, providing basic data for determining action monitoring indicators. When segmenting the action video, the segmentation time of the action video can be determined based on a first timestamp, a second timestamp, and a preset reaction time, thereby obtaining the action video segments. Considering that users need a certain reaction time before they can begin imitating the actions of characters in preset action video frames, the second timestamp is accumulated with the preset reaction time to obtain the segmentation end time. This allows for partial overlap between adjacent action video segments, ensuring that each action video segment can include the complete action performed by the user imitating the corresponding preset video segment, thus improving the accuracy, objectivity, and comprehensiveness of action video segment segmentation. When determining the action monitoring index, weight allocation data, action judgment index, and trend judgment index are determined based on preset video segments, action video segments, and their relative positional relationships, thereby determining the action monitoring index. Considering that users are in the rehabilitation phase and have difficulty performing standard actions, weight allocation data is determined, and the action judgment index and trend judgment index are weighted and summed based on the weight allocation data to determine the action monitoring index. Higher action monitoring indexes are given when users perform poorly but make an effort to perform correct actions, improving the accuracy, objectivity, and comprehensiveness of the action monitoring index determination. During training at each level, a weighted loss function can be obtained based on weighted sample data, sample action video frames, and preset action video frames.Considering that longer preset action videos make it harder for users to maintain standard movements, the duration of the preset action video is used as a weight for allocating sample data to the accuracy of the sample action judgment index. This means that the longer the preset action video, the faster the weight allocation of the sample action judgment index decreases, resulting in lower weight allocation sample data for the sample action judgment index. This makes it harder for the weight allocation sample data for the sample action judgment index corresponding to longer preset action videos to approach accurate weight allocation sample data, further improving the accuracy of determining the weight allocation sample data for the sample action judgment index and increasing the training value of longer preset action videos. This allows the simple dynamic weight layer after training to simulate the operation of complex 3D convolutional neural network models. Furthermore, action judgment loss functions and trend judgment loss functions are determined, improving the accuracy of obtaining the sample action judgment matrix and sample trend judgment matrix, thereby assisting the training of each layer, improving the accuracy and targeting of training, and enhancing the performance of each layer. Furthermore, at the end of the stimulation cycle, the device can display motion monitoring indicators and a prompt message indicating whether to continue to the next stimulation cycle. Upon receiving feedback, it can determine whether to continue to the next stimulation cycle or stop stimulation. This allows for real-time evaluation of the user's motion state in conjunction with motion trends, and dynamic adjustments to the stimulation method, improving the accuracy of assessing the user's motion standard and enhancing the effectiveness of acupoint stimulation.

[0071] Figure 3 An exemplary flowchart of a dynamic acupoint stimulation method according to an embodiment of the present invention is shown. The method includes: Step S1, at the beginning of a stimulation cycle, playing a preset action video through a display device; Step S2, generating a first control command based on the preset action video and sending it to an acupoint stimulation component; Step S3, splitting the preset action video to obtain multiple preset video segments; Step S4, determining action monitoring indicators based on each preset video segment, the user's action video captured by a video monitoring component, and the position signals of multiple parts of the user obtained by a wearable monitoring component, and generating a second control command based on the preset video segments and sending it to the acupoint stimulation component; Step S5, at the end of a stimulation cycle, displaying the action monitoring indicators through a display device and displaying a prompt message indicating whether to continue to the next stimulation cycle; Step S6, after receiving feedback information indicating that to continue to the next stimulation cycle, continuing to the next stimulation cycle, and stopping stimulation after receiving feedback information indicating that to no longer continue to the next stimulation cycle.

[0072] This invention can be a method, apparatus, system, and / or computer program product. The computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for performing various aspects of the invention.

[0073] Those skilled in the art should understand that the embodiments of the present invention described above and shown in the accompanying drawings are merely examples and do not limit the present invention. The objectives of the present invention have been fully and effectively achieved. The functions and structural principles of the present invention have been demonstrated and explained in the embodiments, and any variations or modifications may be made to the implementation of the present invention without departing from the stated principles.

Claims

1. An acupoint-targeted dynamic stimulation device, characterized in that, include: The system comprises an acupoint stimulation component, a motion monitoring component, a display device, and a controller. The acupoint stimulation component includes a fixing component and a bionic stimulation component, wherein the fixing component is used to fix the bionic stimulation component at a preset acupoint location, and the bionic stimulation component is used to output pressure stimulation according to control commands from the controller. The motion monitoring component includes a video monitoring component and a wearable monitoring component; the video monitoring component is used to capture video of the user's movements, and the wearable monitoring component is used to monitor position signals of multiple body parts of the user. The controller is used to: play a preset motion video through the display device at the start of the stimulation cycle; and generate a first control command based on the preset motion video. The control commands are sent to the acupoint stimulation component, wherein the control commands include a first control command and a second control command; the preset action video is split into multiple preset video segments; based on each preset video segment, the action video, and the position signal, action monitoring indicators are determined, and a second control command is generated based on the preset video segments and sent to the acupoint stimulation component; at the end of the stimulation cycle, the action monitoring indicators are displayed on the display device, and a prompt message indicating whether to continue to the next stimulation cycle is displayed; after receiving feedback information to continue to the next stimulation cycle, the next stimulation cycle continues; after receiving feedback information not to continue to the next stimulation cycle, the stimulation stops.

2. The acupoint-targeted dynamic stimulation device according to claim 1, characterized in that, The preset action video is split into multiple preset video segments, including: using a key point detection model to detect multiple key points of the characters in multiple preset action video frames of the preset action video; determining whether each preset action video frame is a segmented video frame based on the position coordinates of the key points in the multiple preset action video frames; and using each segmented video frame as the first video frame in the preset video segment to obtain multiple preset video segments.

3. The acupoint-targeted dynamic stimulation device according to claim 2, characterized in that, Based on the position coordinates of key points in multiple preset action video frames, determine whether each preset action video frame is a segmented video frame, including: according to the formula: Obtain the segmentation criteria for the i-th preset action video frame. ,in, Let J be the coordinates of the j-th keypoint in the i-th preset action video frame. Let J be the coordinates of the j-th keypoint in the (i-1)-th preset motion video frame. Let J be the coordinates of the j-th keypoint in the (i+1)-th preset action video frame. For the preset angle, For vectors with vector The angle between the points, n is the number of key points, j≤n, and j and n are both positive integers, max is the function that takes the maximum value; in the segmentation judgment condition If the condition is met, the i-th preset action video frame is determined to be a segmented video frame.

4. The acupoint-targeted dynamic stimulation device according to claim 1, characterized in that, Determining motion monitoring indicators based on each preset video segment, the motion video, and the location signal includes: acquiring a first timestamp of the first video frame and a second timestamp of the last video frame of each preset video segment; determining the segmentation time of the motion video based on the first timestamp, the second timestamp, and a preset reaction time; segmenting the motion video according to the segmentation time to obtain multiple motion video segments; acquiring the relative positional relationship between various wearable monitoring components within the same time period as each motion video segment based on the segmentation time and the location signal; and determining motion monitoring indicators based on the preset video segments, the motion video segments, and the relative positional relationship.

5. The acupoint-targeted dynamic stimulation device according to claim 4, characterized in that, The segmentation time of the motion video is determined based on the first timestamp, the second timestamp, and the preset reaction time, including: taking the time corresponding to the first timestamp as the segmentation start time of the motion video; accumulating the second timestamp and the preset reaction time to obtain the segmentation end time; and obtaining the segmentation time based on the segmentation start time and the segmentation end time.

6. The acupoint-targeted dynamic stimulation device according to claim 4, characterized in that, Based on the preset video clip, the action video clip, and the relative positional relationship, determine the action monitoring indicators, including: obtaining the first coordinates of key points in each action video frame of the action video clip, and obtaining the second coordinates of key points in each preset action video frame of the preset video clip; combining the first coordinates of multiple key points in the action video frame to obtain a first coordinate vector, and processing it through a first feature extraction layer to obtain first coordinate feature information; combining the second coordinates of multiple key points in the preset action video frame to obtain a second coordinate vector, and processing it through a first feature extraction layer to obtain second coordinate feature information; combining the relative positional relationships of multiple wearable monitoring components to obtain relative positions. The system first extracts a relationship vector and processes it through a second feature extraction layer to obtain relative position feature information. Then, it subtracts the first coordinate vectors of adjacent action video frames to obtain a first motion trend vector, which is also processed through a first feature extraction layer to obtain first motion trend feature information. Next, it subtracts the second coordinate vectors of adjacent preset action video frames to obtain a second motion trend vector, which is also processed through a first feature extraction layer to obtain second motion trend feature information. Finally, it subtracts two adjacent sets of relative position relationship vectors to obtain a relative position trend vector, which is also processed through a second feature extraction layer to obtain relative position trend feature information. Finally, it uses a first cross-attention mechanism to process the first and second coordinate feature information. Obtain first action determination feature information corresponding to the second coordinate feature information; process the relative position feature information and the second coordinate feature information through a second cross-attention mechanism to obtain second action determination feature information corresponding to the second coordinate feature information; process the second motion trend feature information and the first motion trend feature information through a third cross-attention mechanism to obtain first trend determination feature information corresponding to the second motion trend feature information; process the second motion trend feature information and the relative position trend feature information through a fourth cross-attention mechanism to obtain second trend determination feature information corresponding to the second motion trend feature information; and combine the first action determination feature information and the second action determination feature information corresponding to the same preset action video frame. The motion determination feature information is obtained by splicing the fixed feature information. Then, the first trend determination feature information and the second trend determination feature information corresponding to the same preset motion video frames are spliced ​​together to obtain trend determination feature information. The motion determination feature information is combined to obtain a motion determination matrix, which is then processed by a dynamic weighting layer to obtain weight allocation data. The trend determination feature information is then combined to obtain a trend determination matrix. The motion determination matrix is ​​processed through a first fully connected layer and a first activation layer to obtain motion determination indicators. The trend determination matrix is ​​then processed through a second fully connected layer and a second activation layer to obtain trend determination indicators. Based on the weight allocation data, motion determination indicators, and trend determination indicators, motion monitoring indicators are determined.

7. The acupoint-targeted dynamic stimulation device according to claim 6, characterized in that, The controller is also used to: acquire sample coordinate feature information and sample motion trend feature information of sample action video frames, and acquire sample relative position relationship, as well as sample relative position feature information and relative position trend feature information; Based on the sample coordinate feature information, sample motion trend feature information, sample relative position feature information and relative position trend feature information, as well as the second coordinate feature information and second motion trend feature information of the preset action video frame, the sample action judgment matrix and the sample trend judgment matrix are obtained. The sample action judgment matrix is ​​processed by dynamic weight hierarchy to obtain weight allocation sample data; The weight loss function is obtained by allocating sample data, sample action video frames, and preset action video frames based on weights; the sample action judgment matrix is ​​processed by the first fully connected layer and the first activation layer to obtain the sample action judgment index; the action judgment loss function is obtained based on the sample action judgment index and the annotation information of the sample action video frames; the sample trend judgment matrix is ​​processed by the second fully connected layer and the second activation layer to obtain the sample trend judgment index. Based on the sample trend determination index and the annotation information of the sample action video frames, the trend determination loss function is obtained; based on the weight loss function, action determination loss function and trend determination loss function, the training loss function is determined; the first feature extraction layer, the second feature extraction layer, the first cross-attention mechanism, the second cross-attention mechanism, the third cross-attention mechanism, the fourth cross-attention mechanism, the dynamic weight layer, the first fully connected layer and the first activation layer, as well as the second fully connected layer and the second activation layer are trained using the training loss function.

8. The acupoint-targeted dynamic stimulation device according to claim 7, characterized in that, Based on the weighted sample data, sample action video frames, and preset action video frames, a weighted loss function is obtained, including: performing feature extraction processing on multiple sample action video frames using a 3D convolutional neural network model to obtain first action video feature information, and performing feature extraction processing on multiple preset action video frames using a 3D convolutional neural network model to obtain second action video feature information; according to the formula: Obtain the weight loss function ,in, Assign sample data to the weights of the sample action judgment index, where T is the duration of the preset action video segment. This is the first action video feature information. For the second action video feature information, sim is the similarity function, and the weight allocation of the sample trend judgment index is 1- .

9. A method for targeted dynamic stimulation of acupoints, wherein the method is performed by the device as described in any one of claims 1-8, characterized in that, include: At the start of the stimulation cycle, a preset action video is played via a display device; The first control command is generated based on the preset action video and sent to the acupoint stimulation component; The preset action video is split into multiple preset video segments. Based on each preset video segment, the user's action video captured by the video monitoring component, and the position signals of multiple parts of the user's body obtained by the wearable monitoring component, action monitoring indicators are determined. A second control command is generated based on the preset video segments and sent to the acupoint stimulation component. At the end of the stimulation cycle, the action monitoring indicators are displayed on the display device, along with a prompt message indicating whether to continue to the next stimulation cycle. Upon receiving feedback indicating that the user should continue to the next stimulation cycle, the stimulation cycle continues. Upon receiving feedback indicating that the user should not continue to the next stimulation cycle, the stimulation is stopped.

Images