Image processing method and system for rehabilitation training motion analysis

Abstract

The present application provides an image processing method and system for rehabilitation training motion analysis, which relates to the field of image processing technology. By establishing a sample motion image set and annotating it, a feature recognition channel is established to perform rehabilitation motion fitting, and a time-series rehabilitation motion fitting result is established; a joint image capture of the user's rehabilitation motion is performed to establish a joint time-series image set including local images and global images; contour extraction is performed on the global image and the local image, and feature recognition is performed on the contour extraction result using the feature recognition channel to establish a feature recognition result; a user motion time-series set is constructed based on the corresponding image time sequence; and image annotation is performed on the user motion time-series set using the time-series rehabilitation motion fitting result. The present application solves the technical problem that the prior art cannot effectively distinguish and process image information from different perspectives and different parts when performing image recognition, resulting in low accuracy in rehabilitation training motion analysis, and improves the accuracy and efficiency of rehabilitation motion analysis.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to an image processing method and system for rehabilitation training motion analysis. Background Art

[0002] In the field of rehabilitation training, motion analysis is an important means to evaluate the patient's rehabilitation process and effect. Traditional motion analysis methods rely on manual observation and evaluation by rehabilitation physicians or therapists. Although this method can provide certain feedback, its accuracy and consistency are affected by subjective factors, and the process is cumbersome and time-consuming.

[0003] With the advancement of computer vision technology, more and more studies have begun to explore automated rehabilitation motion analysis methods based on image processing technology. These technologies capture the patient's motion images and extract key information from them to assist in the evaluation and adjustment of the effects during rehabilitation, thereby improving the efficiency of analysis and evaluation. However, existing image processing technologies are insufficient in processing complex movements and accurately identifying joint positions. They usually ignore the relationship between local images and global images, resulting in the failure to effectively distinguish and process image information from different perspectives and different parts during image capture and state extraction, which directly affects the accurate identification of joint positions and motion trajectories, resulting in deviations in the evaluation of the user's motion characteristics and rehabilitation progress. Summary of the invention

[0004] The present application provides an image processing method and system for rehabilitation training motion analysis, which solves the technical problem that the prior art cannot effectively distinguish and process image information of different perspectives and different parts when performing image recognition, thereby affecting the precise positioning of joint motion information and resulting in low accuracy of rehabilitation training motion analysis, thereby achieving the technical effect of improving the accuracy and efficiency of rehabilitation motion analysis.

[0005] In view of the above problems, on the one hand, the present application provides an image processing method for rehabilitation training action analysis, the method comprising: establishing a sample action image set, performing sample annotation on the sample action image set, performing feature extraction, establishing a feature recognition channel, and using the sample action image set to perform rehabilitation action fitting, and establishing a time-series rehabilitation action fitting result, wherein the sample action image set is an image data set constructed by performing user image acquisition; performing joint image capture of user rehabilitation actions to establish a joint time-series image set, wherein each time node image in the joint time-series image set includes a local image and a global image, and performing image state analysis on the time-series image set. After extraction, adaptive image enhancement is performed, and a mapping between the local image and the global image is established for each time node; contour extraction is performed on the enhanced global image, a global joint distribution identifier is established, and the global joint distribution identifier is annotated to the local image according to the mapping, and contour extraction is performed on the local image. After the feature recognition channel is initialized based on the annotation of the global joint distribution identifier, feature recognition is performed on the contour extraction result using the feature recognition channel to establish a feature recognition result; after recognition enhancement of the feature recognition result, a user action timing set is constructed based on the corresponding image timing; and image annotation is performed on the user action timing set using the timing rehabilitation action fitting result.

[0006] On the other hand, the present application also provides an image processing system for rehabilitation training motion analysis, the system comprising: a rehabilitation motion fitting module, the rehabilitation motion fitting module is used to establish a sample motion image set, after sample annotation of the sample motion image set, perform feature extraction, establish a feature recognition channel, and use the sample motion image set to perform rehabilitation motion fitting to establish a time-series rehabilitation motion fitting result, wherein the sample motion image set is an image data set constructed by executing user image acquisition; a user image capture module, the user image capture module is used to perform joint image capture of the user's rehabilitation motion to establish a joint time-series image set, wherein each time node image in the joint time-series image set includes a local image and a global image, after image state extraction of the time-series image set, perform adaptive image Enhancement, and establish a local image and a global image mapping for each time node; an image feature recognition module, the image feature recognition module is used to extract the contour of the enhanced global image, establish a global joint distribution identifier, and annotate the global joint distribution identifier to the local image according to the mapping, extract the contour of the local image, initialize the feature recognition channel based on the annotation of the global joint distribution identifier, use the feature recognition channel to perform feature recognition on the contour extraction result, and establish a feature recognition result; an action timing set construction module, the action timing set construction module is used to construct a user action timing set based on the corresponding image timing after performing recognition enhancement on the feature recognition result; an image annotation module, the image annotation module uses the timing rehabilitation action fitting result to perform image annotation on the user action timing set.

[0007] One or more technical solutions provided in this application have at least the following technical effects or advantages:

[0008] By collecting and annotating sample action image sets, an image dataset for training and testing was constructed, which provided accurate data support for subsequent feature extraction and model training, ensuring that subsequent analysis could accurately identify and fit the patient's movements. By performing feature extraction on the sample action image set and establishing a feature recognition channel, a targeted path or framework was provided for subsequent image feature recognition, making feature recognition more efficient and accurate. The sample action image set was used to fit rehabilitation movements, a standard rehabilitation movement model was constructed, and the development process of ideal rehabilitation movements was fully presented from the perspective of time series. The time series rehabilitation movement fitting results were established, laying the foundation for subsequent comparative analysis of user movements. By performing joint image capture of user rehabilitation movements and establishing a joint time series image set, each time node image in the joint time series image set includes local images and global images, thereby more comprehensively recording the image information of users at different time nodes and different perspectives when performing rehabilitation training, and improving the comprehensiveness and accuracy of movement analysis. After extracting the image state of the time series image set, adaptive image enhancement was performed to improve image quality and reduce the interference of environmental factors on image analysis. By establishing a local and global image mapping, local images and global images can be associated, which facilitates comprehensive analysis of user rehabilitation movements from the whole to the part. The contours of local and global images are extracted and marked according to the global joint distribution, which can accurately locate the joint position and movement trajectory of the patient, improve the accuracy of joint movement information, and help further analyze the standardization of user movements. The feature recognition channel is initialized based on the annotation of the global joint distribution mark, so that the feature recognition channel can perform feature recognition based on accurate joint distribution information, which improves the accuracy of feature recognition. The feature recognition channel is used to perform feature recognition on the contour extraction results, and the feature recognition results are established to provide data for the subsequent construction of the user action time series set. The recognition enhancement of the feature recognition results further improves the accuracy and reliability of the feature recognition results. The user action time series set is constructed based on the corresponding image time series, and the process of the user's rehabilitation movement is presented from the perspective of time series. The user action time series set is annotated using the time series rehabilitation movement fitting results, which can compare and analyze the user's rehabilitation movement with the standard rehabilitation movement, and intuitively reflect the difference between the user's rehabilitation movement and the ideal movement.

[0009] To summarize, this application has constructed a complete image processing process for rehabilitation training motion analysis from the preparation of sample data to the capture, processing and analysis of user motions. It can accurately identify the key information in rehabilitation motions, especially joint motion information, and comprehensively analyze the differences between user rehabilitation motions and standard motions from the perspective of time series, thereby providing a more precise, efficient and personalized means for the evaluation of rehabilitation training, greatly improving the accuracy, effectiveness and practical application value of rehabilitation training motion analysis.

[0010] The above description is only an overview of the technical solution of the present application. In order to more clearly understand the technical means of the present application, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the present application more obvious and easy to understand, the specific implementation methods of the present application are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Figure 1 A flowchart of an image processing method for rehabilitation training motion analysis provided in an embodiment of the present application.

[0012] Figure 2 A schematic diagram of a process for annotating global joint distribution identifiers to local images according to mapping in an image processing method for rehabilitation training motion analysis provided in an embodiment of the present application.

[0013] Figure 3 A schematic diagram of a flow chart for establishing feature recognition results in an image processing method for rehabilitation training motion analysis provided in an embodiment of the present application.

[0014] Figure 4 A schematic diagram of the structure of an image processing system for rehabilitation training motion analysis provided in an embodiment of the present application.

[0015] Explanation of the reference numerals: rehabilitation motion fitting module 10 , user image capturing module 20 , image feature recognition module 30 , motion timing set construction module 40 , image annotation module 50 . DETAILED DESCRIPTION

[0016] The embodiment of the present application provides an image processing method and system for rehabilitation training motion analysis, which performs feature extraction and rehabilitation motion fitting after establishing a sample motion image set and performing sample annotation, captures a joint image of user rehabilitation motions, distinguishes between local and global images, and performs state extraction, enhancement and mapping processing. By establishing a global joint distribution identifier and performing feature recognition after local image annotation, a user motion timing set is constructed, and annotation is performed based on the previous timing rehabilitation motion fitting results, thereby achieving accurate analysis and evaluation of rehabilitation training motions. It solves the technical problem that the prior art cannot effectively distinguish and process image information from different perspectives and different parts during image recognition, thereby affecting the precise positioning of joint motion information and resulting in low accuracy of rehabilitation training motion analysis, and achieves the technical effect of improving the accuracy and efficiency of rehabilitation motion analysis.

[0017] Embodiment 1, as Figure 1 As shown, the embodiment of the present application provides an image processing method for rehabilitation training action analysis, the method comprising:

[0018] Step S1: Establish a sample action image set, perform sample annotation on the sample action image set, perform feature extraction, establish a feature recognition channel, and use the sample action image set to perform rehabilitation action fitting to establish a time-series rehabilitation action fitting result, wherein the sample action image set is an image data set constructed by executing user image acquisition.

[0019] Specifically, the sample action image set refers to a series of image data sets of standard or sample actions collected by image acquisition equipment during rehabilitation training, which is used to establish a benchmark for action analysis. The feature recognition channel is an algorithm or model used to identify and classify features in an image for subsequent action analysis. The time-series rehabilitation action fitting result is a rehabilitation action fitting result recorded in chronological order, reflecting the state and change process of the rehabilitation action at different time points, and is used to provide a comparative benchmark for user rehabilitation action analysis.

[0020] Through image acquisition devices such as cameras, images of users performing rehabilitation training are collected from different angles (front, side, back), and an image data set containing these actions is constructed, namely, a sample action image set. Then, these images are annotated in detail using image annotation software, key body parts and joint positions are marked, and accurate reference points are provided for feature extraction. For example, when annotating waist rehabilitation action images, information such as the bending point of the waist and the stretching state of the muscles will be marked on the image. By establishing a sample action image set and performing sample annotation, an accurate reference framework can be provided for feature extraction and action recognition.

[0021] Next, feature extraction is performed on the annotated sample action image set to extract key features such as joint positions, limb angles, and body part contours from these annotated images. The feature extraction process can be completed using feature extraction tools such as OpenPose and PoseNet, which can efficiently identify key points of the human body (such as wrists, elbows, knees, shoulders, etc.) and joint angles.

[0022] According to the feature extraction results and the labeled sample image set, a machine learning model (such as a convolutional neural network or a recurrent neural network) is trained to make the model learn how to automatically extract features from the action image, thereby constructing a feature recognition channel for subsequent extraction of action features from the user's actual rehabilitation action. Exemplarily, a convolutional neural network is initialized, and the labeled sample image set and the corresponding feature extraction results are input into the initialized convolutional neural network model for training. In each training batch, the model performs forward propagation based on the input data, calculates the prediction results, and calculates the error between the prediction results and the true annotations through the loss function. For example, a batch of sample images of knee joint rehabilitation actions are input into the model, and the model predicts the features of the knee joint, which are then compared with the features of the true annotations to calculate the error. According to the calculated error, the optimizer is used to back-propagate the error and adjust the parameters in the model (convolution kernel weights, biases, etc.). This process will be repeated continuously. After multiple training cycles, the model will continuously learn the features in the sample images and gradually improve the ability to recognize the features until the loss function value no longer changes significantly, and a feature recognition channel with satisfactory performance is constructed.

[0023] Finally, the sample action image set is analyzed and integrated to form a model that can represent the ideal rehabilitation action. For example, in leg rehabilitation training, the sample image data of different leg actions are fitted by the least squares method to obtain a complete leg rehabilitation action model. The rehabilitation action fitting results are displayed in the form of a time series to form a time series rehabilitation action fitting result, which lays the foundation for subsequent comparative analysis of user actions.

[0024] Step S2: Perform joint image capture of the user's rehabilitation movements to establish a joint time-series image set, wherein each time node image in the joint time-series image set includes a local image and a global image. After image state extraction of the time-series image set, perform adaptive image enhancement, and establish a local image and global image mapping for each time node.

[0025] Specifically, during the user's rehabilitation training, multiple image acquisition devices are used to simultaneously capture local and global images of the user. For example, when capturing images of hip rehabilitation movements, one lens can capture the global movement of the user's entire body, and another lens can focus on local areas such as the femoral head and acetabulum. The local image and global image at the same time node are combined, and the images at different time nodes are integrated in chronological order to form a complete joint time-series image set. This image set can fully display the changes in the user's rehabilitation movements over time from local and global perspectives at different time points.

[0026] Computer vision technology is used to extract image states from a set of time-series images, extracting key information such as joint positions, limb movement directions, and angle changes. For example, OpenPose or PoseNet can be used to extract key points of the human body (such as knees, shoulders, elbows, etc.). After image state extraction, image processing algorithms are used to automatically adjust image parameters according to the specific conditions of each image to improve image quality. For example, if the image at a certain moment has unclear local details due to insufficient lighting, the algorithm will increase the brightness and contrast of the image; if there is noise in the image, the algorithm will suppress the noise to ensure that the details of the movements and joints are clearly visible.

[0027] Finally, the local action at each time node is associated with the whole body movement, and a mapping between the local image and the global image is established. For example, tags are added to the image data or indexes are established. For example, when collecting arm rehabilitation action images, for the images of each time node, associated tags are added to the file metadata or database records of the global image and the local image, so that the corresponding local image can be easily found according to the global image, or its position and function in the global image can be understood from the local image. By establishing a mapping relationship between the local image and the global image, local and global information can be associated during analysis, thereby improving the accuracy of action analysis.

[0028] Step S3: perform contour extraction on the enhanced global image, establish a global joint distribution identifier, and annotate the global joint distribution identifier to the local image according to the mapping, perform contour extraction on the local image, initialize the feature recognition channel based on the annotation of the global joint distribution identifier, use the feature recognition channel to perform feature recognition on the contour extraction result, and establish a feature recognition result.

[0029] Specifically, the global joint distribution mark is a mark that marks the position and distribution of joints in the global image, which can reflect the position of each joint (such as knee joint, elbow joint, etc.) and the relative relationship between them in the entire rehabilitation action image. For example, in the global image of the whole body rehabilitation action, the global joint distribution mark can mark the position of the hip joint, knee joint, ankle joint and other joints, as well as the connection relationship of these joints in the human skeletal structure.

[0030] The edge detection algorithm is used to extract the contour of the enhanced global image, identify the external contour of the human body and the contour of key joints. For example, the Canny edge detection algorithm can be used to calculate the gradient of the image, set high and low thresholds to detect the edges in the image, and thus obtain the contour of the global image. For example, when processing the global image of spinal rehabilitation movements, the Canny edge detection algorithm can detect the contour of the spine and the contour of the surrounding muscles.

[0031] After obtaining the global image contour, a global joint distribution mark is established by analyzing the shape, position and matching of the contour with a known joint model. Establishing a global joint distribution mark helps to accurately identify and track the motion trajectory of the joint, ensuring the accuracy of the movement in subsequent analysis. Using the mapping relationship between the local image and the global image established in step S2, the global joint distribution mark is annotated to the local image according to this mapping. The contour of the local image that has been mapped and annotated is extracted, and the Canny edge detection algorithm can also be used to extract the contours of the joints and limbs in the local image.

[0032] Initialize the feature recognition channel based on the annotation of the global joint distribution mark, and set the initial parameters in the feature recognition channel. Exemplarily, in the feature recognition channel built based on the convolutional neural network, adjust the initial weight or bias of the network according to the global joint distribution mark. For example, if the global joint distribution mark shows the specific position and angle of a joint in the image, the feature recognition channel can initialize the parameters related to the joint feature recognition based on this information, so as to better identify the features in the local image.

[0033] The initialized feature recognition channel is used to perform feature recognition on the contour extraction result, extract features such as the movement state of the joints and the contraction state of the muscles, and integrate the various identified features to form a complete feature recognition result.

[0034] Through precise contour extraction and feature recognition, it is possible to track and analyze the user's movements, providing key body part information for rehabilitation training movement analysis.

[0035] Step S4: After the feature recognition result is enhanced, a user action timing set is constructed based on the corresponding image timing.

[0036] Specifically, in rehabilitation training, due to the complexity of the movements and the differences in image acquisition equipment, the captured images may be distorted. Therefore, it is necessary to correct the distortion in the image to restore the true geometric shape of the image, thereby improving the accuracy of feature recognition. The correction process first uses a camera calibration plate (such as a chessboard) to detect the distortion of the camera and identify the type of distortion in the image, such as radial distortion, tangential distortion, etc. Then, the camera's intrinsic parameters and distortion parameters are calculated, and the distortion correction algorithm is applied to correct the image distortion. Next, image post-processing techniques such as morphological operations, filtering, or non-maximum suppression (NMS) are applied to remove erroneous recognition results and enhance correct features.

[0037] In rehabilitation motion analysis, each image at each time point has a corresponding feature recognition result. According to the time sequence of image acquisition, the feature recognition results after recognition enhancement are arranged in chronological order to construct a user motion timing set. This user motion timing set records each movement stage of the patient during rehabilitation training. Through continuous time-series images, the entire process of the movement can be described in detail. For example, in wrist rehabilitation training, the wrist joint features (such as joint angle, flexion and extension speed, etc.) at each time point from the start movement to the end movement are arranged in sequence to form an orderly user motion timing set.

[0038] Step S5: using the time-series rehabilitation action fitting result to perform image annotation on the user action time-series set.

[0039] Specifically, the results of timing rehabilitation movement fitting are used to evaluate and guide each frame image in the user's movement timing set, and the differences between the user's movements at each moment and the corresponding standard movements are compared. The comparison results are marked on each frame image in the user's movement timing set. The marked content includes whether the movement is standard, whether the joint position is correct, whether the movement angle meets the standard, etc.

[0040] Through the results of sequential rehabilitation action fitting, each frame of the user's action timing set is accurately annotated, so that the patient's action execution can be clearly recorded, and the annotation can clearly indicate whether the user's action meets the rehabilitation standards, thereby providing users with personalized training feedback and helping to improve rehabilitation effects.

[0041] Further, such as Figure 2 As shown, in step S3, the global joint distribution identifier is annotated to the local image according to the mapping, including:

[0042] Step S31: Acquire the first acquisition coordinates of the global acquisition device, acquire depth data based on the enhanced global image, and construct the relative joint coordinates of the global joint according to the first acquisition coordinates and the depth data.

[0043] Step S32: Acquire second acquisition coordinates of the local acquisition device, and construct relative local image coordinates according to the second acquisition coordinates and depth data of the enhanced local image.

[0044] Step S33: using the first acquisition coordinates and the second acquisition coordinates to establish normalized coordinates, converting the relative joint coordinates and the local image coordinates into the same coordinates through the normalized coordinates, and then marking the global joint distribution identifier to the local image according to the mapping.

[0045] Specifically, the first acquisition coordinate is the coordinate position information of the global image acquisition device when acquiring the global image, which is usually the device installation position information known before the image acquisition. For example, in the acquisition of whole-body rehabilitation movements, the global image acquisition device is a camera located in the center of the top of the room. Its first acquisition coordinate may include the coordinate value of the camera in the three-dimensional space of the room. These coordinate values ​​determine the position of the camera, which affects the judgment of the joint position in the global image. The depth data reflects the distance from the object in the image to the acquisition device. When acquiring a global image, the depth data is obtained using the depth sensor integrated in the acquisition device. The relative joint coordinates of the global joint are constructed based on the first acquisition coordinates and the depth data. The relative joint coordinates accurately describe the position of each joint in the coordinate system with the global acquisition device as the origin, including the coordinate values ​​in the X, Y, and Z directions. Exemplarily, the first acquisition coordinate is, for a certain joint point P, the distance from the joint point P to the acquisition device is obtained through the depth data, and then the two-dimensional coordinates in the image are obtained through the image pixel coordinate conversion. , using the spatial coordinate transformation formula (such as, ,in Construct relative joint coordinates for the scale factor associated with the device parameters .

[0046] For the local image, similarly, the installation or positioning coordinates of the local acquisition device, that is, the second acquisition coordinates, are obtained. When acquiring the local image, the depth data is acquired using the depth sensor integrated in the acquisition device. Similar to constructing the relative joint coordinates, the relative local image coordinates are constructed based on the second acquisition coordinates and the depth data in combination with the spatial coordinate formula. The relative local image coordinates accurately describe the position of the object in the local image in the coordinate system with the local acquisition device as the origin. For example, in the local image of the elbow rehabilitation movement, the relative local image coordinates can determine the position of the elbow in the coordinate system with the local acquisition device as the origin.

[0047] The first acquisition coordinates and the second acquisition coordinates are aligned by using coordinate translation, scaling and other operations so that the two can be compared in the same spatial coordinate system to establish normalized coordinates. In the normalized coordinates, the relative joint coordinates and the local image coordinates are converted to coordinates in the normalized coordinate system through the same coordinate transformation. Then, according to the converted coordinates, the relative joint coordinates in the global image are matched with the relative local image coordinates in the local image, and the corresponding global joint distribution marks are marked on the local image.

[0048] Through precise coordinate transformation and mapping, the joint distribution information in the whole-body image can be accurately mapped to the local image, improving the accuracy of joint position recognition and analysis in the local image, thereby improving the accuracy and reliability of rehabilitation training movement analysis.

[0049] Further, such as Figure 3 As shown, after the feature recognition channel is initialized based on the annotation of the global joint distribution mark in step S1, the feature recognition channel is used to perform feature recognition on the contour extraction result to establish a feature recognition result, including:

[0050] Step S34: Establish an associated feature set based on the annotated matching associated features of the global joint distribution identifier, and establish feature call attention using the association degree of the associated feature set.

[0051] Step S35: establishing a temporary feature attention matching library of the feature recognition channel through the feature call attention, so as to complete the initialization of the feature recognition channel.

[0052] Step S36: performing feature matching recognition of the contour feature extraction result based on the feature recognition channel and the feature focus matching library to establish a feature recognition result.

[0053] Specifically, image processing algorithms such as corner detection and edge detection are used to identify image features near the global joint distribution mark, including muscles and bone structures related to specific joints. The detected features are matched with the global joint distribution mark to establish an associated feature set. The entire associated feature set contains all image features associated with the global joint distribution mark. For example, when analyzing the rehabilitation movement of the knee joint, the global joint distribution mark marks the position and direction of the knee joint. Based on this annotation information, associated features such as muscles and ligaments related to the knee joint are extracted, such as the quadriceps, patellar tendon, medial collateral ligament, etc. By analyzing the relationship between these associated features, such as the contraction direction of the muscle and the tension distribution of the ligament, an associated feature set can be established. After the associated feature set is established, the correlation between these features is used to establish feature call attention. The correlation can be understood as the correlation or dependency between features, which can be determined according to the dynamic principle of rehabilitation training. For example, the contraction of the quadriceps will affect the stability of the knee joint, so there is a high correlation between the quadriceps and the stability of the knee joint. Based on this correlation information, it is determined which features should be given priority and which features can be appropriately ignored in the subsequent feature recognition process.

[0054] According to the established feature call attention, a temporary feature attention matching library is constructed. This matching library contains the features and their associations that need to be focused on in the current analysis task. For example, when analyzing knee rehabilitation movements, if the contraction of the quadriceps femoris is determined to be a key feature, then in the temporary feature attention matching library, the quadriceps femoris and its related features (such as the muscle's starting and ending points, contraction direction, etc.) will be focused on. At the same time, the matching library will also record the associations between these features, such as the relationship between quadriceps femoris contraction and knee extension. Use this temporary feature library to initialize the feature recognition channel and adjust the algorithm parameters to improve the recognition sensitivity of specific features. By matching the associated features of the global joint distribution identifier and establishing a feature attention matching library, the feature recognition channel can be initialized so that it can focus on the most relevant features, improving the accuracy and efficiency of feature recognition, thereby better understanding and evaluating the user's rehabilitation training movements and providing more effective feedback.

[0055] Using the initialized feature recognition channel, the results of contour feature extraction are matched and recognized, and key features related to rehabilitation movements are identified from the contour feature extraction results as feature recognition results. These feature recognition results will provide important data support for subsequent rehabilitation movement analysis and evaluation.

[0056] Further, step S36 includes:

[0057] Step S361: Performing user body gesture recognition on the enhanced global image to establish a body gesture recognition result.

[0058] Step S362: input the first acquisition coordinates, the second acquisition coordinates, the device acquisition angle and the limb posture recognition result into the deformation fitting network to establish the deformation fitting result.

[0059] Step S363: establishing a matching tolerance threshold according to the deformation fitting result, and configuring a deformation bias.

[0060] Step S364: performing feature matching recognition of the contour feature extraction result based on the matching tolerance threshold and the deformation bias.

[0061] Specifically, the enhanced global image is analyzed, and a body posture recognition algorithm (such as OpenPose, MediaPipe, etc.) is used to identify the patient's body posture, extract the position, angle and relative relationship of each joint of the patient, and establish the body posture recognition result. The body posture recognition result includes the coordinates, angles and action types of each joint, which are used to analyze the user's current action or posture.

[0062] During the image acquisition process, changes in the device angle will cause the acquired image to deform. For example, when using a camera to shoot an object, if the camera angle is not perpendicular to the surface of the object, the shape of the object in the image will be distorted. Different acquisition devices may have different characteristics, such as resolution, lens distortion, etc. These differences will also cause the acquired image to deform. The deformation fitting network can analyze the deformation caused by factors such as device angle and acquisition device differences, and correct the shape of the object in the image to a state close to its true shape. The first acquisition coordinates, second acquisition coordinates, device acquisition angle, and limb posture recognition results extracted previously are input into the deformation fitting network for processing. The deformation fitting network corrects the image based on this information, restores the correct joint position, angle and other posture information, and establishes a deformation fitting result.

[0063] The deformable fitting network can be trained based on machine learning algorithms such as convolutional neural networks and graph convolutional networks. First, collect image data from different devices and perspectives. Collect a large amount of image data. These images include deformed images under different deformation conditions and corresponding non-deformed or corrected images. Initialize the weights and biases of the network, and use the collected data to supervise the initialization network. The network learns how to predict the non-deformed posture from the deformed image by adjusting its parameters to minimize the difference between the predicted posture and the true posture. Perform multiple training iterations until the convergence conditions are met, such as the change in the loss function value is less than the preset threshold or the maximum number of training iterations is reached. The trained convolutional network is output as the deformable fitting network.

[0064] The matching tolerance threshold refers to the error range allowed when performing feature matching. This threshold sets the tolerance of the matching algorithm and can handle small deviations in coordinates or angles. Deformation bias refers to the type or direction of features that the deformation fitting network needs to focus on during the deformation fitting process. These biases are usually some kind of trend deformation that may exist in the limbs during image acquisition, such as natural stretching or bending trends in a specific direction of movement. Analyze the deformation fitting results, determine the tolerance in the feature matching process, and set a matching tolerance threshold to allow a certain degree of error to avoid matching due to slight differences. At the same time, identify the trend deformation that exists uniformly in the image in the deformation fitting results, and configure the deformation bias parameters.

[0065] Based on the setting of matching tolerance threshold and deformation bias, the matching algorithm, such as template matching or feature point matching, is adjusted so that it can identify features within the tolerance range and correct systematic deviations. The contour feature extraction results are further matched and identified, thereby accurately identifying and classifying features in the image, providing precise data support for rehabilitation training motion analysis.

[0066] Furthermore, step S5 of the embodiment of the present application includes:

[0067] Step S51: obtaining a motion starting point of a sequential rehabilitation motion fitting result, and performing timing alignment between the sequential rehabilitation motion fitting result and the user motion timing set based on the motion starting point.

[0068] Step S52: performing frame-by-frame comparison on the timing rehabilitation action fitting result after timing alignment and the user action timing set to establish a deviation mark.

[0069] Step S53: using the inertial measurement unit to perform synchronous data collection of the user's rehabilitation movements and establish time series speed data.

[0070] Step S54: performing movement steady-state identification according to the time-series speed data and the time-series rehabilitation movement fitting result.

[0071] Step S55: completing image annotation according to the deviation mark and the action steady-state mark.

[0072] Specifically, the action starting point of the sequential rehabilitation action fitting result is obtained, that is, the starting state point in the sequential rehabilitation action. According to this action starting point, the user action timing set and the sequential rehabilitation action fitting result are matched on the time axis to ensure that the time axes of the two are consistent, so that subsequent comparison and analysis can be carried out in the same time frame.

[0073] The time-series rehabilitation action fitting results and the user action timing set that have been time-series aligned are compared frame by frame, and the rehabilitation action fitting results of each frame (i.e., each time node) are compared with the user's actual action, and the difference (i.e., deviation) between the fitting data and the actual action is identified, and these deviations are marked in the specific frame. For example, if in a certain frame, the fitting result shows that the arm should be bent to 90 degrees, but the user's actual action bends the arm to 85 degrees, then the angle difference will be recorded in the deviation mark corresponding to this frame.

[0074] When collecting user motion images, the inertial measurement unit is used to synchronously collect the user's rehabilitation movement speed data, and these data are arranged in chronological order to establish time series speed data. When the user performs rehabilitation movements, the inertial measurement unit can be attached to relevant parts of the body (such as wrists, ankles, etc.) to measure the acceleration, angular velocity and other information of the user's limbs during movement in real time. For example, in wrist rehabilitation training, the inertial measurement unit can measure data such as acceleration and angular velocity of wrist movement in real time, and then establish time series speed data by integrating and analyzing these data in time series. This time series speed data can reflect the speed characteristics of the user during rehabilitation training, such as the acceleration and deceleration process of the movement, and provide an important basis for the subsequent steady-state identification of the movement.

[0075] The action steady-state mark is a mark for whether the rehabilitation action is in a stable state. The action steady-state mark is performed based on the time series speed data and the time series rehabilitation action fitting results. For example, in waist rehabilitation training, if the waist movement speed is maintained within a small fluctuation range for a period of time, and its shape, position, etc. also meet the expected state of the rehabilitation action (based on the time series rehabilitation action fitting results), then the action in this time period can be marked as being in a steady state. The action steady-state mark helps to more comprehensively evaluate the quality and effect of rehabilitation actions.

[0076] Combine the deviation mark and the action steady-state mark to determine the action features that need to be marked. Mark the corresponding user action time series images. For example, for images with large deviations and actions that are not in a steady state, they can be marked as areas that need to be focused on or improved. Through this kind of marking, you can more intuitively understand the execution of rehabilitation actions, providing a strong basis for the guidance and evaluation of rehabilitation training.

[0077] Furthermore, after extracting the image state of the time-series image set in step S2, adaptive image enhancement is performed, including:

[0078] Step S21: Segment the time series image set into a focus area and a background area to create a segmentation result.

[0079] Step S22: extracting the image state of interest according to the segmentation result, and using the image state of interest to perform image enhancement, as follows: .

[0080] in, Characterize the enhanced image, is the brightness gain coefficient, is the brightness offset, Characterize the image state of interest of the original image, is the contrast gain coefficient, The minimum pixel value that represents the image. is the edge enhancement coefficient, Characterize edge-enhanced images.

[0081] Specifically, image segmentation algorithms such as threshold segmentation, region growing, semantic segmentation algorithms based on deep learning, or edge detection are used to divide each frame of the time series image set into a focus area and a background area to generate a segmentation result. The focus area is the body part undergoing rehabilitation action, while the background area is the surrounding environment. Image segmentation can focus on the image part that is meaningful to the rehabilitation action and reduce the interference of irrelevant information such as background.

[0082] According to the segmentation results, the image information related to the area of ​​interest is extracted from the original image, that is, the image state of interest. These image states of interest include average brightness, minimum pixel value, etc. Substitute the extracted image state information of interest into the formula Perform image enhancement to obtain the enhanced image .

[0083] Among them, the brightness gain coefficient and brightness offset Mainly affects the brightness of the image. Brightness gain coefficient Control the brightness gain by adjusting The brightness of the image can be changed to make the area of ​​interest easier to identify. , it will increase the brightness of the image, otherwise it will reduce the brightness. Brightness offset Directly in the original image state Brightness offset adjustment is performed based on . The minimum pixel value that represents the image. The difference between each pixel in the image and the minimum pixel value is calculated, and the contrast gain coefficient The size of determines the degree of contrast enhancement. By increasing the contrast, key details in the image can be made more obvious, especially the joint positions and motion trajectories. Edge-enhanced image It is an image obtained by edge detection algorithm. Edge enhancement coefficient Used to adjust the degree of edge enhancement in the image. The larger the value, the more obvious the edge enhancement effect.

[0084] When performing image enhancement, we first The brightness of the original image is increased by a constant offset , and then through the coefficient Adjust the brightness gain to ensure the image brightness is moderate. Subtract the minimum pixel value in the image and further enhance the contrast to make the details in the image more prominent, especially the user's joints and other key parts. Introducing edge-enhanced images , strengthening the edge of the image, making the boundaries of objects in the image clearer, helping to extract the boundaries of joints and bones in the image, thereby improving the visibility of important features in the image.

[0085] Through the above-mentioned image enhancement process, the quality of the image is improved, and the focus area is made clearer, so that the details in the focus area can be effectively extracted, ensuring that the features extracted from the image are accurate and reliable, and improving the accuracy and efficiency of the analysis.

[0086] Furthermore, the method described in the embodiment of the present application also includes:

[0087] Step S61: Obtain the user's rehabilitation log, and create an action warning according to the rehabilitation log.

[0088] Step S62: using the action warning to perform warning trigger recognition on the image annotation, and establishing a warning trigger recognition result.

[0089] Step S63: issuing a pre-warning alarm based on the pre-warning trigger recognition result.

[0090] Specifically, the rehabilitation log records various data of the user during the rehabilitation process, including action execution, physiological data, progress, etc. Collect and organize the user's rehabilitation log, including historical action data, treatment effects, doctor's evaluation, etc. Analyze the rehabilitation log to identify action patterns or trends that may cause problems, such as frequent incorrect actions, progress lags, etc. Based on the analysis results, create action warnings, which can be specific action standards, progress markers, or risk warnings.

[0091] During the image annotation process, the annotation results are compared with the action warning. All situations that meet the warning conditions are identified, such as action deviations exceeding the preset threshold, the occurrence of specific risk actions, etc. The specific circumstances of the warning trigger are recorded, and the warning trigger identification results are established, including the trigger time, type and related image annotation information.

[0092] According to the warning trigger identification results, the warning information that needs to be reported is determined. The warning information is conveyed to relevant personnel, such as rehabilitation doctors, therapists or users themselves, through the user interface, SMS, email or other notification methods. At the same time, a detailed explanation of the warning information and recommended response measures are provided to help relevant personnel take timely action.

[0093] The above steps constitute a complete early warning and feedback process. By analyzing the rehabilitation log to create action warnings, potential risks and problems can be identified in advance during the rehabilitation process. Using these warnings to trigger image annotations can monitor the quality and effectiveness of rehabilitation training in real time. Finally, through early warning alerts, it can ensure that relevant personnel are aware of the situation in a timely manner and take necessary measures, thereby improving the safety and effectiveness of rehabilitation training.

[0094] In summary, the image processing method for rehabilitation training motion analysis provided by the embodiment of the present application has the following technical effects:

[0095] The embodiment of the present application establishes a sample action image set and performs sample annotation, utilizes feature extraction and rehabilitation action fitting technology, and combines the time-series rehabilitation action fitting results to achieve accurate analysis and evaluation of rehabilitation training actions. The method captures the user's rehabilitation action joint image, distinguishes local and global images, and performs state extraction, enhancement and mapping processing to fully obtain action information. By establishing a global joint distribution identifier and performing feature recognition after local image annotation, the joint action information is accurately located. The user action time series set is constructed and annotated based on the previous time series rehabilitation action fitting results, which improves the accuracy of action analysis. In addition, the image is segmented into the focus area and the background area, and the image is enhanced based on the segmentation results, which further improves the image quality and the accuracy of feature recognition. Finally, by obtaining the user's rehabilitation log and creating an action warning, the early identification and early warning of potential risks in the rehabilitation training process are achieved, and the safety of rehabilitation training is improved. Overall, the embodiment of the present application can accurately analyze and evaluate the user's rehabilitation training actions, provide users with real-time rehabilitation feedback and training guidance, and greatly improve the accuracy, effectiveness and practical application value of rehabilitation training action analysis.

[0096] Embodiment 2, as Figure 4 As shown, the embodiment of the present application provides an image processing system for rehabilitation training action analysis, the system comprising:

[0097] A rehabilitation action fitting module 10 is used to establish a sample action image set, perform feature extraction on the sample action image set after sample annotation, establish a feature recognition channel, and use the sample action image set to perform rehabilitation action fitting to establish a time-series rehabilitation action fitting result, wherein the sample action image set is an image data set constructed by executing user image acquisition.

[0098] The user image capture module 20 is used to perform joint image capture of the user's rehabilitation movements and establish a joint time-series image set, wherein each time node image in the joint time-series image set includes a local image and a global image, and after image state extraction is performed on the time-series image set, adaptive image enhancement is performed, and a local image and global image mapping for each time node is established.

[0099] The image feature recognition module 30 is used to perform contour extraction on the enhanced global image, establish a global joint distribution identifier, and annotate the global joint distribution identifier to the local image according to the mapping, perform contour extraction on the local image, initialize the feature recognition channel based on the annotation of the global joint distribution identifier, and use the feature recognition channel to perform feature recognition on the contour extraction result to establish a feature recognition result.

[0100] The action timing set construction module 40 is used to construct a user action timing set based on the corresponding image timing after performing recognition enhancement on the feature recognition result.

[0101] The image annotation module 50 performs image annotation on the user action timing set by using the timing rehabilitation action fitting result.

[0102] Furthermore, the execution steps of the image feature recognition module 30 in the embodiment of the present application include:

[0103] Acquire the first acquisition coordinates of the global acquisition device, acquire depth data based on the enhanced global image, and construct the relative joint coordinates of the global joint according to the first acquisition coordinates and the depth data; acquire the second acquisition coordinates of the local acquisition device, and construct the relative local image coordinates according to the second acquisition coordinates and the depth data of the enhanced local image; use the first acquisition coordinates and the second acquisition coordinates to establish normalized coordinates, convert the relative joint coordinates and the local image coordinates into the same coordinates through the normalized coordinates, and then annotate the global joint distribution marker to the local image according to the mapping.

[0104] Furthermore, the execution steps of the image feature recognition module 30 in the embodiment of the present application also include:

[0105] Based on the annotated matching associated features of the global joint distribution identifier, an associated feature set is established, and the correlation degree of the associated feature set is used to establish a feature call attention; a temporary feature attention matching library of the feature recognition channel is established through the feature call attention to complete the initialization of the feature recognition channel; the feature recognition channel is used to perform feature matching recognition of the contour feature extraction result based on the feature attention matching library to establish a feature recognition result.

[0106] Furthermore, the feature recognition channel performs feature matching recognition of the contour feature extraction result based on the feature focus matching library, and the execution steps include:

[0107] Perform user limb posture recognition on the enhanced global image to establish a limb posture recognition result; input the first acquisition coordinates, the second acquisition coordinates, the device acquisition angle and the limb posture recognition result into a deformation fitting network to establish a deformation fitting result; establish a matching tolerance threshold according to the deformation fitting result, and configure a deformation bias; perform feature matching recognition of the contour feature extraction result based on the matching tolerance threshold and the deformation bias.

[0108] Furthermore, the execution steps of the image annotation module 50 in the embodiment of the present application include:

[0109] Obtain the action starting point of the sequential rehabilitation action fitting result, and perform timing alignment between the sequential rehabilitation action fitting result and the user action timing set based on the action starting point; perform frame-by-frame comparison between the sequential rehabilitation action fitting result and the user action timing set after timing alignment, and establish a deviation mark; use an inertial measurement unit to perform synchronous data collection of the user's rehabilitation action, and establish sequential speed data; perform a steady-state mark of the action according to the sequential speed data and the sequential rehabilitation action fitting result; and complete image annotation according to the deviation mark and the steady-state mark of the action.

[0110] Furthermore, the execution steps of the user image capture module 20 in the embodiment of the present application include:

[0111] The time series image set is segmented into a focus area and a background area to establish a segmentation result; the focus image state is extracted according to the segmentation result, and the focus image state is used to perform image enhancement as follows: .

[0112] in, Characterize the enhanced image, is the brightness gain coefficient, is the brightness offset, Characterize the image state of interest of the original image, is the contrast gain coefficient, The minimum pixel value that represents the image. is the edge enhancement coefficient, Characterize edge-enhanced images.

[0113] Furthermore, the execution steps of the system described in the embodiment of the present application also include:

[0114] The user's rehabilitation log is obtained, and an action warning is created according to the rehabilitation log; the image annotation is subjected to warning trigger recognition using the action warning to establish a warning trigger recognition result; and a pre-warning alarm is issued based on the warning trigger recognition result.

[0115] Through the above detailed description of the image processing method for rehabilitation training motion analysis in this specification, those skilled in the art can clearly understand the image processing system for rehabilitation training motion analysis in this embodiment. For the system disclosed in Example 2, since it corresponds to the method disclosed in Example 1 and has corresponding functional modules and beneficial effects, the relevant parts can be referred to the description of the method part.

[0116] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. An image processing method for rehabilitation training motion analysis, characterized in that: The method comprises: Establishing a sample action image set, performing sample annotation on the sample action image set, performing feature extraction, establishing a feature recognition channel, and using the sample action image set to perform rehabilitation action fitting, and establishing a time-series rehabilitation action fitting result, wherein the sample action image set is an image data set constructed by performing user image acquisition; Performing joint image capture of rehabilitation movements of the user to establish a joint time-series image set, wherein each time node image in the joint time-series image set includes a local image and a global image, performing image state extraction on the time-series image set, performing adaptive image enhancement, and establishing a mapping between the local image and the global image at each time node; Performing contour extraction on the enhanced global image, establishing a global joint distribution mark, and annotating the global joint distribution mark to the local image according to the mapping, performing contour extraction on the local image, initializing the feature recognition channel based on the annotation of the global joint distribution mark, and using the feature recognition channel to perform feature recognition on the contour extraction result to establish a feature recognition result; After the recognition enhancement of the feature recognition results, a user action time series set is constructed based on the corresponding image time series; Using the time-series rehabilitation action fitting result to perform image annotation on the user action time-series set; The step of labeling the global joint distribution identifier to the local image according to the mapping includes: Acquire a first acquisition coordinate of a global acquisition device, acquire depth data based on the enhanced global image, and construct a relative joint coordinate of a global joint according to the first acquisition coordinate and the depth data; Acquire a second acquisition coordinate of the local acquisition device, and construct a relative local image coordinate according to the second acquisition coordinate and depth data of the enhanced local image; The first acquisition coordinates and the second acquisition coordinates are used to establish normalized coordinates, the relative joint coordinates and the local image coordinates are converted into the same coordinates by the normalized coordinates, and then the global joint distribution identifier is marked on the local image according to the mapping.

2. The image processing method for rehabilitation training motion analysis according to claim 1, characterized in that: After the feature recognition channel is initialized based on the annotation of the global joint distribution identifier, the feature recognition channel is used to perform feature recognition on the contour extraction result to establish a feature recognition result, including: Based on the annotated matching associated features of the global joint distribution identifier, an associated feature set is established, and the association degree of the associated feature set is used to establish feature call attention; Establishing a temporary feature attention matching library of the feature recognition channel through the feature calling attention to complete the initialization of the feature recognition channel; The feature recognition channel is used to perform feature matching recognition of the contour feature extraction result based on the feature focus matching library to establish a feature recognition result.

3. The image processing method for rehabilitation training motion analysis according to claim 2, characterized in that: The feature matching recognition of the contour feature extraction result based on the feature recognition channel and the feature focus matching library includes: Performing user body posture recognition on the enhanced global image and establishing body posture recognition results; Inputting the first acquisition coordinates, the second acquisition coordinates, the device acquisition angle and the limb posture recognition result into a deformation fitting network to establish a deformation fitting result; Establishing a matching tolerance threshold according to the deformation fitting result and configuring a deformation bias; Feature matching recognition of the contour feature extraction result is performed based on the matching tolerance threshold and the deformation bias.

4. The image processing method for rehabilitation training motion analysis according to claim 1, characterized in that: The step of using the time sequence rehabilitation action fitting result to perform image annotation on the user action time sequence set includes: Acquire a motion starting point of a time-series rehabilitation motion fitting result, and perform timing alignment of the time-series rehabilitation motion fitting result and the user motion timing set based on the motion starting point; Comparing the time sequence rehabilitation action fitting result after time sequence alignment with the user action time sequence set frame by frame to establish a deviation mark; Use the inertial measurement unit to perform synchronous data collection of the user's rehabilitation movements and establish time-series velocity data; Performing movement steady-state identification according to the time-series speed data and the time-series rehabilitation movement fitting result; The image annotation is completed according to the deviation mark and the action steady-state mark.

5. The image processing method for rehabilitation training motion analysis according to claim 1, characterized in that: After extracting the image state of the time-series image set, performing adaptive image enhancement includes: Segmenting the time series image set into a focus area and a background area, and establishing a segmentation result; The image state of interest is extracted according to the segmentation result, and the image state of interest is used for image enhancement as follows: ； in, Characterize the enhanced image, is the brightness gain coefficient, is the brightness offset, Characterize the image state of interest of the original image, is the contrast gain coefficient, The minimum pixel value that represents the image. is the edge enhancement coefficient, Characterize edge-enhanced images.

6. The image processing method for rehabilitation training motion analysis according to claim 1, characterized in that: The method further comprises: Obtaining a user's rehabilitation log, and creating an action alert based on the rehabilitation log; Using the action warning to perform warning trigger recognition on the image annotation, and establishing a warning trigger recognition result; A pre-warning alarm is issued based on the pre-warning trigger identification result.

7. An image processing system for rehabilitation training motion analysis, characterized in that: The system is used to execute the image processing method for rehabilitation training motion analysis according to any one of claims 1 to 6, comprising: A rehabilitation action fitting module, wherein the rehabilitation action fitting module is used to establish a sample action image set, perform sample annotation on the sample action image set, perform feature extraction, establish a feature recognition channel, and use the sample action image set to perform rehabilitation action fitting to establish a time-series rehabilitation action fitting result, wherein the sample action image set is an image data set constructed by performing user image acquisition; A user image capture module, the user image capture module is used to perform rehabilitation action joint image capture on the user, establish a joint time-series image set, wherein each time node image in the joint time-series image set includes a local image and a global image, after image state extraction is performed on the time-series image set, adaptive image enhancement is performed, and a mapping between the local image and the global image at each time node is established; An image feature recognition module, the image feature recognition module is used to perform contour extraction on the enhanced global image, establish a global joint distribution mark, and annotate the global joint distribution mark to the local image according to the mapping, perform contour extraction on the local image, initialize the feature recognition channel based on the annotation of the global joint distribution mark, and use the feature recognition channel to perform feature recognition on the contour extraction result to establish a feature recognition result; An action timing set construction module, wherein the action timing set construction module is used to construct a user action timing set based on the corresponding image timing after performing recognition enhancement on the feature recognition result; An image annotation module, wherein the image annotation module uses the time sequence rehabilitation action fitting result to perform image annotation on the user action time sequence set; The execution steps of the image feature recognition module include: Acquire the first acquisition coordinates of the global acquisition device, acquire depth data based on the enhanced global image, and construct the relative joint coordinates of the global joint according to the first acquisition coordinates and the depth data; acquire the second acquisition coordinates of the local acquisition device, and construct the relative local image coordinates according to the second acquisition coordinates and the depth data of the enhanced local image; use the first acquisition coordinates and the second acquisition coordinates to establish normalized coordinates, convert the relative joint coordinates and the local image coordinates into the same coordinates through the normalized coordinates, and then annotate the global joint distribution marker to the local image according to the mapping.

Images