Lower limb motor function evaluation method, system and device based on deep learning image processing and storage medium

Abstract

The application discloses a lower limb motor function evaluation method, system and device based on deep learning image processing and a storage medium, and comprises the following steps: a common mobile device is used to shoot a patient's lower limb movement process and collect action images; image preprocessing is performed to obtain a preprocessed action image dataset; the action image dataset is input into a trained row-by-row mode model, feature points in a three-dimensional space of a joint are extracted, and parameters related to lower limb motor function are calculated according to the feature points; the parameters calculated in real time are fitted with set parameters to calculate a coincidence degree, real-time evaluation is performed on rehabilitation training according to the coincidence degree, and an optimized movement curve is given according to a real-time evaluation result to guide a patient to perform rehabilitation; through an optimized image processing algorithm, a row-by-row processing mode is used to replace a current full-image processing mode, evaluation and analysis on lower limb movement are realized under shooting of a common camera in a mobile device, and remote medical diagnosis is quickly and efficiently completed.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of medical imaging, and in particular to a lower limb motor function evaluation method, system and device based on deep learning image processing and a storage medium. BACKGROUND

[0002] Orthopedics is a discipline that studies the anatomy, physiology and pathology of the skeletal muscle system, and applies drugs, surgery and physical methods to maintain, develop and restore the normal form and function of this system. It is an important part of surgery. The biomechanical properties of the skeleton gradually increase during the adolescent development period, remain stable in adulthood, and gradually decline in the elderly, leading to a series of diseases such as osteoporosis and cervical spondylosis. Osteoarthritis (OA) is the most common orthopedic disease, with more than 300 million OA patients worldwide (Safiri Set al. Global, regional and national burden of osteoarthritis 1990-2017: a systematic analysis of the Global Burden of Disease Study 2017. Annals of the Rheumatic Diseases 2020; 79(6): 819-28.), of which there are about 130 million OA patients in China (China Osteoarthritis Diagnosis and Treatment Guidelines (2021 Edition). Chinese Journal of Orthopedics 2021; 41(18): 1291-314.).

[0003] Currently, the main technologies used in clinical diagnosis and screening of orthopedic diseases and rehabilitation reference are X-ray or digital X-ray, ultrasound, computed tomography (CT) and magnetic resonance imaging (MRI). X-ray is the most widely used diagnostic method and has been widely used in the screening, preoperative diagnosis, intraoperative navigation and postoperative evaluation of orthopedic diseases. Medical ultrasound examination is a medical imaging diagnostic technique based on ultrasound waves, which has the advantages of no radiation, non-invasive, millisecond-level imaging time, low price, and can collect morphological information of muscles, tendons, cartilage, and bone contours, and perform 3D data reconstruction. CT is a technique that uses precisely collimated X-rays and high-sensitivity detector arrays to complete cross-sectional scanning and 3D reconstruction around specific tissues and organs of the human body layer by layer, with fast scanning time and high image resolution (sub-millimeter level). It is currently an essential examination item in orthopedic clinics. MRI uses the principle of nuclear magnetic resonance and can achieve high-resolution imaging of human bones, muscles, ligaments, cartilage and other soft tissues by configuring different scanning sequence parameters.

[0004] With the continuous innovation of medical means, traditional image processing algorithms have achieved good results under limited computational complexity, to some extent, improving the clarity, resolution and shooting efficiency of diagnostic images. Homoscedastic and heteroscedastic Gaussian noise models provide smaller acceptance fields, so that the image denoising capability is enhanced (A. Foi, M. Trimeche, V. Katkovnik, and K. Egiazarian, "Practical Poissonian-Gaussian noise modeling and fitting for single-image raw data," IEEE Trans. Image Process., vol. 17, no. 10, pp. 1737-1754, Oct. 2008.). At the same time, based on the traditional image processing algorithm mainly based on Gaussian noise model, deep learning means has made great progress in advanced computer vision tasks such as classification, detection and tracking, as well as image processing such as denoising, super-resolution and white balance (K. Simonyan and A. Zisserman, "Very deep convolutional networks for large-scale image recognition," in Proc. Int. Conf. Learn. Represent. (ICLR), 2015, pp. 1-14.).

[0005] Currently, the image processing method for lower extremity shooting cannot be used on mobile devices, the main reasons are 1) high memory requirement and 2) large model size.

[0006] 1) High memory requirement:

[0007] Deep learning-based algorithms require huge data sets and acceptance fields to complete training and processing, and thus achieve better image processing functions. This full-image processing mode also requires the model to receive the entire image before processing, which requires a large amount of memory and a long training and processing time, making it difficult to use on mobile devices.

[0008] Lower extremity medical images involve a lot of bone and muscle information, and this information is crucial for doctors to diagnose diseases and track conditions. The huge amount of information has higher requirements for current image processing and generation algorithms, which require higher memory and longer time, and the full-image processing mode has already been difficult to meet the current demand for fast and convenient medical care.

[0009] 2) Large model size:

[0010] Deep learning based algorithms need to balance the improvement of performance and the complexity of the algorithm. For example, a modern commercial image signal processor has about 30-100 parameters, while a representative deep convolutional neural network UNet used in image processing has about 7.8 million parameters. The size of these models plus the huge computational complexity makes the size and power consumption of the required chip very large, which cannot be used on mobile devices.

[0011] Medical images have individual and dynamic differences, especially in scenarios such as mobile phones with lower resolution and insufficient configuration to support full image restoration. This difference needs to be compensated by algorithm performance improvement. This requires high algorithm complexity and performance, and at the same time brings the problem of excessive size and power consumption of the required chip, which cannot be used on mobile devices.

[0012] In summary, lower limb shooting and image processing are difficult to implement on mobile devices, and patients must go to the hospital for examination after symptoms appear, and cannot learn about their own problems in the first time. However, the waiting time required from shooting to finally receiving images and diagnosis opinions is very long when patients go to the hospital for examination at present, which greatly reduces the patient's experience of visiting the doctor. SUMMARY

[0013] The purpose of the present application is to provide a lower limb motor function evaluation method and device based on deep learning image processing.

[0014] The present application provides a lower limb motor function evaluation method based on deep learning image processing, comprising:

[0015] The patient's lower limb movement process is shot by a common mobile device, and the action image to be recognized is collected in real time;

[0016] The action image is pre-processed to optimize without increasing the image memory space, and a pre-processed action image data set is obtained;

[0017] The pre-processed action image data set is input into a trained row-by-row mode model, each frame of action image data is labeled in real time, the target parameters corresponding to the human joints and related parts in the image are identified and the feature points of the joint three-dimensional space are extracted, including the center of the hip joint, the center of the knee joint, the center of the ankle joint, the heel and the toe, the center of the patella and the forward and backward inclination angle of the pelvis;

[0018] The function parameters related to the lower limb motor function are calculated according to the feature points, and the target parameters include joint angle, gait cycle, gait stability, range of motion and symmetry analysis;

[0019] The function parameters calculated in real time are fitted with the set function parameters to calculate coincidence degree, real-time evaluation is performed on the rehabilitation training according to the coincidence degree, and an optimized motion curve is given according to the real-time evaluation result to guide the patient to perform rehabilitation, and finally rehabilitation progress monitoring is completed based on data statistics and tracking of a preset time.

[0020] The trained row-by-row mode model is based on deep learning image processing.

[0021] Preferably, the row-by-row mode model comprises an encoder, an information transmission module and a decoder connected in sequence, the encoder is used for preliminary convolution processing of original action image data to extract initial features;

[0022] The information transmission module comprises an ILCEM interline correlation extraction module and an IFM integrated feature module, the ILCEM interline correlation extraction module is used for extracting and processing correlation features between different processing lines in the action image to enhance context information in the image representation, and the IFM integrated feature module is used for integrating or fusing the features processed by the ILCEM interline correlation extraction module;

[0023] The decoder is used for generating output reconstruction image data from the features processed by the IFM integrated feature module. Preferably, the image preprocessing of the action image for optimization without increasing image memory space further comprises:

[0024] The action image containing the lower limb part is identified and filtered, and the filtered action image is classified according to the shooting part, which includes the knee joint, the hip joint, the ankle joint and all joints;

[0025] The required parameter analysis amount of output is selected, and the corresponding action image data is obtained, the parameter analysis amount includes joint angle, gait cycle, gait stability, motion range and symmetry analysis;

[0026] The action image data obtained is input into the model according to the selected parameter analysis amount, the file path is set, the local path and the cloud path are selected according to the requirement, the action image data is shot, and the image is grayed, geometrically transformed and enhanced, so that the optimization is performed without increasing the image memory space.

[0027] Preferably, the training of the trained row-by-row mode model comprises the following steps:

[0028] Before image processing, all the collected action image data is data segmented, the data segmentation refers to cutting into a series of horizontal lines according to rows, and the horizontal lines are converted from two-dimensional image information to one-dimensional sequence data for row-by-row processing;

[0029] The action image data after data segmentation is independently sent to the row-wise mode model at different times to reduce memory usage, that is, the row-wise mode model is represented as follows:

[0030] wherein, li / o n represents the nth row of input / output action image data, and LM is the row-wise mode model;

[0031] By the double buffering mechanism of the ILCEM module, the global and local context information is dynamically updated when processing each row of action image data, and the global and local information is dynamically fused by the IFM integrated feature module to enhance the image features.

[0032] Preferably, the double buffering mechanism of the ILCEM module dynamically updates the global and local context information when processing each row of action image data includes:

[0033] The action image data of the previous row is divided into two parts, global information G about the processed row and useful information U related to the current row, and this part of information is updated when the new row is input in order to capture the local features and global features in the action image, and further includes:

[0034] The size of the global information G and the useful information U related to the current row is limited to l lines;

[0035] When processing each row, the global information G is updated according to the last state of the useful information U related to the current row itself and the features x n of the current processing line / row to ensure that the global information G always contains the latest global context information;

[0036] When processing each row, the useful information U related to the current row is updated according to the last state of U, the current global information G, and the features x n of the current processing line / row to ensure that U always contains the latest local context information;

[0037] wherein, n represents the nth processing line of the action image data.

[0038] Preferably, the dynamic fusion of global and local information by the IFM integrated feature module to enhance the image features includes:

[0039] The features x n of the current processing line / row are input;

[0040] Strongly related information is integrated with the features of the current processing line / row by residual feature enhancement, and strong related spatial context information is injected into x n ;

[0041] The enhanced x n With the current useful information U n Cascade, process the cascaded features through convolution operation, and complete the entire integration process by retaining more original information through residual connection to enhance important features in the image.

[0042] The application also provides a lower limb motor function evaluation system based on deep learning image processing, which realizes the lower limb motor function evaluation method based on deep learning image processing as described above, and comprises:

[0043] A common mobile device is used to take pictures of the patient's lower limb movement process and collect real-time action images to be identified.

[0044] An image processing module is used to pre-process the action images to optimize them without increasing the image memory space, so as to obtain a pre-processed action image dataset; the pre-processed action image dataset is input into a trained row-by-row mode model to real-time label each frame of action image data, identify the target parameters corresponding to the human joints and related parts in the image, and extract the feature points of the joints in the three-dimensional space, including the center of the hip joint, the center of the knee joint, the center of the ankle joint, the heel and the toe, the center of the patella, and the forward and backward inclination angle of the pelvis.

[0045] A parameter calculation module is used to calculate the function parameters related to the lower limb motor function according to the feature points, and the target parameters include joint angle, gait cycle, gait stability, motion range, and symmetry analysis.

[0046] A data analysis module is used to fit and calculate the coincidence degree of the real-time calculated function parameters and the set function parameters, real-time evaluate the rehabilitation training according to the coincidence degree, and give an optimized motion curve to guide the patient to perform rehabilitation according to the real-time evaluation result, and finally complete the rehabilitation progress monitoring based on the data statistics and tracking of the preset time.

[0047] The trained row-by-row mode model is a row-by-row mode based on deep learning image processing.

[0048] The application also provides a computer device, which comprises:

[0049] A memory is used to store a processing program.

[0050] A processor is used to realize the lower limb motor function evaluation method based on deep learning image processing as described in the embodiments of the application when executing the processing program.

[0051] The application further provides a readable storage medium, characterized in that the readable storage medium stores a processing program, and the processing program is executed by a processor to realize the deep learning image processing-based lower limb motor function evaluation method.

[0052] The deep learning image processing-based lower limb motor function evaluation method provided by the application proposes a row-by-row mode of deep learning image processing, eliminates the need to store a large amount of intermediate data of the entire image, and completes low-memory, short-time, high-efficiency and high-resolution lower limb image reconstruction, so that high-quality image reconstruction can be completed based on a video shot by an ordinary consumer camera, and information can be obtained by a patient in a timely manner.

[0053] The application proposes two information processing modules, simplifies and optimizes the processing flow, reduces the size of the model, and shows good denoising and super-resolution effects, restores more details, and meets the accuracy requirements of clinical examination and diagnosis.

[0054] The application further analyzes the lower limb mechanics based on the reconstructed image, analyzes the joint disease progression, postoperative rehabilitation and the like of a patient based on common analysis quantities of joint physiological parameters, provides first-hand information of a disease of the patient, assists a doctor in remote diagnosis, and helps to optimize the operation mode and the rehabilitation scheme. BRIEF DESCRIPTION OF DRAWINGS

[0055] Figure 1 FIG. 1 is a schematic diagram of a deep learning image processing-based lower limb motor function evaluation system according to an embodiment of the application;

[0056] Figure 2 FIG. 2 is a schematic diagram of a deep learning image processing-based lower limb motor function evaluation method according to an embodiment of the application;

[0057] Figure 3 FIG. 3 is a row-by-row mode model architecture diagram according to an embodiment of the application;

[0058] Figures 4-7 FIG. 4 is a diagram of a parameter related to lower limb motor function according to an embodiment of the application. DETAILED DESCRIPTION

[0059] To make the objectives, technical solutions and advantages of the embodiments of the application clearer, the technical solutions in the embodiments of the application will be described below in connection with the drawings of the embodiments of the application. Obviously, the described embodiments are some but not all of the embodiments of the application. Based on the embodiments in the application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the protection scope of the application.

[0060] Embodiment One

[0061] The application provides a lower limb movement function evaluation method based on deep learning image processing, and is suitable for a lower limb movement function evaluation system based on deep learning image processing, that is, a portable device applied to lower limb movement video shooting and processing. The portable device is composed of a shooting device, an image processing module and a data analysis module. Referring to FIG. Figure 1 The shooting device is consistent with the shooting structure of mobile phones and cameras currently circulating in the market. The image processing module includes an encoder, a decoder and an information transmission module located in the middle. The encoder and the decoder perform simple convolution processing on the collected data. The information transmission module is composed of an inter-line correlation extraction module (ILCEM) and an integrated feature module (IFM). When implemented, the following steps are specifically included, as shown in FIG. Figure 2

[0062] S1: Shoot the lower limb movement process of a patient through a common mobile device to collect the action images to be identified in real time; knee joint surgery patients (anterior cruciate ligament reconstruction, UKA, TKA, etc.) collect lower limb rehabilitation training actions and image information data in real time through video shooting according to the guide instructions and standard training actions preset in the system;

[0063] S2: Optimize the action images under the premise of not increasing the image memory space through image preprocessing to obtain the preprocessed action image data set, which further includes:

[0064] Identify and filter the action images containing the lower limb part, and classify the filtered action images according to the shooting parts, including the knee joint, the hip joint, the ankle joint and all joints;

[0065] Select the required parameter analysis amount and obtain the corresponding action image data. The parameter analysis amount includes joint angle, gait cycle, gait stability, movement range and symmetry analysis, etc.

[0066] According to the selected parameter analysis amount, the action image data is input into the model for training, the file path is set, the local path and the cloud path are selected according to the demand, the action image data is grayed, geometrically transformed and image enhanced, so as to optimize under the premise of not increasing the image memory space.

[0067] In this embodiment, the action image data set is preprocessed to improve the accuracy and efficiency of subsequent processing. The preprocessing steps include denoising, enhancement, standardization and the like, which are described in detail as follows:

[0068] 1. The purpose of denoising is to reduce random noise in the image. These noises may come from sensors, interference in the transmission or storage process. Gaussian filtering method is used for denoising. ​where σ is the standard deviation, controlling the width of the Gaussian kernel.

[0069] 2. The enhanced image is intended to improve the visual effect of the image, making it more suitable for further processing or analysis, and the histogram equalization method is used for processing, L is the number of gray levels, h is the histogram of the image, p(i) is the frequency of the i-th gray level, and g(x) is the equalized gray value.

[0070] 3. The purpose of standardization is to scale the image data to a unified range for processing and comparison. f(x,y) is the original pixel value, g(x,y) is the normalized pixel value, min(f) and max(f) are the minimum and maximum pixel values of the image, respectively.

[0071] S3: input the pre-processed action image data set into the trained row-by-row mode model, and label each frame of action image data in real time, identify the target parameters corresponding to the human joints and related parts in the image, and extract the feature points of the joints in three-dimensional space, including the center of the hip joint, the center of the knee joint, the center of the ankle joint, the heel and the tip of the foot, the center of the patella, and the forward and backward inclination angle of the pelvis. This model can analyze image data frame by frame and identify key parts of the human body. Real-time labeling of each frame of image using the model can identify human joints and specific parts in the image, such as the center of the hip joint, the center of the knee joint, the center of the ankle joint, the heel and the tip of the foot, the center of the patella, and the forward and backward inclination angle of the pelvis.

[0072] In this embodiment, in order to eliminate the need to store a large amount of intermediate data of the entire image, complete low-memory, short-time, high-efficiency, high-resolution lower limb image reconstruction, and only based on the video shot by a general consumer-grade camera, high-quality image reconstruction can be completed, which is convenient for patients to obtain information in time. The row-by-row mode model includes an encoder, an information transmission module and a decoder connected in turn, the encoder is used for preliminary convolution processing of the original action image data to extract initial features;

[0073] The information transmission module includes an ILCEM inter-line correlation extraction module and an IFM integrated feature module, the ILCEM inter-line correlation extraction module is used for extracting and processing the correlation features between different processing lines in the action image, and enhancing the context information in the image representation, and the IFM integrated feature module is used for integrating or fusing the features processed by the ILCEM inter-line correlation extraction module;

[0074] The decoder is used to generate output reconstructed image data after the feature processed by the IFM integrated feature module. Those skilled in the art can understand that the row-by-row mode model is the composition structure of the image processing module described above, and the trained row-by-row mode model is a row-by-row mode based on deep learning image processing. In this embodiment, the training of the trained row-by-row mode model includes the following steps:

[0075] Step one, before image processing, all the action image data collected is subjected to data segmentation, which means that the data is segmented into a series of horizontal lines, and the horizontal lines are converted from two-dimensional image information to one-dimensional sequence data for row-by-row processing; this is the basic step of the row-by-row mode model, which aims to convert two-dimensional image information into one-dimensional sequence data for subsequent row-by-row processing, that is, before image processing, all image data needs to be cut according to each row, so that the whole image is divided into a series of horizontal lines. The specific implementation steps are as follows: 1. Segmentation by row: read the data of the whole image, and then cut according to each row to obtain a series of horizontal lines. Each line contains the pixel value of the row. 2. Convert to one-dimensional sequence: convert these horizontal lines into one-dimensional sequence data. For example, if a horizontal line has 100 pixel points, the line can be represented as a one-dimensional array with a length of 100. 3. Row-by-row processing: after obtaining these one-dimensional sequence data, the data can be processed row by row. This method is particularly suitable for tasks that can be processed independently row by row, such as line segmentation in text detection, OCR recognition of scanned documents, etc.

[0076] Step two, the action image data after data segmentation is independently sent to the row-by-row mode model at different times to reduce memory usage, that is, the row-by-row mode model is represented as follows:

[0077] Wherein, li / o n represents the nth row of input / output action image data, and LM is the row-by-row mode model.

[0078] Step three, through the double buffering mechanism of the ILCEM module, the global and local context information is dynamically updated when processing each row of action image data, and the global and local information is dynamically fused through the IFM integrated feature module to enhance the image features.

[0079] Referring to Figure 3 , the dynamic updating of global and local context information through the double buffering mechanism of the ILCEM module when processing each row of action image data in step three includes:

[0080] dividing the action image data of the previous line into two parts, a global information G about the processed line and a useful information U related to the current line, the partial information being updated when a new line is inputted in sequence to capture the local and global features in the action image, further comprising:

[0081] limiting the size of the global information G and the useful information U related to the current line to l lines;

[0082] updating the global information G according to the last state of the useful information U related to the current line itself and the feature x of the current processing line / row when processing each line to ensure that the global information G always contains the latest global context information; n

[0083] updating the useful information U related to the current line according to the last state of U, the current global information G and the feature x of the current processing line / row when processing each line to ensure that U always contains the latest local context information; n

[0084] wherein n represents the n-th processing line of the action image data.

[0085] ​​As can be understood by those skilled in the art, the core idea of the ILCEM (Incremental Learning-based Contextual Error Metric) module adopted in the embodiment is to more effectively utilize contextual information for error measurement and correction by maintaining and updating information of two key parts: global information (G) and useful information (U) related to the current line when processing each line of the image. The details are as follows: 1. Definition and function of global information (G): Global information (G) is a vector that stores the global feature information of the processed lines. It contains the comprehensive features of all lines from the top of the image to the current line. Size limit: The size of G is limited to l lines, which means it can only save the feature information of the last l lines. Update mechanism: When processing each line, G is updated according to the feature xn of the current line and the last state of U, ensuring that G always contains the latest contextual information. The update formula can be as follows: Gnew = f(Gold, Ulast, xn), where f is a function that combines the old global information, the last state of U Ulast and the feature xn of the current line. 2. Definition and function of useful information (U) related to the current line: Useful information (U) related to the current line is a vector that stores feature information directly related to the current line, focusing on the details and specific information of the current line. Size limit: The size of U is also limited to l lines, which means it can only save the feature information of the last l lines. Update mechanism: When processing each line, U is updated according to its last state, the current global information G and the feature xn of the current line, which ensures that U always contains the latest local contextual information. The update formula can be as follows: Unew = g(Uold, Gcurrent, xn), where g is a function that combines the old U state Uold, the current global information Gcurrent and the feature xn of the current line. The above ILCEM module can dynamically update global and local contextual information when processing each line of the image through the above double buffering mechanism (G and U), quickly respond to changes in the features of the current line, consider global and local information, make error measurement more accurate and robust, limit the size of G and U to l lines to avoid unlimited growth of memory, and improve computational efficiency.

[0086] Further, the step three of dynamically fusing global and local information through the IFM integrated feature module to enhance image features comprises:

[0087] Input the feature x of the current processing line / row n ;

[0088] Integrate strong correlation information with the feature of the current processing line / row to x n Inject strong related spatial context information;

[0089] The enhanced x n with the current useful information U n is concatenated, and the concatenated features are processed through convolution operations, and more original information is retained through residual connections to complete the entire integration process to enhance important features in the image. The concatenation operation is to concatenate two feature vectors in the channel dimension to form a new and richer feature representation. The concatenation operation: assuming that the feature dimension of x n is Cx, and the feature dimension of Un is CU, then the dimension of the concatenated feature vector will be Cx+CU. After completing the concatenation operation, the IFM module further processes the new feature vector through convolution operations and residual operations to realize the fusion and integration of features. Convolution operation: a convolution kernel (such as a 1x1 or 3x3 convolution kernel) is used to convolve the concatenated feature vector to capture the interaction information between different channels and generate a new feature representation. Residual operation: through residual connection (Residual Connection), the original feature is added to the convolved feature, which can retain more original information while introducing new context information.

[0090] The residual feature augmentation (RFA) used in this embodiment is a technique for improving feature representation by adding additional spatial context information to the original feature map to enhance the expressive power of the features. Specifically, the RFA module works in the following ways: injecting spatial context information: the RFA module injects spatial context information of different scales into the original feature map, thereby enriching the semantic content of the features. Reduce information loss: at the highest layer of the feature pyramid, due to the reduction of feature channels, information loss may occur. RFA adds residual connections to restore these lost information, thereby improving the integrity and accuracy of the features. By using the above residual feature augmentation, the IFM module can integrate more strongly related information into the current row of features xn. This means that xn not only contains its own local information, but also incorporates global and contextual information from different levels. Through the above IFM module, local and global information can be dynamically integrated when processing each row, improving the robustness and accuracy of feature representation, and is particularly suitable for tasks that require row-by-row processing of images.

[0091] S4: Calculate the function parameters related to the lower limb motor function according to the feature points, and the target parameters include joint angle, gait cycle, gait stability, range of motion and symmetry analysis; then according to the extracted feature points, calculate the parameters related to the lower limb motor function, as shown in Figures 4-7 , which is specifically explained as follows:

[0092] Joint angles:

[0093] Hip flexion / extension angle: calculated by comparing the position changes of the hip center and the knee center. The hip and knee centers are determined based on anatomical structures, and the center position changes after flexion / extension are calculated to determine the flexion / extension angle Knee flexion / extension angle: calculated by comparing the position changes of the knee center and the ankle center. The knee and ankle centers are determined based on anatomical structures, and the center position changes after flexion / extension are calculated to determine the flexion / extension angle where the horizontal position of the second posture x2 - the horizontal position of the first posture x1 = the difference Δx; the vertical position of the second posture y2 - the vertical position of the first posture y1 = the difference Δy.

[0094] Gait cycle:

[0095] Gait cycle time: calculated by tracking the movement of the foot tip in the gait cycle. Step length: calculated by measuring the horizontal movement distance of the heel or foot tip in consecutive gait cycles.

[0096] Gait stability:

[0097] Center of gravity movement: evaluated by analyzing the changes in the pelvic anteversion and retroversion angles. The anterior superior iliac spine, posterior superior iliac spine, and knee center are determined based on anatomical landmarks, and the angle between the center line of the anterior superior iliac spine and posterior superior iliac spine and the knee center is defined as the pelvic inclination angle. The range of this angle change can indicate the center of gravity movement and gait stability. Range of motion: calculated by tracking the maximum and minimum angles of the joint during movement. Range of motion = max(θ 髋 / 膝 / 踝 ). Symmetry analysis: evaluated by comparing the joint angles and gait parameters of both lower limbs. Rehabilitation progress monitoring: evaluated by tracking the changes in the above parameters over a long period of time to assess the effectiveness and progress of rehabilitation training.

[0098] S5: fitting the real-time calculated functional parameters with the set functional parameters to calculate the coincidence degree, real-time evaluating the rehabilitation training according to the coincidence degree, and giving an optimized motion curve to guide the patient to perform rehabilitation according to the real-time evaluation result;

[0099] In this embodiment, the physiological function of the photographed lower limb is analyzed by the data processing module based on the photographed and processed image information. The specific parameters include the hip center, knee center, ankle center, heel and foot tip, patella center, and pelvic anteversion and retroversion angles.

[0100] The specific operation steps of the lower limb motion function evaluation method based on deep learning image processing are realized by the portable device for lower limb motion video shooting and processing in this embodiment.

[0101] a. Use a mobile phone or camera to take front and side photos of the patient's lower extremities (below the pelvis to the bottom of both feet) as a whole, and require the patient to wear non-loose type trousers to expose both feet for easy identification.

[0102] b. Screen the photos taken, select representative photos, and the criteria are non-damaged photos, non-blurred photos, and non-incomplete photos, and input the qualified photos into the program.

[0103] c. Check the required output parameter analysis, including joint angle, gait cycle, gait stability, range of motion and symmetry analysis.

[0104] d. Train the selected photos to obtain the corresponding ILCEM module and IFM module.

[0105] e. Analyze the learning effect of the model, and repeatedly strengthen the training for the parameter mis-detection part.

[0106] f. After optimization, the optimized reconstructed lower extremity image and the specific data of the selected analysis and the corresponding clinical suggestions can be obtained, and the obtained data is synchronized to the cloud server in real time.

[0107] The present application optimizes the image processing algorithm, adopts a row-by-row processing mode instead of the current full image processing mode, develops a memory compression method, reduces the size of the model, maintains or even improves the performance, and integrates the two optimization algorithms in a product, which can realize the evaluation and analysis of lower extremity movement under the shooting of a general camera in a mobile device, and quickly and efficiently complete remote medical diagnosis.

[0108] In step S3 of the embodiment, the pre-processed action image dataset is input into the trained row-by-row model to label and extract the three-dimensional spatial feature points of the joints in real time, and calculate the parameters related to the lower limb motor function according to the feature points. This can be divided into the following key steps: 1. Data preprocessing: First, the action image dataset needs to be pre-processed to ensure that the data format and quality meet the input requirements of the model. Image cropping and standardization: crop the image to the appropriate size and perform standardization processing (such as normalizing pixel values). Noise removal: apply noise reduction algorithms (such as Gaussian filtering) to reduce noise in the image. Contrast enhancement: use histogram equalization and other methods to enhance the contrast of the image, making the joint features more obvious. 2. Model input: input the pre-processed image data into the trained row-by-row model. Read the image by row: input the image data of each row into the model for processing. Feature extraction: use the IFM module and other related modules of the model to extract useful feature information from each row of image. 3. Joint three-dimensional space feature point labeling: in the process of processing the image by row, label and extract the three-dimensional spatial feature points of the joints in real time. Two-dimensional feature point detection: use a deep learning model (such as a convolutional neural network) to detect the two-dimensional coordinates (x, y) of each joint in the image. Three-dimensional reconstruction: use multi-view geometry or deep learning methods to reconstruct the two-dimensional feature points into three-dimensional coordinates (x, y, z) in space. This may require data from a multi-camera system or depth sensor. 4. Calculate lower limb motor function parameters: calculate various parameters related to lower limb motor function according to the extracted three-dimensional spatial feature points of the joints. Joint angle calculation: calculate the angles of various joints such as the hip, knee, and ankle flexion angles through three-dimensional coordinates. Gait analysis: analyze each stage of the gait cycle (such as the stance phase and swing phase), calculate parameters such as step length and step frequency. Motion trajectory analysis: plot the motion trajectory of the joints and analyze their smoothness and stability. 5. Real-time feedback and adjustment: in order to achieve real-time performance, efficient algorithms and hardware support are needed to ensure that image processing and parameter calculation are completed within a short time.

[0109] In this embodiment, the fitting calculation of the real-time generated parameters and the set parameters in step S5 is performed, and the real-time evaluation of the rehabilitation training is performed according to the coincidence degree, and the motion curve is optimized according to the real-time evaluation result to guide the patient to perform rehabilitation. It can be divided into the following key steps: 1. Real-time parameter generation: first, the parameters related to the lower limb motor function are extracted and calculated in real time from the row-by-row mode model. Joint angle: the angle of each joint is calculated by the detected three-dimensional space feature points. Gait parameters: including step length, step frequency, etc. 2. Set parameters: set a set of ideal rehabilitation target parameters, which can be provided by doctors or rehabilitation experts according to the specific situation of the patient and the rehabilitation target. Ideal joint angle: set the ideal angle change range of each joint during the movement. Ideal gait parameters: set the ideal step length, step frequency, etc. 3. Fitting calculation of coincidence degree: fitting calculation is performed between the real-time generated parameters and the set parameters to obtain the coincidence degree. Mean square error (MSE): calculate the mean square error between the real-time parameters and the set parameters to evaluate the difference between them. Correlation coefficient (R 2 ): calculate the correlation coefficient between them to evaluate the linear correlation. 4. Real-time evaluation: real-time evaluation of rehabilitation training is performed according to the calculated coincidence degree. Evaluation criteria: set the threshold of coincidence degree, for example, when the coincidence degree is higher than a certain value, it is considered that the rehabilitation effect is good, and when it is lower than the value, it is considered that it needs to be adjusted. Feedback mechanism: according to the result of the coincidence degree, give the patient immediate feedback to help him adjust the action. 5. Optimize the motion curve: according to the real-time evaluation result, optimize the motion curve of the patient to guide him to perform more effective rehabilitation training. Adjust the motion trajectory: adjust the motion trajectory of the joint according to the evaluation result to make it closer to the ideal state. Dynamically adjust the difficulty: dynamically adjust the difficulty and intensity of the training according to the rehabilitation progress of the patient.

[0110] The fitting calculation of the real-time calculated functional parameters and the set functional parameters in S5, the real-time evaluation of the rehabilitation training according to the coincidence degree, and the optimization of the motion curve to guide the patient to perform rehabilitation according to the real-time evaluation result include:

[0111] The expected value of the set functional parameters according to the target of the rehabilitation training should match the rehabilitation target and reflect the rehabilitation progress of the patient;

[0112] Compare the real-time calculated functional parameters with the set functional parameters to calculate the coincidence degree between them, and use the Bezier curve to fit: where P i is the control point, n is the order of the curve, t is the parameter, is the binomial coefficient, where the shape of the fitted Bezier curve is determined by the control points, and the Bezier curve passes through the first and last control points to evaluate the proximity of the patient's action to the standard action. The higher the coincidence degree, the closer the patient's motor function is to the expected target.

[0113] According to the coincidence degree calculated by fitting, the rehabilitation training of the patient is evaluated in real time, the real-time evaluation result is obtained, and the optimized motion curve is generated to guide the patient to perform more effective rehabilitation training. This may involve adjusting the original motion trajectory to better meet the rehabilitation goal, feeding back the optimized motion curve to the patient, and further adjusting according to the actual performance of the patient.

[0114] Embodiment two

[0115] The application also provides a lower limb motor function evaluation system based on deep learning image processing, characterized in that the lower limb motor function evaluation method based on deep learning image processing is realized, which comprises:

[0116] A common mobile device is used to take pictures of the lower limb movement process of the patient and collect action images to be identified in real time.

[0117] An image processing module is used to pre-process the action images to optimize without increasing the image memory space, so as to obtain a pre-processed action image data set; the pre-processed action image data set is input into a trained row-by-row mode model, each frame of action image data is labeled in real time, the target parameters corresponding to the human joints and related parts in the image are identified and the feature points of the joints in three-dimensional space are extracted, including the center of the hip joint, the center of the knee joint, the center of the ankle joint, the heel and the toe, the center of the patella and the forward and backward inclination angle of the pelvis.

[0118] A parameter calculation module is used to calculate the function parameters related to the lower limb motor function according to the feature points, and the target parameters include joint angle, gait cycle, gait stability, motion range and symmetry analysis.

[0119] A data analysis module is used to calculate the coincidence degree of the function parameters calculated in real time and the set function parameters by fitting, to evaluate the rehabilitation training in real time according to the coincidence degree, and to give an optimized motion curve to guide the patient to perform rehabilitation according to the real-time evaluation result, and finally to complete the rehabilitation progress monitoring based on long-term data statistics and tracking.

[0120] The trained row-by-row mode model is based on deep learning image processing.

[0121] The specific content and implementation method of the above-mentioned modules are equivalent to the portable device for lower limb movement video shooting and processing as described above, which are as described in Embodiment one, and will not be repeated here.

[0122] Finally, in order to apply the above lower limb motor function evaluation method based on deep learning image processing to an image acquisition generation system or device or equipment with relevant hardware conditions, the present application also provides a computer readable storage medium, wherein the computer readable storage medium is loaded with a computer program, and the computer program is executed by a computer to realize the functions of the above corresponding method embodiments.

[0123] Similarly, the present application also protects a computer readable storage medium loaded with a computer program for implementing the lower limb motor function evaluation method based on deep learning image processing.

[0124] In the above embodiments, all or part of them can be realized by software, hardware, firmware or any combination thereof. When realized by software, all or part of them can be realized in the form of a computer program. The computer program includes one or more computer programs. When the computer program is loaded and executed on a computer, all or part of the processes or functions described in the embodiments of the present disclosure are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network or other programmable devices. The computer program can be stored in a computer readable storage medium or transmitted from one computer readable storage medium to another computer readable storage medium, for example, the computer program can be transmitted from one website, computer, server or data center to another website, computer, server or data center through wired (such as coaxial cable, optical fiber, DDL (Digital Dub DDriber Line, Digital Subscriber Line)) or wireless (such as infrared, wireless, microwave, etc.) mode. The computer readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server, data center, etc. integrated with one or more available media. The available medium can be a magnetic medium (such as a floppy disk, a hard disk, a magnetic tape), an optical medium (such as a high-density DVD (Digital Video DiDD, Digital Video Disk)) or a semiconductor medium (such as a DDD (olid Dtate DiDk, Solid State Disk)) and the like.

[0125] It should be noted that in the above-described embodiments, the terminology used herein is for the purpose of describing particular example embodiments only and is not intended to be limiting. As used herein, the singular forms "a", "an" and "the" can be intended to include the plural forms as well, unless the context clearly indicates otherwise. The terms "comprises", "comprising", "including" and "has" are inclusive and therefore specify the presence of stated features, integers, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof. The method steps, processes, and operations described herein are not to be construed as necessarily requiring their performance in the particular order in which they are discussed or illustrated, unless specifically identified as an order dependent step. It is also to be understood that additional or alternative steps can be employed.

[0126] It should be noted that the above-mentioned embodiments are merely used to illustrate the technical solutions of the present application, rather than limit the present application; even though the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that the technical solutions recorded in the foregoing embodiments can be modified, or some technical features can be replaced by equivalent features; and the modification or replacement does not cause the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A lower extremity motor function assessment method based on deep learning image processing, characterized in that, The application relates to a lower limb movement function real-time evaluation and rehabilitation training method based on a deep learning image processing row-by-row mode. The method comprises the following steps: a common mobile device is used to shoot a patient's lower limb movement process, and real-time collection of action images to be recognized is performed; image preprocessing is performed on the action images to optimize the images without increasing the image memory space, and a preprocessed action image dataset is obtained; the preprocessed action image dataset is input into a trained row-by-row mode model, real-time labeling of each frame of action image data is performed, target parameters corresponding to human body joints and related parts in the images are recognized and feature points of the joints in a three-dimensional space are extracted, including the centers of hip joints, knee joints and ankle joints, heels and toes, the center of a patella and a front-to-back inclination angle of a pelvis; function parameters related to lower limb movement functions are calculated according to the feature points, and the target parameters include joint angles, gait cycles, gait stability, movement ranges and symmetry analysis; the function parameters calculated in real time are fitted with set function parameters to calculate a coincidence degree, real-time evaluation of rehabilitation training is performed according to the coincidence degree, and an optimized movement curve is given according to a real-time evaluation result to guide a patient to perform rehabilitation, and finally, rehabilitation progress monitoring is completed based on data statistics and tracking of a preset time; wherein the trained row-by-row mode model is a row-by-row mode based on deep learning image processing; the row-by-row mode model comprises an encoder, an information transmission module and a decoder connected in sequence, the encoder is used for performing preliminary convolutional processing on original action image data to extract initial features; the information transmission module comprises an ILCEM interline correlation extraction module and an IFM integrated feature module, the ILCEM interline correlation extraction module is used for extracting and processing correlation features between different processing lines in the action images to enhance context information in image representation, and the IFM integrated feature module is used for integrating or fusing the features processed by the ILCEM interline correlation extraction module; the decoder is used for generating output reconstructed image data from the features processed by the IFM integrated feature module; 2. The deep learning image processing-based lower extremity motor function assessment method of claim 1, wherein, the ILCEM interline correlation extraction module dynamically updates global and local context information when processing each line of the image through a double buffering mechanism of global information G and useful information U related to the current line. the image preprocessing of the action images to optimize the images without increasing the image memory space further comprises the following steps: the action images containing lower limb parts are identified and filtered, and the filtered action images are classified according to shooting parts, the shooting parts including knee joints, hip joints, ankle joints and all joints; parameter analysis amounts required to be output are selected and corresponding action image data is acquired, the parameter analysis amounts including joint angles, gait cycles, gait stability, movement ranges and symmetry analysis; 3. The deep learning image processing-based lower extremity motor function assessment method of claim 1, wherein, the action image data acquired is input into a model for training according to the selected parameter analysis amounts, a file path is set, a local path and a cloud path are selected as required, and the action image data shot is subjected to grayscale processing, geometric transformation and image enhancement, so that the image memory space is optimized without being increased. the training of the trained row-by-row mode model comprises the following steps: Before image processing, all collected action image data is subjected to data segmentation, which refers to cutting into a series of horizontal lines, converting the horizontal lines from two-dimensional image information into one-dimensional sequence data for row-by-row processing; The action image data after data segmentation is independently sent to the row-by-row mode model at different times to reduce memory usage, that is, the row-by-row mode model is as follows: wherein, li / o n represents the input / output action image data of the nth row, LM is a line-wise mode model; Through the double buffering mechanism of the ILCEM module, the global and local context information is dynamically updated when processing each row of action image data, and the global and local information is dynamically fused through the IFM integrated feature module to enhance image features.

4. The deep learning image processing-based lower extremity motor function assessment method of claim 3, wherein, The double buffering mechanism of the ILCEM module dynamically updates the global and local context information when processing each row of action image data, which includes: The action image data of the previous row is divided into two parts, global information G about the processed row and useful information U related to the current row, and this part of information is updated when the new row is sequentially inputted, so as to capture the local features and global features in the action image, further including: The size of the global information G and the useful information U related to the current row is limited to I lines; At processing each row, the global information G is updated according to the last state of the useful information U itself related to the current row and the characteristics x of the current processing line / row n is updated to ensure that the global information G always contains the latest global context information; At processing each row, the useful information U related to the current row is updated according to the last state of U, the current global information G and the characteristics x of the current processing line / row n is updated to ensure that U always contains the latest local context information; Wherein, n represents the nth processing line of the action image data.

5. The deep learning image processing-based lower extremity motor function assessment method of claim 4, wherein, The IFM integrated feature module dynamically fuses the global and local information to enhance the image features, which includes: Input the feature x of the current processing line / row n ; By residual feature enhancement, strong correlation information is integrated with the feature set of the current processing line / row, to x n Injecting strong correlation spatial context information; The enhanced x n With the current useful information U n Cascade, process the cascaded features through convolution operation, and complete the whole integration process by retaining more original information through residual connection to enhance the important features in the image.

6. The deep learning image processing-based lower extremity motor function assessment method of claim 1, wherein, The fitting calculation coincidence degree of the real-time calculated functional parameters and the set functional parameters is calculated, the rehabilitation training is real-time evaluated according to the coincidence degree, and the optimized motion curve is generated to guide the patient to perform rehabilitation according to the real-time evaluation result, which includes: Setting the expected value of the functional parameters according to the target of the rehabilitation training; The function parameter calculated in real time is compared with the set function parameter, the coincidence degree between the two is calculated, and a Bezier curve is used for fitting: Wherein, P i is a control point, n is the order of the curve, t is a parameter, is a binomial coefficient, wherein the shape of the generated Bezier curve is determined by the control points, and the Bezier curve passes through the first and last control points to evaluate the closeness of the patient's action to the standard action; Real-time evaluating the rehabilitation training of the patient according to the coincidence degree obtained by fitting calculation, obtaining the real-time evaluation result and generating the optimized motion curve.

7. A lower extremity motor function assessment system based on deep learning image processing, characterized by, The lower limb motor function evaluation method based on deep learning image processing as claimed in any one of claims 1 to 6 is implemented, which includes: A general mobile device for photographing the patient's lower limb movement process and collecting the action image to be recognized in real time; An image processing module for image preprocessing of the action image to optimize without increasing the image memory space, obtaining a preprocessed action image data set; inputting the preprocessed action image data set into a trained row-by-row mode model, and real-time labeling each frame of action image data, Recognizing the target parameters corresponding to the human joints and related parts in the image and extracting the feature points of the joint three-dimensional space, including the hip joint center, the knee joint center, the ankle joint center, the heel and the toe, the patella center and the pelvic tilt angle; A parameter calculation module for calculating the functional parameters related to the lower limb motor function according to the feature points, the target parameters including joint angle, gait cycle, gait stability, motion range and symmetry analysis; The data analysis module fits the function parameters calculated in real time with the set function parameters to calculate the coincidence degree, evaluates the rehabilitation training in real time according to the coincidence degree, gives the optimized motion curve to guide the patient to perform rehabilitation according to the real-time evaluation result, and finally completes the rehabilitation progress monitoring based on the data statistics and tracking of the preset time. The trained row-by-row mode model is based on deep learning image processing.

8. A computer device, comprising: The method comprises the following steps: A memory is configured to store a processing program. A processor is configured to execute the processing program to implement the method for evaluating lower limb motor function based on deep learning image processing according to any one of claims 1 to 6.

9. A readable storage medium, characterized by, The readable storage medium stores a processing program, and the processing program is executed by a processor to implement the method for evaluating lower limb motor function based on deep learning image processing according to any one of claims 1 to 6.

Images