Lower limb motion function evaluation method, system and equipment based on deep learning image processing and storage medium

Abstract

The invention discloses a lower limb motion function evaluation method, system and device based on deep learning image processing and a storage medium, and the method comprises the steps: shooting a lower limb motion process of a patient through a common mobile device, and collecting a motion image; performing image preprocessing to obtain a preprocessed action image data set; inputting the motion image data set into the trained progressive mode model, extracting feature points of a joint three-dimensional space, and calculating parameters related to a lower limb motion function according to the feature points; the parameters obtained through real-time calculation and set parameters are subjected to fitting calculation to calculate the overlap ratio, rehabilitation training is evaluated in real time according to the overlap ratio, and an optimized motion curve is given according to the real-time evaluation result to guide a patient to conduct rehabilitation; an image processing algorithm is optimized, and a line-by-line processing mode is adopted to replace a current full-image processing mode, so that evaluation and analysis of lower limb movement under shooting of a common camera in a mobile device are realized, and remote medical diagnosis is completed quickly and efficiently.

Description

Technical Field

[0001] The present invention relates to the field of medical imaging technology, and in particular to a method, system, device and storage medium for evaluating lower limb motor function based on deep learning image processing. Background Art

[0002] Orthopedics is a discipline that studies the anatomy, physiology and pathology of the musculoskeletal system and uses drugs, surgery and physical methods to maintain, develop and restore the normal morphology and function of this system. It is an important part of surgery. The biomechanical properties of bones gradually increase during adolescence, remain stable after adulthood, and gradually decline as humans enter middle and old age, leading to a series of geriatric diseases such as osteoporosis and cervical spondylosis. Osteoarthritis (OA) is the most common orthopedic disease, with more than 300 million OA patients worldwide (Safiri Setal. Global, regional and national burden of osteoarthritis 1990-2017: a systematic analysis of the Global Burden of Disease Study 2017. Annals of the Rheumatic Diseases 2020; 79(6): 819-28.), of which about 130 million are OA patients in China (Guidelines for the diagnosis and treatment of osteoarthritis in China (2021 edition). Chinese Journal of Orthopedics 2021; 41(18): 1291-314.).

[0003] At present, the main technologies used for diagnosis, screening and rehabilitation reference of orthopedic diseases in clinical practice are X-ray or digital X-ray, ultrasound, computed tomography (CT) and magnetic resonance imaging (Magnetic Resonance Imaging). X-ray is the most widely used diagnostic method and has been widely used in screening, preoperative diagnosis, intraoperative navigation and postoperative evaluation of orthopedic diseases. Medical ultrasound is a medical imaging diagnostic technology based on ultrasound. Its advantages are no radiation, non-invasive, millisecond imaging time, low price, and can collect morphological information of muscle, tendon, cartilage and bone contour, and perform 3D data reconstruction. CT uses precisely collimated X-rays and a high-sensitivity detector array to synchronously complete cross-sectional scanning and 3D reconstruction layer by layer around specific tissues and organs of the human body. It has the characteristics of fast scanning time and high image resolution (sub-millimeter level). It is currently an indispensable examination item in orthopedic clinics. MRI uses the principle of nuclear magnetic resonance and can achieve high-resolution imaging of human bones and soft tissues such as muscles, ligaments, and cartilage by configuring different scanning sequence parameters.

[0004] With the continuous innovation of medical methods, traditional image processing algorithms have achieved good results with limited computational complexity, and have improved the clarity, resolution and shooting efficiency of diagnostic images to a certain extent. Homoscedastic and heteroscedastic Gaussian noise models provide a smaller receptive field, which enhances the ability of image denoising (A. Foi, M. Trimeche, V. Katkovnik, and K. Egiazarian, "Practical Poissonian-Gaussian noise modeling and fitting for single-image raw data," IEEE Trans. Image Process., vol. 17, no. 10, pp. 1737-1754, Oct. 2008.). At the same time, based on traditional image processing algorithms based on Gaussian noise models, deep learning methods have made significant progress in advanced computer vision tasks such as classification, detection, and tracking, as well as image processing such as denoising, super-resolution, and white balance (K. Simonyan and A. Zisserman, "Very deep convolutional networks for large-scale image recognition," in Proc. Int. Conf. Learn. Represent. (ICLR), 2015, pp. 1–14.).

[0005] Current image processing methods for lower limb images cannot be used on mobile devices due to 1) high memory requirements and 2) large model size.

[0006] 1) High memory requirements:

[0007] Deep learning-based algorithms require huge data sets and receptive fields to complete training and processing, and thus achieve better image processing capabilities. This full image processing mode also requires the model to receive all images before processing. It has a very large memory requirement and requires a long training and processing time, making it difficult to use on mobile devices.

[0008] Lower limb medical imaging involves a lot of bone and muscle information, which is crucial for doctors to diagnose and track diseases. The huge amount of information has higher requirements for current image processing and generation algorithms, requiring higher memory and longer time. The full image processing mode can no longer meet the current needs of fast and convenient medical treatment.

[0009] 2) Large model size:

[0010] Deep learning-based algorithms need to balance performance improvements and algorithm complexity. For example, a modern commercial image signal processor has about 30-100 parameters, while the representative deep convolutional neural network UNet used in image processing has about 7.8 million parameters. The size of these models, coupled with the huge computational complexity, makes the size and power consumption of the chips required large, making them unusable on mobile devices.

[0011] Medical images have individual and dynamic differences, especially in scenarios such as mobile phones where the resolution is low and the configuration is not sufficient to support full image restoration. This difference needs to be compensated by improving the algorithm performance. This requires the algorithm complexity and the performance provided to be high enough, and at the same time brings the problem that the size and power consumption of the required chip are too large to be used on mobile devices.

[0012] In summary, lower limb photography and image processing are difficult to implement on mobile devices. Patients must go to the hospital for examination after symptoms appear, and they cannot learn about their problems immediately. However, when patients go to the hospital for examination, the waiting time from the completion of the photography to the final receipt of the image and diagnosis is very long, which greatly reduces the patient's medical experience. Summary of the invention

[0013] The purpose of the present invention is to provide a method and device for evaluating lower limb motor function based on deep learning image processing.

[0014] The present invention provides a method for evaluating lower limb motor function based on deep learning image processing, comprising:

[0015] Use common mobile devices to shoot the patient's lower limb movement process and collect the action images to be identified in real time;

[0016] Performing image preprocessing on the action image to optimize the image without increasing the image memory space, thereby obtaining a preprocessed action image data set;

[0017] The preprocessed action image dataset is input into the trained line-by-line model, and each frame of action image data is annotated in real time to identify the target parameters corresponding to the human joints and related parts in the image and extract the feature points of the joints in three-dimensional space, including the center of the hip joint, the center of the knee joint, the center of the ankle joint, the heel and toes, the center of the patella, and the anterior-posterior tilt of the pelvis;

[0018] Calculating functional parameters related to lower limb motor function based on the feature points, the target parameters include joint angle, gait cycle, gait stability, range of motion and symmetry analysis;

[0019] The functional parameters calculated in real time are fitted with the set functional parameters to calculate the coincidence, and the rehabilitation training is evaluated in real time according to the coincidence. An optimized motion curve is given according to the real-time evaluation result to guide the patient to perform rehabilitation, and finally the rehabilitation progress monitoring is completed based on the data statistics and tracking of the preset time;

[0020] The trained line-by-line mode model is a line-by-line mode based on deep learning image processing.

[0021] Preferably, the line-by-line mode model comprises an encoder, an information transmission module and a decoder connected in sequence, wherein the encoder is used to perform preliminary convolution processing on the original action image data to extract initial features;

[0022] The information transmission module includes an ILCEM inter-line correlation extraction module and an IFM integrated feature module, wherein the ILCEM inter-line correlation extraction module is used to extract and process correlation features between different processed lines in the action image to enhance context information in the image representation, and the IFM integrated feature module is used to integrate or fuse features processed by the ILCEM inter-line correlation extraction module;

[0023] The decoder is used to generate and output reconstructed image data from the features processed by the IFM integrated feature module. Preferably, the image preprocessing of the action image to optimize without increasing the image memory space further includes:

[0024] Identifying and filtering the action images containing lower limbs, and classifying the filtered action images according to the shooting parts, wherein the shooting parts include knee joints, hip joints, ankle joints and all joints;

[0025] Select the parameter analysis quantity required for output and obtain the corresponding motion image data, wherein the parameter analysis quantity includes joint angle, gait cycle, gait stability, range of motion and symmetry analysis;

[0026] According to the selected parameter analysis quantity, the acquired action image data input model is trained, the file path is set, the local path and the cloud path are selected as required, and the captured action image data is grayed, geometrically transformed, and enhanced, so as to optimize it without increasing the image memory space.

[0027] Preferably, the training of the trained line-by-line mode model comprises the following steps:

[0028] Before image processing, all the collected action image data are segmented, and the data segmentation refers to segmenting into a series of horizontal lines by row, and converting the horizontal lines from two-dimensional image information into one-dimensional sequence data for line-by-line processing;

[0029] The action image data after data segmentation is sent to the line-by-line mode model independently at different times to reduce memory usage, that is, the line-by-line mode model is represented as follows:

[0030] Among them, li / o n represents the nth row of input / output action image data, LM is the row-by-row mode model;

[0031] The double buffering mechanism of the ILCEM module dynamically updates the global and local context information when processing each line of the action image data, and the IFM integrated feature module dynamically fuses the global and local information to enhance the image features.

[0032] Preferably, the dynamically updating global and local context information when processing each line of the action image data through the double buffering mechanism of the ILCEM module includes:

[0033] The motion image data of the previous row is divided into two parts: global information G about the processed row and useful information U related to the current row. The information is updated when new rows are input in sequence to capture local features and global features in the motion image, further comprising:

[0034] Limit the size of the global information G and the useful information U related to the current line to l lines;

[0035] When processing each row, the global information G is converted into the last state of the useful information U related to the current row and the feature x of the current processing line / row. n Update to ensure that the global information G always contains the latest global context information;

[0036] When processing each row, the useful information U related to the current row is calculated based on the last state of U, the current global information G and the feature x of the current processing line / row. n Update to ensure that U always contains the latest local context information;

[0037] Here, n represents the n-th processing line of the motion image data.

[0038] Preferably, the method of dynamically fusing global and local information to enhance image features through an IFM integrated feature module includes:

[0039] Input the feature x of the current processing line / row n ;

[0040] Through residual feature enhancement, strongly related information is integrated with the features of the current processing line / row, and x n Injecting strongly relevant spatial contextual information;

[0041] The enhanced x n Useful information with current n The cascade is performed, the cascaded features are processed through convolution operations, and more original information is retained through residual connections to complete the entire integration process to enhance the important features in the image.

[0042] The present invention also provides a lower limb motor function assessment system based on deep learning image processing, which implements the lower limb motor function assessment method based on deep learning image processing as described above, comprising:

[0043] Ordinary mobile devices are used to shoot the patient's lower limb movement process and collect the action images to be identified in real time;

[0044] An image processing module is used to perform image preprocessing on the action image to optimize it without increasing the image memory space, and obtain a preprocessed action image data set; input the preprocessed action image data set into the trained line-by-line mode model, annotate each frame of action image data in real time, identify the target parameters corresponding to the human joints and related parts in the image, and extract the feature points of the joint three-dimensional space, including the hip joint center, knee joint center, ankle joint center, heel and toe, patella center and pelvic anterior and posterior inclination;

[0045] A parameter calculation module, used to calculate functional parameters related to lower limb motor function according to the feature points, wherein the target parameters include joint angle, gait cycle, gait stability, range of motion and symmetry analysis;

[0046] The data analysis module calculates the coincidence between the function parameters calculated in real time and the set function parameters, evaluates the rehabilitation training in real time according to the coincidence, and provides an optimized motion curve to guide the patient to perform rehabilitation according to the real-time evaluation results, and finally completes the rehabilitation progress monitoring based on the data statistics and tracking of the preset time;

[0047] The trained line-by-line mode model is a line-by-line mode based on deep learning image processing.

[0048] The present invention also provides a computer device, comprising:

[0049] A memory, the memory being used to store a processing program;

[0050] A processor, when executing the processing program, implements the lower limb motor function assessment method based on deep learning image processing as described in an embodiment of the present invention.

[0051] The present invention also provides a readable storage medium, characterized in that a processing program is stored on the readable storage medium, and when the processing program is executed by a processor, a lower limb motor function assessment method based on deep learning image processing as described in an embodiment of the present invention is implemented.

[0052] The method for assessing lower limb motor function based on deep learning image processing provided by the present invention proposes a line-by-line mode of image processing based on deep learning, eliminates the need to store a large amount of intermediate data of the entire image, completes low-memory, short-time, high-efficiency, and high-resolution lower limb image reconstruction, and can complete high-quality image reconstruction based only on videos shot by ordinary consumer-grade cameras, making it easier for patients to obtain information in a timely manner;

[0053] The present invention simplifies and optimizes the processing flow, reduces the size of the model, and exhibits good denoising and super-resolution effects by proposing two information processing modules, thereby restoring more details and meeting the diagnostic accuracy requirements of clinical examinations.

[0054] The present invention conducts further preliminary analysis of lower limb mechanics by reconstructing images, and analyzes the progression of the patient's joint disease, postoperative rehabilitation, etc. based on common analysis quantities of joint physiological parameters, providing patients with first-hand disease information, while assisting doctors in remote diagnosis and helping to optimize surgical methods and rehabilitation plans. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] Figure 1 Schematic diagram of a lower limb motor function assessment system based on deep learning image processing in one embodiment of the present invention;

[0056] Figure 2 This is a schematic diagram of the steps of a method for assessing lower limb motor function based on deep learning image processing in one embodiment of the present invention;

[0057] Figure 3 A diagram showing a row-by-row model architecture in one embodiment of the present invention;

[0058] Figure 4-7 This is a specific example diagram of calculating parameters related to lower limb motor function in one embodiment of the present invention. DETAILED DESCRIPTION

[0059] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0060] Embodiment 1

[0061] The present invention provides a method for evaluating lower limb motor function based on deep learning image processing, which is applicable to a lower limb motor function evaluation system based on deep learning image processing, that is, a portable device for lower limb motion video shooting and processing, the portable device consists of three parts: a shooting device, an image processing module and a data analysis module, see Figure 1 As shown in the figure, the shooting device is consistent with the shooting structure of mobile phones and cameras currently on the market. The image processing module includes an encoder, a decoder and an information transmission module located in the middle. The encoder and decoder perform simple convolution processing on the collected data, and the information transmission module consists of an inter-line correlation extraction module (ILCEM) and an integrated feature module (IFM). The implementation specifically includes the following steps, such as Figure 2 As shown:

[0062] S1: Use common mobile devices to shoot the patient's lower limb movement process and collect the action images to be identified in real time; patients undergoing knee surgery (anterior cruciate ligament reconstruction, UKA, TKA, etc.) collect lower limb rehabilitation training movements and image information data in real time through video shooting according to the guidance instructions and standard training movements preset in the system;

[0063] S2: performing image preprocessing on the action image to optimize the image without increasing the image memory space, and obtaining a preprocessed action image data set, further comprising:

[0064] Identifying and filtering the action images containing lower limbs, and classifying the filtered action images according to the shooting parts, wherein the shooting parts include knee joints, hip joints, ankle joints and all joints;

[0065] Select the parameter analysis quantity required for output and obtain the corresponding motion image data, wherein the parameter analysis quantity includes optional analysis quantities such as joint angle, gait cycle, gait stability, range of motion and symmetry analysis;

[0066] According to the selected parameter analysis quantity, the acquired action image data input model is trained, the file path is set, the local path and the cloud path are selected as required, and the captured action image data is grayed, geometrically transformed, and enhanced, so as to optimize it without increasing the image memory space.

[0067] In this embodiment, the action image dataset is preprocessed to improve the accuracy and efficiency of subsequent processing. The preprocessing steps include denoising, enhancement, standardization and other operations, which are described in detail as follows:

[0068] 1. The purpose of denoising is to reduce random noise in the image. This noise may come from interference in the sensor, transmission or storage process. Gaussian filtering method is used for denoising. Where σ is the standard deviation, which controls the width of the Gaussian kernel.

[0069] 2. The purpose of enhancement is to improve the visual effect of the image and make it more suitable for further processing or analysis. The histogram equalization method is used for processing. L is the number of gray levels, h is the histogram of the image, p(i) is the frequency of the i-th gray level, and g(x) is the gray value after equalization.

[0070] 3. The purpose of normalization is to scale image data to a uniform range for ease of processing and comparison. f(x,y) is the original pixel value, g(x,y) is the normalized pixel value, min(f) and max(f) are the minimum and maximum pixel values ​​of the image respectively.

[0071] S3: Input the preprocessed action image data set into the trained line-by-line model, annotate each frame of action image data in real time, identify the target parameters corresponding to the human joints and related parts in the image, and extract the feature points of the joints in three-dimensional space, including the center of the hip joint, the center of the knee joint, the center of the ankle joint, the heel and toes, the center of the patella, and the anteroposterior tilt of the pelvis; this model can analyze the image data frame by frame and identify the key parts of the human body. Use the model to annotate each frame of the image in real time, and identify the human joints and specific parts in the image, such as the center of the hip joint, the center of the knee joint, the center of the ankle joint, the heel and toes, the center of the patella, and the anteroposterior tilt of the pelvis.

[0072] In this embodiment, in order to eliminate the need to store a large amount of intermediate data of the entire image, complete the reconstruction of lower limb images with low memory, short time, high efficiency and high resolution, high-quality image reconstruction can be completed based only on the video captured by an ordinary consumer-grade camera, so that patients can obtain information in a timely manner. The line-by-line mode model includes an encoder, an information transmission module and a decoder connected in sequence, and the encoder is used to perform preliminary convolution processing on the original action image data to extract initial features;

[0073] The information transmission module includes an ILCEM inter-line correlation extraction module and an IFM integrated feature module, wherein the ILCEM inter-line correlation extraction module is used to extract and process correlation features between different processed lines in the action image to enhance context information in the image representation, and the IFM integrated feature module is used to integrate or fuse features processed by the ILCEM inter-line correlation extraction module;

[0074] The decoder is used to generate and output reconstructed image data from the features processed by the IFM integrated feature module. Those skilled in the art can understand that the line-by-line mode model is the composition structure of the above-mentioned image processing module, and the trained line-by-line mode model is a line-by-line mode based on deep learning image processing. In this embodiment, the training of the trained line-by-line mode model includes the following steps:

[0075] Step 1: Before image processing, all the collected action image data are segmented. The data segmentation refers to segmenting into a series of horizontal lines by row, and converting the horizontal lines from two-dimensional image information into one-dimensional sequence data for row-by-row processing; this is the basic step of the row-by-row mode model, which aims to convert two-dimensional image information into one-dimensional sequence data for subsequent row-by-row processing. It means that before image processing, all image data must be segmented according to each row, so that the entire image is segmented into a series of horizontal lines. The specific implementation steps are as follows: 1. Segmentation by row: Read the data of the entire image, and then segment it according to each row to obtain a series of horizontal lines. Each line contains the pixel value of the row. 2. Convert to a one-dimensional sequence: Convert these horizontal lines into one-dimensional sequence data. For example, if a horizontal line has 100 pixels, then this line can be represented as a one-dimensional array with a length of 100. 3. Row-by-row processing: After obtaining these one-dimensional sequence data, these data can be processed row by row. This approach is particularly suitable for tasks that can be processed line by line independently, such as line segmentation in text detection, OCR recognition of scanned documents, etc.

[0076] Step 2: Send the segmented action image data to the line-by-line model independently at different times to reduce memory usage. The line-by-line model is represented as follows:

[0077] Among them, li / o n represents the nth row of input / output action image data, LM is the row-by-row mode model;

[0078] Step 3: When processing each line of the action image data, the global and local context information is dynamically updated through the double buffering mechanism of the ILCEM module, and the global and local information are dynamically fused through the IFM integrated feature module to enhance the image features.

[0079] See also Figure 3 As shown, in step 3, when processing each line of the action image data through the double buffering mechanism of the ILCEM module, dynamically updating the global and local context information includes:

[0080] The motion image data of the previous row is divided into two parts: global information G about the processed row and useful information U related to the current row. The information is updated when new rows are input in sequence to capture local features and global features in the motion image, further comprising:

[0081] Limit the size of the global information G and the useful information U related to the current line to l lines;

[0082] When processing each row, the global information G is converted into the last state of the useful information U related to the current row and the feature x of the current processing line / row. n Update to ensure that the global information G always contains the latest global context information;

[0083] When processing each row, the useful information U related to the current row is calculated based on the last state of U, the current global information G and the feature x of the current processing line / row. n Update to ensure that U always contains the latest local context information;

[0084] Here, n represents the n-th processing line of the motion image data.

[0085] Those skilled in the art can understand that the core idea of ​​the ILCEM (Incremental Learning-based Contextual Error Metric) module adopted in this embodiment is to more effectively utilize context information for error measurement and correction by maintaining and updating two key parts of information when processing each line of the image: global information (G) and useful information related to the current line (U). The detailed description is as follows: 1. Definition and function of global information (G): Global information (G) is a vector that stores the global feature information of the processed lines. It contains the comprehensive features of all lines from the top of the image to the current line. Size restriction: The size of G is limited to l lines, which means that it can only save the feature information of the last l lines. Update mechanism: When processing each line, G will be updated according to the feature xn of the current line and the last state of U, ensuring that G always contains the latest context information. The update formula may be as follows: Gnew = f (Gold, Ulast, xn), where f is a function used to combine the old global information, Gold, the last state Ulast of U, and the feature xn of the current line. 2. Useful information related to the current row (U) Definition and function: Useful information related to the current row (U) is a vector that stores feature information directly related to the current row, focusing on the details and specific information of the current row. Size restriction: The size of U is also limited to l lines, which means that it can only store feature information of the latest l lines. Update mechanism: When processing each line, U is updated based on its last state, the current global information G, and the feature xn of the current row, which ensures that U always contains the latest local context information. The update formula may be as follows: Unew = g(Uold, Gcurrent, xn), where g is a function used to combine the old U state Uold, the current global information Gcurrent, and the feature xn of the current row. The above ILCEM module can dynamically update the global and local context information when processing each line of the image through the above double buffering mechanism (G and U), and can quickly respond to the feature changes of the current row. It takes into account both global and local information, making the error metric more accurate and robust. By limiting the size of G and U to l lines, it avoids unlimited memory growth and improves computational efficiency.

[0086] Furthermore, in step 3, dynamically fusing global and local information through the IFM integrated feature module to enhance image features includes:

[0087] Input the feature x of the current processing line / row n ;

[0088] Through residual feature enhancement, strongly related information is integrated with the features of the current processing line / row, and x n Injecting strongly relevant spatial contextual information;

[0089] The enhanced x n Useful information with current n Cascade, process the cascaded features through convolution operations, and retain more original information through residual connections to complete the entire integration process to enhance the important features in the image. The cascade operation is to splice two feature vectors together according to the channel dimension to form a new, richer feature representation. Cascade operation: Assume x n The feature dimension of is Cx, and the feature dimension of Un is CU, then the dimension of the cascaded feature vector will be Cx+CU. After completing the cascade operation, the IFM module further processes the new feature vector through convolution operation and residual operation to achieve feature fusion and integration. Convolution operation: Use a convolution kernel (such as 1x1 or 3x3 convolution kernel) to convolve the cascaded feature vector to capture the interaction information between different channels and generate a new feature representation. Residual operation: Through the residual connection (ResidualConnection), the original features are added to the convolved features, which can retain more original information while introducing new contextual information.

[0090] The residual feature enhancement (RFA) used in this embodiment is a technology for improving feature representation, which enhances the expressiveness of features by adding additional spatial context information to the original feature map. Specifically, the RFA module works in the following ways: Injecting spatial context information: The RFA module injects spatial context information of different scales into the original feature map to enrich the semantic content of the feature. Reduce information loss: At the highest level of the feature pyramid, information may be lost due to the reduction of feature channels. RFA adds residual connections to make up for the lost information, thereby improving the integrity and accuracy of the features. The integrated strong related information adopted is enhanced through the above-mentioned residual features, and the IFM module can integrate more strong related information into the feature xn of the current row. This means that xn not only contains its own local information, but also incorporates global and contextual information from different levels. Through the above-mentioned IFM module, local and global information can be dynamically fused when processing each row, improving the robustness and accuracy of feature representation, which is particularly suitable for tasks that require image processing line by line.

[0091] S4: Calculate the functional parameters related to the motor function of the lower limbs according to the feature points, the target parameters include joint angle, gait cycle, gait stability, range of motion and symmetry analysis; then calculate the parameters related to the motor function of the lower limbs according to the extracted feature points, see Figure 4-7 The specific instructions are as follows:

[0092] Joint Angle:

[0093] Hip flexion / extension angle: calculated by comparing the position changes of the hip joint center and the knee joint center. Determine the hip and knee joint centers according to the anatomical structure, calculate the center position changes after flexion and extension, and the flexion and extension angles. Knee flexion / extension angle: calculated by comparing the position changes of the knee joint center and the ankle joint center. Determine the knee and ankle joint centers according to the anatomical structure, calculate the center position changes after flexion and extension, and the flexion and extension angle Among them, the horizontal position x2 of the second posture minus the horizontal position x1 of the first posture = the difference Δx; the vertical position y2 of the second posture minus the position y1 of the first vertical posture = the difference Δy.

[0094] Gait cycle:

[0095] Gait cycle time: Calculates the time of the gait cycle by tracking the movement of the toes during the gait cycle. Stride length: Calculates the horizontal distance moved by the heel or toes during consecutive gait cycles.

[0096] Gait stability:

[0097] Center of gravity shift: Gait stability is assessed by analyzing the changes in the anteroposterior tilt of the pelvis. The anterior superior iliac spine, posterior superior iliac spine and knee joint center are determined based on anatomical landmarks. The angle between the center of the anterior superior iliac spine and the posterior superior iliac spine and the center of the knee joint is defined as the pelvic tilt. The change in this angle range can explain the center of gravity shift and gait stability. Range of motion: The range of motion of the joint is calculated by tracking the maximum and minimum angles of the joint during movement. Range of motion = max(θ 髋 / 膝 / 踝 ). Symmetry analysis: evaluate the symmetry of gait by comparing the joint angles and gait parameters of the lower limbs on both sides. Rehabilitation progress monitoring: evaluate the effect and progress of rehabilitation training by tracking the changes of the above parameters over a long period of time.

[0098] S5: Fitting the function parameters calculated in real time with the set function parameters to calculate the coincidence, performing real-time evaluation on the rehabilitation training according to the coincidence, and providing an optimized motion curve according to the real-time evaluation result to guide the patient to perform rehabilitation;

[0099] In this embodiment, the data processing module analyzes the physiological functions of the photographed lower limbs based on the photographed and processed graphic information. The specific parameters include: hip joint center, knee joint center, ankle joint center, heel and toes, patella center and pelvic anterior and posterior tilt angle.

[0100] The specific operation steps of the method for evaluating lower limb motor function based on deep learning image processing in this embodiment are as follows:

[0101] a. Use a mobile phone or camera to take front and side photos of the patient's lower limbs (from below the pelvis to the soles of both feet). Ask the patient to wear loose pants and expose both feet for easy identification.

[0102] b. Screen the photos taken, select representative photos, and the standard is that the photos are not damaged, not blurred, and not incomplete. Input the photos that meet the requirements into the program.

[0103] c. Check the parameters you want to output, including joint angles, gait cycle, gait stability, range of motion, and symmetry analysis.

[0104] d. Train the selected photos to obtain the corresponding ILCEM module and IFM module.

[0105] e. Analyze the learning effect of the model and repeatedly strengthen the training of the parameter misdetection part.

[0106] f. After the optimization is completed, the optimized and reconstructed lower limb images, the specific data of the selected analysis volume and the corresponding clinical recommendations can be obtained, and the obtained data can be synchronized to the cloud server in real time.

[0107] The present invention optimizes the image processing algorithm, adopts the line-by-line processing mode to replace the current full image processing mode, and develops a memory compression method to reduce the size of the model while maintaining or even improving the performance. The two optimization algorithms are integrated into one product, which can realize the evaluation and analysis of lower limb movements under the capture of ordinary cameras in mobile devices, and complete remote medical diagnosis quickly and efficiently.

[0108] In this implementation, step S3 implements the input of the preprocessed action image data set into the trained line-by-line model, real-time annotation and extraction of feature points in the three-dimensional space of the joint, and calculation of parameters related to the lower limb motor function based on the feature points, which can be divided into the following key steps: 1. Data preprocessing: First, the action image data set needs to be preprocessed to ensure that the data format and quality are suitable for the input requirements of the model. Image cropping and standardization: The image is cropped to a suitable size and standardized (such as normalized pixel values). Noise removal: A denoising algorithm (such as Gaussian filtering) is applied to reduce noise in the image. Contrast enhancement: The contrast of the image is enhanced using methods such as histogram equalization to make the joint features more obvious. 2. Model input: The preprocessed image data is input into the trained line-by-line model line by line. Read the image line by line: The image data of each line is input into the model for processing. Feature extraction: Use the IFM module and other related modules of the model to extract useful feature information from each line of the image. 3. Joint three-dimensional space feature point annotation: In the process of processing the image line by line, the three-dimensional space feature points of the joint are annotated and extracted in real time. 2D feature point detection: Use deep learning models (such as convolutional neural networks) to detect the 2D coordinates (x, y) of each joint in the image. 3D reconstruction: Reconstruct the 2D feature points into coordinates (x, y, z) in 3D space through multi-view geometry or deep learning methods. This may require data from a multi-camera system or depth sensor. 4. Calculate lower limb motor function parameters: Calculate various parameters related to lower limb motor function based on the extracted 3D spatial feature points of the joints. Joint angle calculation: Calculate the angles of each joint such as the flexion and extension angles of the hip, knee, and ankle joints through 3D coordinates. Gait analysis: Analyze each phase of the gait cycle (such as stance phase and swing phase) and calculate parameters such as step length and step frequency. Motion trajectory analysis: Draw the motion trajectory of the joint and analyze its smoothness and stability. 5. Real-time feedback and adjustment: In order to achieve real-time performance, efficient algorithms and hardware support are required to ensure that image processing and parameter calculation are completed in a short time.

[0109] In this implementation, step S5 is used to fit the real-time generated parameters with the set parameters to calculate the degree of overlap, and to evaluate the rehabilitation training in real time based on the degree of overlap, and to optimize the motion curve based on the real-time evaluation results to guide the patient to perform rehabilitation. It can be divided into the following key steps: 1. Generate parameters in real time: First, it is necessary to extract and calculate the parameters related to the lower limb motor function from the line-by-line pattern model in real time. Joint angle: The angle of each joint is calculated by detecting the three-dimensional spatial feature points. Gait parameters: Including step length, step frequency, etc. 2. Set parameters: Set a set of ideal rehabilitation target parameters, which can be provided by doctors or rehabilitation experts based on the patient's specific situation and rehabilitation goals. Ideal joint angle: Set the ideal angle change range of each joint during the movement. Ideal gait parameters: Set the ideal step length, step frequency, etc. 3. Fit and calculate the degree of overlap: Fit the real-time generated parameters with the set parameters to obtain the degree of overlap. Mean square error (MSE): Calculate the mean square error between the real-time parameters and the set parameters to evaluate the difference between the two. Correlation coefficient (R 2 ): Calculate the correlation coefficient between the two and evaluate their linear correlation. 4. Real-time evaluation: Conduct real-time evaluation of rehabilitation training based on the calculated overlap. Evaluation criteria: Set a threshold for overlap. For example, when the overlap is higher than a certain value, the rehabilitation effect is considered good, and when it is lower than this value, it is considered that adjustment is required. Feedback mechanism: Based on the results of the overlap, give the patient instant feedback to help them adjust their movements. 5. Optimize the motion curve: Based on the real-time evaluation results, optimize the patient's motion curve and guide them to conduct more effective rehabilitation training. Adjust the motion trajectory: Based on the evaluation results, adjust the motion trajectory of the joint to make it closer to the ideal state. Dynamically adjust the difficulty: Dynamically adjust the difficulty and intensity of the training according to the patient's rehabilitation progress.

[0110] In S5, the functional parameters calculated in real time are fitted with the set functional parameters to calculate the coincidence, the rehabilitation training is evaluated in real time according to the coincidence, and an optimized motion curve is given according to the real-time evaluation result to guide the patient to perform rehabilitation, including:

[0111] The expected values ​​of functional parameters should be set according to the goals of rehabilitation training, which should match the rehabilitation goals and reflect the patient's rehabilitation progress;

[0112] Compare the function parameters calculated in real time with the set function parameters, calculate the overlap between the two, and use the Bezier curve for fitting: Among them, P i is the control point, n is the order of the curve, t is the parameter, is the binomial coefficient, where the shape of the Bezier curve generated by the fitting is determined by the control points. The Bezier curve passes through the first and last control points to evaluate the closeness between the patient's movement and the standard movement. The higher the overlap, the closer the patient's motor function is to the expected goal.

[0113] The patient's rehabilitation training is evaluated in real time based on the overlap calculated by fitting, and the real-time evaluation results are obtained and an optimized motion curve is generated to guide the patient to conduct more effective rehabilitation training. This may involve adjusting the original motion trajectory to better meet the rehabilitation goals, feeding back the optimized motion curve to the patient, and making further adjustments based on the patient's actual performance.

[0114] Embodiment 2

[0115] The present invention also provides a lower limb motor function assessment system based on deep learning image processing, characterized in that the lower limb motor function assessment method based on deep learning image processing as described above is implemented, including:

[0116] Ordinary mobile devices are used to shoot the patient's lower limb movement process and collect the action images to be identified in real time;

[0117] An image processing module is used to perform image preprocessing on the action image to optimize it without increasing the image memory space, and obtain a preprocessed action image data set; input the preprocessed action image data set into the trained line-by-line mode model, annotate each frame of action image data in real time, identify the target parameters corresponding to the human joints and related parts in the image, and extract the feature points of the joint three-dimensional space, including the hip joint center, knee joint center, ankle joint center, heel and toe, patella center and pelvic anterior and posterior inclination;

[0118] A parameter calculation module, used to calculate functional parameters related to lower limb motor function according to the feature points, wherein the target parameters include joint angle, gait cycle, gait stability, range of motion and symmetry analysis;

[0119] The data analysis module is used to fit the functional parameters calculated in real time with the set functional parameters to calculate the coincidence, conduct real-time evaluation of rehabilitation training based on the coincidence, and provide an optimized motion curve to guide the patient to perform rehabilitation based on the real-time evaluation results, and finally complete the rehabilitation progress monitoring based on long-term data statistics and tracking;

[0120] The trained line-by-line mode model is a line-by-line mode based on deep learning image processing.

[0121] The specific content and implementation method of the above-mentioned module are equivalent to the above-mentioned portable device for shooting and processing lower limb movement videos, which are as described in Example 1 and will not be repeated here.

[0122] Finally, in order to apply the above-mentioned lower limb motor function assessment method based on deep learning image processing to an image acquisition and generation system, device or equipment with relevant hardware conditions, the present application also provides a computer-readable storage medium, in which a computer program is loaded. When the computer program is executed by a computer, it realizes the functions of the corresponding method embodiment as described above.

[0123] Similarly, the present application also protects a computer-readable storage medium loaded with a computer program for implementing the lower limb motor function assessment method based on deep learning image processing.

[0124] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program. The computer program includes one or more computer programs. When the computer program is loaded and executed on a computer, the process or function described in the embodiment of the present disclosure is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer program can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer program can be transmitted from a website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, DDL (Digital DubDriberLine, digital subscriber line)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrations. The available medium may be a magnetic medium (eg, a floppy disk, a hard disk, a magnetic tape), an optical medium (eg, a high-density DVD (Digital Video DiDD)), or a semiconductor medium (eg, a solid-state drive (SSD)).

[0125] It should be noted that, in the above-mentioned embodiments, the terms used herein are only for describing specific exemplary embodiments, and are not intended to be restrictive. As used herein, the singular forms "one", "an" and "the or described" may be intended to also include plural forms, unless the context clearly indicates otherwise. The terms "include", "comprise" and "have" are inclusive, and therefore specify the existence of stated features, wholes, steps, operations, elements and / or parts, but do not exclude the existence or addition of one or more other features, wholes, steps, operations, elements, parts and / or their groups. The method steps, processes and operations described herein should not be interpreted as necessarily requiring the method steps, processes and operations to be performed in the specific order discussed or shown, unless it is specifically indicated that the steps are to be performed in the set order. It should also be understood that additional or alternative steps may be adopted.

[0126] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for evaluating lower limb motor function based on deep learning image processing, characterized in that: include: Use common mobile devices to shoot the patient's lower limb movement process and collect the action images to be identified in real time; Performing image preprocessing on the action image to optimize the image without increasing the image memory space, thereby obtaining a preprocessed action image data set; The preprocessed action image dataset is input into the trained line-by-line model, and each frame of action image data is annotated in real time to identify the target parameters corresponding to the human joints and related parts in the image and extract the feature points of the joints in three-dimensional space, including the center of the hip joint, the center of the knee joint, the center of the ankle joint, the heel and toes, the center of the patella, and the anterior-posterior tilt of the pelvis; Calculating functional parameters related to lower limb motor function based on the feature points, the target parameters include joint angle, gait cycle, gait stability, range of motion and symmetry analysis; The functional parameters calculated in real time are fitted with the set functional parameters to calculate the coincidence, and the rehabilitation training is evaluated in real time according to the coincidence. An optimized motion curve is given according to the real-time evaluation result to guide the patient to perform rehabilitation, and finally the rehabilitation progress monitoring is completed based on the data statistics and tracking of the preset time; The trained line-by-line mode model is a line-by-line mode based on deep learning image processing.

2. The method for evaluating lower limb motor function based on deep learning image processing according to claim 1, characterized in that: The line-by-line mode model includes an encoder, an information transmission module and a decoder connected in sequence, wherein the encoder is used to perform preliminary convolution processing on the original action image data to extract initial features; The information transmission module includes an ILCEM inter-line correlation extraction module and an IFM integrated feature module, wherein the ILCEM inter-line correlation extraction module is used to extract and process correlation features between different processed lines in the action image to enhance context information in the image representation, and the IFM integrated feature module is used to integrate or fuse features processed by the ILCEM inter-line correlation extraction module; The decoder is used to generate and output reconstructed image data from features processed by the IFM integrated feature module.

3. The method for evaluating lower limb motor function based on deep learning image processing as claimed in claim 1, characterized in that: The performing image preprocessing on the action image to optimize the action image without increasing the image memory space further comprises: Identifying and filtering the action images containing lower limbs, and classifying the filtered action images according to the shooting parts, wherein the shooting parts include knee joints, hip joints, ankle joints and all joints; Select the parameter analysis quantity required for output and obtain the corresponding motion image data, wherein the parameter analysis quantity includes joint angle, gait cycle, gait stability, range of motion and symmetry analysis; According to the selected parameter analysis quantity, the acquired action image data input model is trained, the file path is set, the local path and the cloud path are selected as required, and the captured action image data is grayed, geometrically transformed, and enhanced, so as to optimize it without increasing the image memory space.

4. The method for evaluating lower limb motor function based on deep learning image processing according to claim 1, characterized in that: The training of the trained line-by-line mode model comprises the following steps: Before image processing, all the collected action image data are segmented, and the data segmentation refers to segmenting into a series of horizontal lines by row, and converting the horizontal lines from two-dimensional image information into one-dimensional sequence data for line-by-line processing; The action image data after data segmentation is sent to the line-by-line mode model independently at different times to reduce memory usage, that is, the line-by-line mode model is represented as follows: Among them, li / o n represents the nth row of input / output action image data, LM is the row-by-row mode model; The double buffering mechanism of the ILCEM module dynamically updates the global and local context information when processing each line of the action image data, and the IFM integrated feature module dynamically fuses the global and local information to enhance the image features.

5. The method for evaluating lower limb motor function based on deep learning image processing as claimed in claim 4, characterized in that: The double buffering mechanism of the ILCEM module dynamically updates the global and local context information when processing each line of the action image data, including: The motion image data of the previous row is divided into two parts: global information G about the processed row and useful information U related to the current row. The information is updated when new rows are input in sequence to capture local features and global features in the motion image, further comprising: Limit the size of the global information G and the useful information U related to the current line to l lines; When processing each row, the global information G is converted into the last state of the useful information U related to the current row and the feature x of the current processing line / row. n Update to ensure that the global information G always contains the latest global context information; When processing each row, the useful information U related to the current row is calculated based on the last state of U, the current global information G, and the feature x of the current processing line / row. n Update to ensure that U always contains the latest local context information; Here, n represents the n-th processing line of the motion image data.

6. The method for evaluating lower limb motor function based on deep learning image processing as claimed in claim 5, characterized in that: The method of dynamically fusing global and local information to enhance image features through the IFM integrated feature module includes: Input the feature x of the current processing line / row n ; Through residual feature enhancement, strong related information is integrated with the features of the current processing line / row, and x n Injecting strongly relevant spatial contextual information; The enhanced x n Useful information with current n The cascade is performed, the cascaded features are processed through convolution operations, and more original information is retained through residual connections to complete the entire integration process to enhance the important features in the image.

7. The method for evaluating lower limb motor function based on deep learning image processing according to claim 1, characterized in that: The fitting and calculating the coincidence degree of the functional parameters calculated in real time and the set functional parameters, performing real-time evaluation on the rehabilitation training according to the coincidence degree, and providing an optimized motion curve to guide the patient to perform rehabilitation according to the real-time evaluation result includes: Set expected values ​​of functional parameters according to the goals of rehabilitation training; Compare the function parameters calculated in real time with the set function parameters, calculate the overlap between the two, and use the Bezier curve for fitting: Among them, P i is the control point, n is the order of the curve, t is the parameter, is the binomial coefficient, where the shape of the Bezier curve generated by the fitting is determined by the control points, and the Bezier curve passes through the first and last control points to evaluate the closeness of the patient's movement to the standard movement; The patient's rehabilitation training is evaluated in real time according to the coincidence degree obtained by fitting calculation, and the real-time evaluation result is obtained and an optimized motion curve is generated.

8. A lower limb motor function assessment system based on deep learning image processing, characterized in that: Implementing the method for evaluating lower limb motor function based on deep learning image processing as described in any one of claims 1 to 7, comprising: Ordinary mobile devices are used to shoot the patient's lower limb movement process and collect the action images to be identified in real time; The image processing module is used to perform image preprocessing on the action image to optimize it without increasing the image memory space, and obtain a preprocessed action image data set; input the preprocessed action image data set into the trained line-by-line mode model, and perform real-time annotation on each frame of action image data, - Identify the target parameters corresponding to the human joints and related parts in the image and extract the feature points of the joints in three-dimensional space, including the center of the hip joint, the center of the knee joint, the center of the ankle joint, the heel and toes, the center of the patella and the anterior-posterior tilt of the pelvis; A parameter calculation module, used to calculate functional parameters related to lower limb motor function according to the feature points, wherein the target parameters include joint angle, gait cycle, gait stability, range of motion and symmetry analysis; The data analysis module calculates the coincidence between the function parameters calculated in real time and the set function parameters, evaluates the rehabilitation training in real time according to the coincidence, and provides an optimized motion curve to guide the patient to perform rehabilitation according to the real-time evaluation results, and finally completes the rehabilitation progress monitoring based on the data statistics and tracking of the preset time; The trained line-by-line mode model is a line-by-line mode based on deep learning image processing.

9. A computer device, characterized in that: include: A memory, the memory being used to store a processing program; A processor, wherein when executing the processing program, the processor implements the lower limb motor function assessment method based on deep learning image processing as described in any one of claims 1 to 7.

10. A readable storage medium, characterized in that: The readable storage medium stores a processing program, and when the processing program is executed by the processor, the method for evaluating lower limb motor function based on deep learning image processing as described in any one of claims 1 to 7 is implemented.

Images