Image processing method and device for scoliosis training, medium and product

Abstract

The embodiment of the invention discloses an image processing method and device for scoliosis training, a medium and a product, and belongs to the technical field of image processing. The method comprises the following steps: in response to a first trigger operation, determining a scoliosis identifier of a target object, and displaying a reference video corresponding to the scoliosis identifier of the target object; acquiring a training video frame corresponding to the current reference video frame; determining a plurality of reference triangles corresponding to the demonstration action displayed by the current reference video frame and a plurality of training triangles corresponding to the current training action in the training video frame; determining target similarities between the plurality of reference triangles and corresponding triangles in the plurality of training triangles, wherein the target similarities comprise orientation similarities and shape similarities; and according to all the target similarities, determining and displaying training evaluation information corresponding to the current training action. According to the technical scheme, the scoliosis training experience of the user can be remarkably improved.

Description

TECHNICAL FIELD

[0001] Embodiments of the present application relate to the field of image processing, and particularly to an image processing method, device, medium and product for scoliosis training. BACKGROUND

[0002] Scoliosis is a three-dimensional deformity characterized by coronal plane deviation, abnormal sagittal plane physiological curvature, and vertebral rotation. Scoliosis can cause body asymmetry (shoulder height difference, pelvic tilt, hump / flat back), impaired posture and balance control, back pain, decreased exercise tolerance, and respiratory function decline and psychosocial impact (body anxiety, decreased quality of life) in some patients. For patients in the growth and development period, the curvature has the risk of progression; for adult patients, degenerative changes and pain / functional limitations are more prominent.

[0003] Mild to moderate curvature usually requires brace treatment and "scoliosis-specific rehabilitation training". Existing scoliosis-specific rehabilitation training is mainly completed offline by a specialist doctor / therapist, who formulates a prescription based on imaging (such as X-ray Cobb angle), body observation, and functional assessment, and implements it through face-to-face guidance. This results in high costs for patients to perform scoliosis-specific rehabilitation training and relatively poor experience. SUMMARY

[0004] The present application provides an image processing method, device, medium and product for scoliosis training to improve the user's experience level during scoliosis training.

[0005] According to an aspect of the present application, an image processing method for scoliosis training is provided, the method comprising:

[0006] In response to a first trigger operation, determining a scoliosis identifier of a target object, and displaying a reference video corresponding to the scoliosis identifier of the target object, the scoliosis identifier being determined based on scoliosis typing and orientation information, the scoliosis typing and the orientation information being determined based on a first spine X-ray image of the target object;

[0007] During the display of the reference video, obtaining a training video frame corresponding to a current reference video frame;

[0008] Taking the center of the double hip line as the origin and the distance between the double hips as the unit length, determining a plurality of reference triangles corresponding to a demonstration action shown in the current reference video frame and a plurality of training triangles corresponding to a current training action in the training video frame, the triangles including three key points one-to-one corresponding to the vertices, the three key points including three consecutive joints or including a predetermined position of the head and two shoulder joints;

[0009] determine target similarities between the plurality of reference triangles and corresponding triangles in the plurality of training triangles, the target similarities comprising orientation similarities and shape similarities;

[0010] determine and display training evaluation information corresponding to the current training action according to all the target similarities.

[0011] According to another aspect of the present application, there is provided an image processing device for scoliosis training, the device comprising:

[0012] a response module configured to, in response to a first trigger operation, determine a scoliosis identification of a target object, and display a reference video corresponding to the scoliosis identification of the target object, the scoliosis identification being determined based on scoliosis typing and orientation information, the scoliosis typing and the orientation information being determined based on a first spine X-ray image of the target object;

[0013] a training video frame module configured to, during display of the reference video, obtain a training video frame corresponding to a current reference video frame;

[0014] a triangle module configured to, with a center of a double hip line as an origin and with a double hip distance as a unit length, determine a plurality of reference triangles corresponding to a demonstration action displayed by the current reference video frame and a plurality of training triangles corresponding to a current training action in the training video frame, the triangles comprising vertices corresponding to three key points, the three key points comprising three consecutive joints or a predetermined position of a head and two shoulder joints;

[0015] a similarity module configured to determine target similarities between the plurality of reference triangles and corresponding triangles in the plurality of training triangles, the target similarities comprising orientation similarities and shape similarities;

[0016] a feedback module configured to determine and display training evaluation information corresponding to the current training action according to all the target similarities.

[0017] According to another aspect of the present application, there is provided an electronic device, the electronic device comprising:

[0018] one or more processors;

[0019] a storage device configured to store one or more programs,

[0020] when the one or more programs are executed by the one or more processors, the one or more processors implement the image processing method for scoliosis training as described in any of the embodiments of the present application.

[0021] According to another aspect of the present application, there is provided a computer readable storage medium storing computer instructions for causing a processor to implement the image processing method for scoliosis training according to any of the embodiments of the present application when executed.

[0022] According to another aspect of the present application, there is provided a computer program product implementing the image processing method for scoliosis training according to any of the embodiments of the present application when executed by a processor.

[0023] The technical solution of the embodiments of the present application displays the reference video corresponding to the scoliosis identification of the target object, so that the user can follow the correct reference video to perform the scoliosis training. During the display of the reference video, the training video frame corresponding to the current reference video frame is obtained, and the multiple reference triangles corresponding to the demonstration action displayed by the current reference video frame and the multiple training triangles corresponding to the current training action in the training video frame are determined with the center of the double hip line as the origin and the distance between the double hips as the unit length, so as to eliminate the difference in triangle similarity caused by the body proportion. The training evaluation information corresponding to the current training action is determined and displayed through the target similarity between the corresponding triangles in the multiple reference triangles and the multiple training triangles, so as to achieve the technical effect of real-time feedback of the accuracy of the training action, so that the user can know the completion degree of the action in time. In the case that the completion degree of the training action is low, the training action performed by the user is adjusted in time according to the training evaluation information, so that the user only needs to rely on the mobile phone camera to collect the video stream during the training, and the real-time action quality evaluation and correction of the skeleton are completed at the end side, thereby realizing the low-cost, privacy-friendly, standardized, interpretable and high-compliance home rehabilitation closed loop.

[0024] It should be understood that the content described in this part is not intended to identify key or important features of the embodiments of the present application, nor is it used to limit the scope of the present application. Other features of the present application will become apparent from the following description. BRIEF DESCRIPTION OF DRAWINGS

[0025] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0026] Figure 1 The flowchart of the image processing method for scoliosis training provided by the embodiments of the present application is shown in the following figure:

[0027] Figure 2 The scoliosis classification determination process provided by the embodiments of the present application is shown in the following figure:

[0028] Figure 3 Another flowchart of the image processing method for scoliosis training provided by the embodiment of the present application is shown in the figure;

[0029] Figure 4 The structure diagram of the image processing device for scoliosis training provided by the embodiment of the present application is shown in the figure;

[0030] Figure 5 The structure diagram of an electronic device provided by the embodiment of the present application is shown in the figure. DETAILED DESCRIPTION

[0031] In order to make the personnel in the technical field better understand the present application scheme, the technical scheme in the embodiment of the present application will be described clearly and completely in the following with reference to the drawings in the embodiment of the present application. Obviously, the described embodiment is only a part of the embodiment of the present application, not all. Based on the embodiment in the present application, all other embodiments obtained by the person skilled in the art without creative labor should belong to the scope of protection of the present application.

[0032] It should be noted that the terms "first", "second" and the like in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and do not necessarily describe a specific order or sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device including a series of steps or units does not necessarily limit to those steps or units clearly listed, but can include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0033] Figure 1 The flowchart of the image processing method for scoliosis training provided by the embodiment of the present application is shown in the figure. The embodiment can be applicable to the case that the user completes the scoliosis training guided by the terminal and the training completion degree is fed back in real time. The method can be executed by the corresponding image processing device for scoliosis training, which can be realized in the form of hardware and / or software, and can be configured in an electronic device such as a computer or a server. As shown in the figure, the method of the embodiment includes: Figure 1

[0034] ​S110, in response to the first trigger operation, determining a scoliosis identifier of the target object, and displaying a reference video corresponding to the scoliosis identifier of the target object, the scoliosis identifier being determined based on scoliosis typing and orientation information, the scoliosis typing and orientation information being determined based on the first spine X-ray image of the target object.

[0035] The first trigger operation can be an operation of triggering the playing of the reference video by clicking or touching a corresponding control.

[0036] Specifically, in response to the first trigger operation for the first spine X-ray image, the scoliosis typing and orientation information corresponding to the first spine X-ray image are determined, and the scoliosis identifier is determined according to the scoliosis typing and orientation information. Therefore, the scoliosis identifier in this embodiment is determined based on the first spine X-ray image, and thus has high accuracy.

[0037] The scoliosis typing can be understood as the type of scoliosis. The orientation information is used to determine the spatial direction (left convex or right convex) and the rehabilitation priority (primary or secondary) of the scoliosis. For example, for a type S scoliosis, the orientation information is chest left and waist right, and the waist is the primary scoliosis.

[0038] In one embodiment, the first spine X-ray image is analyzed based on a pre-trained neural network model to obtain the scoliosis identifier. Specifically, the pre-trained neural network model is trained to extract the vertebral body (see the vertebral body center in Figure 2 ), the endplate, and the spinal midline (see the red line in Figure 2 ) from the first spine X-ray image, and then calculate the Cobb angle, the apex position, and the apex cone rotation angle based on the vertebral body, the endplate, and the spinal midline, and determine the scoliosis typing and orientation information based on all the determined parameter data, and then determine the scoliosis identifier according to the scoliosis typing and orientation information. In this embodiment, the labels of the training samples used in the training process of the neural network model are determined by professional doctors.

[0039] In an embodiment, in response to the scoliosis identification determination operation, a training list is generated, the training list including one or more candidate reference videos; in response to a candidate reference video selection operation, the selected reference video is displayed. This embodiment displays the reference video required by the user in an interactive manner, which helps to improve the user experience. For example, the scoliosis identification corresponds to an S-type scoliosis, the directional information is chest left and waist right, the waist bend is the main lateral bend, and the chest bend is the secondary lateral bend. The training list includes A sitting respiration, which corresponds to the code A1 sitting respiration; B sitting translation, which corresponds to the code B1 chest left and waist right; C lying, which corresponds to the code C1 chest right and waist left; D kneeling, which corresponds to the code D1 chest left and waist right; E wooden rack translation, which corresponds to the code E1 chest left and waist right; F side rack translation, which corresponds to the code F1 chest left and waist right; G side rack standing, which corresponds to the code G1 chest left and waist right; I ground kneel, which corresponds to the code I1 chest left and waist right; K double bend standing balance, which corresponds to the code K1 chest left and waist right.

[0040] In an embodiment, before S110 is performed, in response to an uploading operation for a scoliosis identification diagnosis report, the scoliosis identification diagnosis report is analyzed to determine the scoliosis identification of the target object, and then a reference video matching the scoliosis identification is determined as the reference video for the target object. This embodiment receives the scoliosis identification of the target object through the terminal, and determines the reference video matching the target object based on the scoliosis identification, thereby ensuring the accuracy of the matching of the reference video and the scientific nature of the scoliosis training of the target object based on the reference video.

[0041] In an embodiment, in response to the scoliosis identification determination operation, individual parameter data of the target object is obtained, the training intensity and frequency are determined according to the scoliosis identification and the individual parameter data, and the current training list is generated according to the training intensity and frequency. The individual parameter data includes age, bone age, Cobb range, pelvic tilt, etc.

[0042] S120, during the display of the reference video, a training video frame corresponding to a current reference video frame of the reference video is obtained.

[0043] During the playing of the reference video, the training video of the target object following the reference video is collected in real time through the camera. For the current reference video frame of the reference video, the latest training video frame can be selected as the training video frame corresponding to the current reference video frame based on the time alignment principle, so as to ensure the accuracy of the real-time acquisition of the training action of the target object; or the training video frame corresponding to the current reference video frame is determined based on the action matching and the predetermined time range. For example, a predetermined number of training video frames within a predetermined time range from the current time are matched with the current reference video frame, and the training video frame with the highest matching degree is selected as the training video frame corresponding to the current reference video frame.

[0044] S130, determining, with the bicondylar center as the origin and the bicondylar distance as the unit length, a plurality of reference triangles corresponding to the demonstration action shown in the current reference video frame and a plurality of training triangles corresponding to the current training action in the training video frame, the triangles including three vertices corresponding to three key points, the three key points including three consecutive joints or including a predetermined head position and two shoulder joints.

[0045] This step completes the normalization of the triangle surrounded by the consecutive joints by taking the bicondylar center as the origin and the bicondylar distance as the unit length, so that two people with different body proportions have highly similar or even identical triangles surrounded by their corresponding joints if they perform the same action, and different triangles surrounded by their corresponding joints if they perform different actions.

[0046] Specifically, a plurality of first initial triangles corresponding to the demonstration action shown in the current reference video frame and a plurality of second initial triangles corresponding to the current training action in the training video frame are determined; the plurality of first initial triangles are normalized to obtain a plurality of reference triangles by taking the bicondylar center of the demonstrator in the current reference video frame as the origin and the bicondylar distance as the unit length; and the plurality of second initial triangles are normalized to obtain a plurality of training triangles by taking the bicondylar center of the trainer in the training video frame as the origin and the bicondylar distance as the unit length.

[0047] In this embodiment, 11 human body key points are defined, which can be 0, nose; 1, right shoulder; 2, left shoulder; 3, right elbow; 4, left elbow; 5, right hip; 6, left hip; 7, right knee; 8, left knee; 9, left ankle; and 10, right ankle.

[0048] The three vertices of the reference triangle and the training triangle correspond to three consecutive joints or a predetermined head position and two shoulder joints.

[0049] The consecutive joints refer to adjacent joints. For example, the shoulder joint, the elbow joint, and the wrist joint are three consecutive joints. The shoulder joint and the elbow joint are two consecutive joints, but the shoulder joint and the wrist joint are not two consecutive joints.

[0050] Regarding the determination of the initial triangle. In one embodiment, the action identifier of the demonstration action shown in the current reference video frame is determined; a triangle identifier combination is determined, the triangle identifier combination corresponding to the scoliosis identifier and the action identifier; and according to the triangle identifier combination, a plurality of first initial triangles corresponding to the demonstration action shown in the current reference video frame and a plurality of second initial triangles corresponding to the current training action in the training video frame are determined.

[0051] S140, determining target similarities between corresponding triangles in the plurality of reference triangles and the plurality of training triangles, the target similarities including orientation similarities and shape similarities.

[0052] For each reference triangle in the plurality of reference triangles, a training triangle corresponding to the current reference triangle in the plurality of training triangles is determined, and then a target similarity between the current reference triangle and the corresponding training triangle is determined.

[0053] For example, if the current reference triangle is a triangle formed by a right shoulder joint, a right elbow joint and a right wrist joint, the corresponding training triangle is also a triangle formed by the right shoulder joint, the right elbow joint and the right wrist joint, and then a target similarity between the two predetermined triangles is determined.

[0054] The target similarities include orientation similarities and shape similarities, and the orientation similarities and the shape similarities are used jointly to accurately reflect the accuracy of the training action presented by the training video frame.

[0055] The plurality of reference triangles can be selected as: 1, nose-left shoulder-right shoulder; 2, left shoulder-right shoulder-right elbow; 3, right shoulder-left shoulder-left elbow; 4, right shoulder-right elbow-right wrist; 5, left shoulder-left elbow-left wrist; 6, right shoulder-right hip-right knee; 7, left shoulder-left hip-left knee; 8, right hip-right knee-right ankle; 9, left hip-left knee-left ankle; 10, right knee-right ankle-right toe; and 11, left knee-left ankle-left toe. Only 11 groups of target similarities of triangles need to be calculated for each video frame, and the complexity is extremely low, and real-time calculation can be achieved on a mobile terminal.

[0056] S150, determining and displaying training evaluation information corresponding to the current training action according to all the target similarities.

[0057] The triangles having a corresponding relationship in the plurality of reference triangles and the plurality of training triangles are taken as a triangle combination. It can be understood that if the current training action of the trainer is relatively accurate, the target similarities corresponding to all the triangle combinations are relatively high, and if the target similarity corresponding to any triangle combination is relatively low, it means that the current training action of the trainer is deficient. Therefore, the training evaluation information includes overall training evaluation information and / or local training evaluation information.

[0058] The overall training evaluation information is used to evaluate the overall completion degree of the current training action, such as an overall score of the current training action. The local training evaluation information is used to evaluate the deficiency of the local action of the target object, such as an insufficient bending angle for a bending action, or an insufficient or excessive lifting height for a leg lifting action. The local training evaluation information is determined based on a comparison result of the target similarity corresponding to the corresponding triangle combination and a corresponding predetermined threshold.

[0059] The driving of the scoliosis identifier, the quantification of the scoliosis index, and the real-time correction of the terminal realize a low-cost, low-radiation, privacy-friendly home high-frequency training and standardized evaluation closed loop without relying on special hardware. The embodiment only uses a mobile phone camera to perform end-side reasoning (at a level of 24-30 FPS), provides instant voice / AR prompts, and does not require motion capture equipment, pressure plates, or multiple IMUs (inertial measurement units), thereby achieving high accessibility at home.

[0060] It should be noted that the method described in the embodiment is executed by a terminal electronic device, which can be configured to not upload the local cache or only upload user-selected information. The terminal can be a mobile terminal, which is more friendly in terms of endurance and operation.

[0061] The technical scheme provided by the embodiment of the present application displays a reference video corresponding to the scoliosis identifier of the target object, so that the user can follow the correct reference video to perform scoliosis training. During the display of the reference video, a training video frame corresponding to the current reference video frame is obtained, and a plurality of reference triangles corresponding to the demonstration action displayed by the current reference video frame and a plurality of training triangles corresponding to the current training action in the training video frame are determined with the center of the double hip line as the origin and the distance between the double hips as the unit length, so as to eliminate the difference in triangle similarity caused by the body proportion of the human body. The training evaluation information corresponding to the current training action is determined and displayed through the target similarity between the corresponding triangles in the plurality of reference triangles and the plurality of training triangles, so as to achieve the technical effect of real-time feedback of the accuracy of the training action, so that the user can know the completion degree of the action in time. In the case that the completion degree of the training action is low, the training action performed by the user is adjusted in time according to the training evaluation information. In this way, the user only relies on the mobile phone camera to collect the video stream during training, and the skeleton real-time action quality evaluation and correction are completed on the end side, thereby realizing a low-cost, privacy-friendly, standardized, interpretable, and highly compliant home rehabilitation closed loop.

[0062] On the basis of the foregoing embodiment, the scoliosis identifier is determined based on the scoliosis index. In response to a review trigger operation, a first scoliosis index of a target object, a shooting time of a first spine X-ray image based on the first scoliosis index, a second scoliosis index, and a shooting time of a second spine X-ray based on the second scoliosis index are obtained. The progression speed of the scoliosis of the target object is determined according to the first scoliosis index, the shooting time of the first spine X-ray image based on the first scoliosis index, the second scoliosis index, and the shooting time of the second spine X-ray based on the second scoliosis index. If the progression speed of the scoliosis is greater than a predetermined speed threshold, a prompt information is output.

[0063] The scoliosis index includes one or more sub-scoliosis indexes.

[0064] The user inputs or selects a first scoliosis index, a first scoliosis X-ray image shooting time on which the first scoliosis index is based, a second scoliosis index, and a second scoliosis X-ray image shooting time on which the second scoliosis index is based in the interactive interface, and then clicks or touches a corresponding review trigger option; the processor acquires the first scoliosis index and the second scoliosis index in response to the review trigger operation.

[0065] Since the first scoliosis index and the second scoliosis index are determined based on different scoliosis X-ray images, the first scoliosis index and the second scoliosis index correspond to different scoliosis X-ray image shooting times, that is, the first scoliosis index and the second scoliosis index can reflect the scoliosis state of the target object at different times. Therefore, the progress speed of the scoliosis of the target object can be determined according to the first scoliosis index and the second scoliosis index. If the progress speed is a positive value, it indicates that the scoliosis speed of the target object is accelerated, if the progress speed is a negative value, it indicates that the scoliosis degree of the target object is getting lighter and lighter, which means that the scoliosis training of the target object has a positive effect; if the progress speed is zero, it indicates that the scoliosis of the target object has no progress. In this embodiment, when the progress speed of the scoliosis is greater than a predetermined speed threshold, a prompt information is output to enable the user to timely understand the scoliosis condition of himself.

[0066] On the basis of the foregoing embodiments, this embodiment precreates a training library, and after the scoliosis index is determined, a training video corresponding to the scoliosis index is matched from the training library. Specifically, the scoliosis is prestructured into types such as thoracic segment / thoracolumbar segment / lumbar segment, C / S type, left / right, and accompanied by pelvic tilt; and for each type, a “correction element combination”, a motion template, a motion key, and an error compensation content are defined. The correction element combination includes axial stretching, de-rotation, lateral shift, pelvic horizontalization, and directional breathing. In the field of scoliosis correction, directional breathing is a rehabilitation training technology based on “respiratory movement and spinal mechanics coupling”, the core of which is to guide the patient to control the contraction of the respiratory muscle in a specific direction, to utilize the biomechanical force of chest expansion / contraction to assist in correcting the scoliosis angle and improving the trunk balance, and the directional breathing is often used in combination with brace treatment and physical therapy.

[0067] Figure 3 Another flowchart of an image processing method for scoliosis training provided by an embodiment of the present application is shown. The technical solution of this embodiment can be combined with other embodiments, and for the same or related parts, the description of other embodiments can be combined and will not be repeated here. As shown in the figure, the method of this embodiment can specifically include: Figure 3

[0068] ​S210, in response to the first trigger operation, determining a scoliosis identifier of the target object, and displaying a reference video corresponding to the scoliosis identifier of the target object, the scoliosis identifier being determined based on scoliosis typing and orientation information, the scoliosis typing and orientation information being determined based on the first spine X-ray image of the target object.

[0069] S220, during the display of the reference video, obtaining a training video frame corresponding to a current reference video frame.

[0070] S230, determining a plurality of reference triangles corresponding to the demonstration action shown in the current reference video frame and a plurality of training triangles corresponding to the current training action in the training video frame, with the center of the double hip line as the origin and the distance between the two hips as the unit length, the triangle including three key points corresponding to the three key points, the three key points including three consecutive joints, or including a predetermined position of the head and two shoulder joints.

[0071] S2401, determining the cosine similarity between the reference triangles and the training triangles in the plurality of reference triangles and the plurality of training triangles that have a corresponding relationship, the angle difference between the target angle of the reference triangle and the target angle of the training triangle that have a corresponding relationship, and the linear attenuation factor corresponding to the angle difference, wherein in the human physiological structure, the target angle corresponds to the head predetermined position or the middle key point of the three key points corresponding to the corresponding triangle.

[0072] The reference triangles and the training triangles in the plurality of reference triangles and the plurality of training triangles that have a corresponding relationship are combined as a triangle group. For each triangle group, the cosine similarity between the reference triangle and the training triangle is determined, the target angle of the reference triangle and the target angle of the training triangle, the angle difference between the two target angles, and the linear attenuation factor corresponding to the angle difference.

[0073] The cosine similarity is determined based on a six-dimensional cosine similarity calculation method, specifically:

[0074] ;

[0075] Wherein, is the training triangle, is the reference triangle. The six-dimensional cosine similarity can measure the orientation similarity and shape similarity between the training triangle and the reference triangle.

[0076] The linear attenuation factor is an attenuation coefficient controlled by the angle difference. The linear attenuation coefficient in the embodiment can be selected as:

[0077] ;

[0078] Wherein, d is the angle difference. The greater the angle difference, the greater the deviation of the training triangle compared to the corresponding reference triangle, that is, the greater the cosine similarity between the two is folded.

[0079] To determine the target angle, the intermediate key point corresponding to the target angle needs to be determined. In this embodiment, the intermediate key point is determined based on the relative positions of the joints in the human physiological structure. Since the three key points corresponding to the three vertices of the triangle are three consecutive joints, or the predetermined position of the head and two shoulder joints, therefore in the human body, the intermediate joint (intermediate key point) in the three consecutive joints is unique; in the predetermined position of the head and two consecutive joints, the predetermined position of the head is taken as the intermediate key point.

[0080] For example, the three vertices of the triangle correspond to the left shoulder joint, the left elbow joint and the left wrist joint respectively, and then the intermediate key point is the left elbow joint.

[0081] For example, the three vertices of the triangle are the nose tip, the left shoulder joint and the right shoulder joint, and then the nose tip is the intermediate key point.

[0082] S2402, the product of the cosine similarity between the reference triangle and the training triangle with the linear decay factor is taken as the target similarity between the reference triangle and the training triangle.

[0083] The product of the cosine similarity between the reference triangle and the training triangle with the linear decay factor is taken as the target similarity between the reference triangle and the training triangle, which can measure the orientation similarity, shape similarity between the reference triangle and the training triangle.

[0084] S2403, for each target similarity, the target similarity is normalized based on the predetermined weight of the reference triangle to update the target similarity, wherein the weight of the reference triangle whose target angle is closer to the spine is greater.

[0085] For different target similarities, the closer the target angle corresponding to the target similarity is to the spine, the greater the weight is, and vice versa. For example, the weight of the target similarity corresponding to the reference triangle whose target angle is the shoulder is greater than the weight of the target similarity corresponding to the reference triangle whose target angle is the wrist.

[0086] Therefore, the updated target similarity can more accurately reflect the completion degree of the corresponding training triangle corresponding to the action element.

[0087] S250, according to all target similarities, determine and display the training evaluation information corresponding to the current training action.

[0088] Each triangle corresponds to an element of the scoliosis training movement. Therefore, the similarity of each target corresponds to the completion of an element. When the similarity of any updated target is lower than the corresponding threshold, the prompt information corresponding to that element is output so that the user can understand the specific reason for the low completion of the movement in a timely manner.

[0089] The technical solution provided in this invention combines cosine similarity with a linear decay factor for the angle difference between the target and the target to determine the target similarity between multiple reference triangles and multiple training triangles. Then, it normalizes the similarity of each target based on the distance between the target angle and the spine, thereby improving the accuracy of target similarity determination.

[0090] Based on the aforementioned embodiments, after the updated similarity of each target is determined, the sum of all target similarities is taken as the total target similarity; according to the pre-created correspondence between the total similarity range and the completion level, the completion level corresponding to the total target similarity is determined; while displaying the training evaluation information, the completion level kernel and / or the evaluation data corresponding to the completion level are displayed.

[0091] Specifically, calculate the sum of all updated target similarities:

[0092] ;

[0093] in, To be identified as The weights of the reference triangle, To be identified as The reference triangle and the label are The product of the cosine similarity between the training triangles and the linear decay factor; N is the number of reference triangles.

[0094] The sum of target similarities reflects the overall similarity between all training triangles and all reference triangles, that is, the degree of similarity between the training actions in the training video frames and the teacher's demonstration actions in the reference video frames. Therefore, a total similarity range is determined where the sum of target similarities falls. The completion level corresponding to this total similarity range is used as the completion level corresponding to the total target similarity. While displaying training evaluation information, this completion level and / or the corresponding evaluation data are also displayed, such as 98 points for completion level A, 90 points for completion level B, 75 points for completion level C, and 60 points for completion level D, etc. This allows users to understand their completion level of training actions in a timely manner and adjust their training actions promptly when the completion level is low.

[0095] Figure 4 This is a schematic diagram of the image processing device for scoliosis training provided in an embodiment of the present invention. This image processing device for scoliosis training can be used in electronic devices.Figure 4 The image processing device for scoliosis training shown in the figure comprises:

[0096] The response module 110 is configured to, in response to the first trigger operation, determine a scoliosis identifier of the target object, and display a reference video corresponding to the scoliosis identifier of the target object, wherein the scoliosis identifier is determined based on scoliosis typing and orientation information, and the scoliosis typing and the orientation information are determined based on a first spine X-ray image of the target object.

[0097] The training video frame module 120 is configured to, during the display of the reference video, acquire a training video frame corresponding to a current reference video frame.

[0098] The triangle module 130 is configured to, with the center of the double hip line as the origin and the double hip distance as the unit length, determine a plurality of reference triangles corresponding to a demonstration action shown in the current reference video frame and a plurality of training triangles corresponding to a current training action in the training video frame, wherein each triangle comprises three vertices corresponding to three key points, and the three key points comprise three consecutive joints or a predetermined position of the head and two shoulder joints.

[0099] The similarity module 140 is configured to determine a target similarity between corresponding triangles in the plurality of reference triangles and the plurality of training triangles, wherein the target similarity comprises an orientation similarity and a shape similarity.

[0100] The feedback module 150 is configured to determine and display training evaluation information corresponding to the current training action according to all the target similarities.

[0101] The technical scheme of the embodiment of the present application displays a reference video corresponding to the scoliosis identifier of the target object, so that the user can follow the correct reference video to perform scoliosis training. During the display of the reference video, a training video frame corresponding to a current reference video frame is acquired, and with the center of the double hip line as the origin and the double hip distance as the unit length, a plurality of reference triangles corresponding to a demonstration action shown in the current reference video frame and a plurality of training triangles corresponding to a current training action in the training video frame are determined, so as to eliminate the difference in triangle similarity caused by the body proportion. Through the target similarity between corresponding triangles in the plurality of reference triangles and the plurality of training triangles, training evaluation information corresponding to the current training action is determined and displayed, so as to achieve the technical effect of real-time feedback of the accuracy of the training action, so that the user can know the completion degree of the action in time. In the case that the completion degree of the training action is low, the training evaluation information is used to adjust the training action in time, so that the user only needs to rely on the mobile phone camera to collect the video stream during training, and the real-time action quality evaluation and correction of the skeleton are completed at the terminal side, so as to realize low-cost, privacy-friendly, standardized, interpretable and high-compliance home rehabilitation closed loop.

[0102] In one embodiment, the similarity module 140 is configured to:

[0103] determine, for each of the reference triangles and the training triangles, a cosine similarity between the reference triangle and the training triangle, an angle difference between a target angle of the reference triangle and a target angle of the training triangle, and a linear attenuation factor corresponding to the angle difference, wherein the target angle corresponds to a vertex of the reference triangle or the training triangle that is closest to a spine of the human physiological structure;

[0104] multiply the cosine similarity and the linear attenuation factor to obtain a target similarity between the reference triangle and the training triangle;

[0105] normalize each of the target similarities based on a predetermined weight of the reference triangle to update the target similarity, wherein the weight of the reference triangle is greater if the target angle is closer to the spine.

[0106] In one embodiment, the feedback module 150 is configured to:

[0107] sum all of the updated target similarities to obtain a target total similarity;

[0108] determine a completion level corresponding to the target total similarity based on a pre-created correspondence between a total similarity range and a completion level;

[0109] display the completion level and / or evaluation data corresponding to the completion level, and training evaluation information corresponding to a target similarity that is lower than a predetermined similarity threshold.

[0110] In one embodiment, the triangle module 130 includes:

[0111] an initial triangle unit configured to determine a plurality of first initial triangles corresponding to the demonstration action shown in the current reference video frame, and a plurality of second initial triangles corresponding to the current training action in the training video frame;

[0112] a first normalization unit configured to normalize the plurality of first initial triangles with a center of a double hip line of the demonstrator in the current reference video frame as an origin and a distance between the double hips as a unit length to obtain a plurality of reference triangles;

[0113] A second normalization unit is configured to normalize the plurality of second initial triangles with respect to a center of a double hip line of the trainer in the training video frame as an origin, and with respect to a double hip distance as a unit length, to obtain a plurality of training triangles.

[0114] In one embodiment, the initial triangle unit is configured to:

[0115] determine an action identifier of a demonstration action shown in the current reference video frame;

[0116] determine a triangle identifier combination corresponding to the scoliosis identifier and the action identifier;

[0117] determine, according to the triangle identifier combination, a plurality of first initial triangles corresponding to the demonstration action shown in the current reference video frame, and a plurality of second initial triangles corresponding to a current training action in the training video frame.

[0118] In one embodiment, the device further comprises a review module configured to:

[0119] in response to a review trigger operation, obtain a first scoliosis indicator of a target object, a shooting time of a first spine X-ray image on which the first scoliosis indicator is based, a second scoliosis indicator, and a shooting time of a second spine X-ray on which the second scoliosis indicator is based;

[0120] determine a scoliosis progression speed of the target object according to the first scoliosis indicator, the shooting time of the first spine X-ray image on which the first scoliosis indicator is based, the second scoliosis indicator, and the shooting time of the second spine X-ray on which the second scoliosis indicator is based;

[0121] if the scoliosis progression speed is greater than a predetermined speed threshold, output a prompt information.

[0122] In one embodiment, the response module determines a scoliosis identifier through an identifier unit configured to:

[0123] in response to a first trigger operation for the first spine X-ray image, determine the scoliosis type corresponding to the first spine X-ray image and the orientation information;

[0124] determine the scoliosis identifier according to the scoliosis type and the orientation information.

[0125] The image processing device for scoliosis training provided in the embodiments of the present application can perform the image processing method for scoliosis training provided in any of the embodiments of the present application, and has the corresponding function modules and beneficial effects of the execution method.

[0126] It is worth noting that the above-mentioned image processing device for scoliosis training includes various units and modules, which are only divided according to functional logic, and are not limited to the above division, as long as the corresponding functions can be realized; in addition, the specific name of each functional unit is only for easy mutual differentiation, and does not limit the protection scope of the embodiment of the present application.

[0127] Figure 5 is a structural schematic diagram of an electronic device provided by an embodiment of the present application. The electronic device 10 is intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular telephones, smart phones, wearable devices (such as helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions, are meant to be examples only, and are not intended to limit implementations of the applications described and / or claimed in this document.

[0128] As shown in Figure 5 , the electronic device 10 includes at least one processor 11, and a memory, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., which is communicatively connected to the at least one processor 11, wherein the memory stores a computer program that can be executed by the at least one processor. The processor 11 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or the computer program loaded from the storage unit 18 into the random access memory (RAM) 13. In the random access memory (RAM) 13, various programs and data required for the operation of the electronic device 10 can also be stored. The processor 11, the read-only memory (ROM) 12, and the random access memory (RAM) 13 are connected to each other through a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0129] A plurality of components in the electronic device 10 are connected to the I / O interface 15, including: an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a magnetic disk, an optical disk, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices through a computer network, such as the Internet, and / or various telecommunications networks.

[0130] The processor 11 can be various general and / or special purpose processing components with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various specialized artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, and the like. The processor 11 performs various methods and processes described above, such as the image processing method for scoliosis training.

[0131] In some embodiments, the image processing method for scoliosis training can be implemented as a computer program tangibly embodied in a computer readable storage medium, such as the storage unit 18. In some embodiments, part or all of the computer program can be loaded and / or installed onto the electronic device 10 via the read-only memory (ROM) 12 and / or the communication unit 19. When the computer program is loaded onto the random access memory (RAM) 13 and executed by the processor 11, one or more steps of the image processing method for scoliosis training described above can be performed. Alternatively, in other embodiments, the processor 11 can be configured to perform the image processing method for scoliosis training by any other suitable means, such as by means of firmware.

[0132] Various implementations of the systems and techniques described above can be realized in digital electronic circuitry, integrated circuitry, a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), a system on a chip (SOC), a programmable logic device (CPLD), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include implementation in one or more computer programs that are executable and / or interpretable on a programmable system including at least one programmable processor, which can be special or general purpose, coupled to receive data and instructions from, and to transmit data and instructions to, a storage system, at least one input device, and at least one output device.

[0133] Computer programs for implementing the image processing method for scoliosis training of the present application can be written in any combination of one or more programming languages. These computer programs can be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the computer program, when executed by the processor, implements the functions / operations specified in the flow diagrams and / or block diagrams. The computer program can be executed entirely on a machine, partially on a machine, partially on a machine as a standalone software package and partially on a remote machine, or entirely on a remote machine or server.

[0134] The embodiment of the present application provides a computer readable storage medium, which stores computer instructions, and the computer instructions are used for causing a processor to execute an image processing method for scoliosis training, comprising the following steps.

[0135] In response to a first trigger operation, a scoliosis identification of a target object is determined, and a reference video corresponding to the scoliosis identification of the target object is displayed, wherein the scoliosis identification is determined based on scoliosis typing and orientation information, and the scoliosis typing and the orientation information are determined based on a first spine X-ray image of the target object;

[0136] During the display of the reference video, a training video frame corresponding to a current reference video frame is obtained;

[0137] A plurality of reference triangles corresponding to a demonstration action displayed in the current reference video frame and a plurality of training triangles corresponding to a current training action in the training video frame are determined, taking the center of the double hip line as the origin and taking the distance between the two hips as the unit length, wherein the triangles comprise three key points corresponding to three vertices, and the three key points comprise three consecutive joints or a predetermined position of a head and two shoulder joints.

[0138] A target similarity between corresponding triangles in the plurality of reference triangles and the plurality of training triangles is determined, wherein the target similarity comprises an orientation similarity and a shape similarity.

[0139] According to all the target similarities, training evaluation information corresponding to the current training action is determined and displayed.

[0140] In the context of the present application, a computer-readable storage medium can be a tangible medium that can contain or store a computer program for use by or in connection with an instruction execution system, apparatus, or device. A computer-readable storage medium can include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium can be a machine-readable signal medium. More specific examples of a machine-readable storage medium will include one or more lines of a program of instructions in a transitory signal, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0141] To provide for interaction with a user, the systems and techniques described here can be implemented on an electronic device having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user and a keyboard and a pointing device (e.g., a mouse or a trackball) by which the user can provide input to the electronic device. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form, including acoustic, speech, or tactile input.

[0142] The systems and techniques described here can be implemented in a computing system that includes a back end component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front end component (e.g., a user computer having a graphical user interface or a Web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such back end, middleware, or front end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), blockchain network, and the Internet.

[0143] The computing system can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. Servers can be cloud servers, also known as cloud computing servers or cloud hosts, which are a host product in the cloud computing service system to solve the defects of large management difficulty and weak business scalability in traditional physical hosts and VPS services.

[0144] In particular, the processes described above with reference to the flowcharts can be implemented as a computer software program according to embodiments of the present application. For example, embodiments of the present application include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for executing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network by the communication unit 19, or installed from the storage unit 18, or installed from the ROM 12. When the computer program is executed by the processor 11, the above-mentioned functions defined in the methods of the embodiments of the present application are performed.

[0145] Embodiments of the present application also provide a computer program product comprising a computer program which, when executed by a processor, implements the image processing method for scoliosis training according to any embodiment of the present application.

[0146] The computer program code implementing the present application can be written in one or more programming languages or combinations of languages including object oriented languages such as Java, Smalltalk, C++ or conventional procedural programming languages such as the "C" programming language or similar programming languages. The program code can execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computer (for example, through the Internet using an Internet Service Provider).

[0147] It should be understood that various forms of flow shown above can be re-ordered, added to, or deleted from without departing from the spirit of the present application. For example, the steps recited in the present application can be performed in parallel, in series, or in a different order, and the present application is not limited in this regard.

[0148] The above detailed description does not limit the scope of the application. Various modifications, combinations, sub-combinations and alternatives can be made to the detailed description. Any modification, equivalent replacement and improvement etc. made within the spirit and principle of the application shall be included in the scope of the application.

Claims

1. An image processing method for scoliosis training, characterized in that, The method includes: In response to the first triggering operation, the scoliosis identifier of the target object is determined, and a reference video corresponding to the scoliosis identifier of the target object is displayed. The scoliosis identifier is determined based on the scoliosis classification and orientation information. The scoliosis classification and orientation information are determined based on the first spinal X-ray image of the target object. During the display of the reference video, a training video frame corresponding to the current reference video frame is acquired; Using the center of the line connecting the two hips as the origin and the distance between the two hips as the unit length, determine multiple reference triangles corresponding to the demonstration action shown in the current reference video frame, and multiple training triangles corresponding to the current training action in the training video frame. Each triangle includes vertices that correspond one-to-one with three key points. The three key points include three consecutive joints, or include a predetermined head position and two shoulder joints. Determine the target similarity between the plurality of reference triangles and the corresponding triangles in the plurality of training triangles, wherein the target similarity includes orientation similarity and shape similarity; Based on the similarity of all the targets, determine and display the training evaluation information corresponding to the current training action.

2. The method according to claim 1, characterized in that, Determining the target similarity between the plurality of reference triangles and corresponding triangles in the plurality of training triangles includes: Determine the cosine similarity between the reference triangles and the training triangles that have a corresponding relationship among the plurality of reference triangles and the plurality of training triangles, as well as the angle difference between the target angle of the reference triangle and the target angle of the training triangle that have a corresponding relationship, and the linear decay factor corresponding to the angle difference. In the human physiological structure, the vertex corresponding to the target angle is the middle key point of the three key points corresponding to the head at a predetermined position or the corresponding triangle. The product of the cosine similarity between the corresponding reference triangle and the training triangle and the linear decay factor is taken as the target similarity between the corresponding reference triangle and the training triangle. For each target similarity, the target similarity is normalized based on the predetermined weight of the reference triangle to update the target similarity, wherein the reference triangle whose included angle is closer to the spine has a larger weight.

3. The method according to claim 2, characterized in that, The step of determining and displaying training evaluation information corresponding to the current training action based on all the target similarities includes: The sum of all updated target similarities is taken as the total target similarity; The completion level corresponding to the target total similarity is determined based on the pre-created correspondence between the total similarity range and the completion level. Display the completion level and / or the evaluation data corresponding to the completion level, as well as the training evaluation information corresponding to the target similarity below a predetermined similarity threshold.

4. The method according to claim 1, characterized in that, The process of determining multiple reference triangles corresponding to the demonstration movement displayed in the current reference video frame, and multiple training triangles corresponding to the current training movement in the training video frame, with the center of the line connecting the two hips as the origin and the distance between the two hips as the unit length, includes: Determine multiple first initial triangles corresponding to the action demonstrated in the current reference video frame, and multiple second initial triangles corresponding to the current training action in the training video frame; Using the center of the line connecting the hips of the demonstrator in the current reference video frame as the origin and the distance between the hips as the unit length, the plurality of first initial triangles are normalized to obtain a plurality of reference triangles. Using the center of the line connecting the trainee's two hips in the training video frame as the origin and the distance between the two hips as the unit length, the multiple second initial triangles are normalized to obtain multiple training triangles.

5. The method according to claim 4, characterized in that, The step of determining the plurality of first initial triangles corresponding to the demonstrated action in the current reference video frame, and the plurality of second initial triangles corresponding to the current training action in the training video frame, includes: Determine the motion identifier of the demonstrated action displayed in the current reference video frame; Determine the triangular symbol combination, which corresponds to the scoliosis symbol and the action symbol; Based on the triangle identifier combination, multiple first initial triangles corresponding to the demonstration action displayed in the current reference video frame and multiple second initial triangles corresponding to the current training action in the training video frame are determined.

6. The method according to claim 1, characterized in that, The method also includes: In response to the review trigger operation, the first scoliosis index of the target object, the acquisition time of the first spinal X-ray image on which the first scoliosis index is based, the second scoliosis index, and the acquisition time of the second spinal X-ray image on which the second scoliosis index is based are obtained. The rate of scoliosis progression of the target subject is determined based on the first scoliosis index, the time of the first spinal X-ray image on which the first scoliosis index is based, the second scoliosis index, and the time of the second spinal X-ray image on which the second scoliosis index is based. If the rate of scoliosis progression exceeds a predetermined speed threshold, a prompt message will be output.

7. The method according to claim 1, characterized in that, The scoliosis marker is determined by the following steps: In response to a first triggering operation on the first spinal X-ray image, the scoliosis classification and orientation information corresponding to the first spinal X-ray image are determined; The scoliosis identifier is determined based on the scoliosis classification and the orientation information.

8. An electronic device, characterized in that, The electronic device includes: One or more processors; Storage device for storing one or more programs. When the one or more programs are executed by the one or more processors, the one or more processors implement the image processing method for scoliosis training as described in any one of claims 1-7.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that cause a processor to execute the image processing method for scoliosis training as described in any one of claims 1-7.

10. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, implements the image processing method for scoliosis training according to any one of claims 1-7.

Images