Construction and application method of disease prediction model, model, equipment and medium

Abstract

The invention relates to the field of medicine, and discloses a disease prediction model construction method, a disease prediction model application method, a model, equipment and a medium. According to the disease prediction model constructed by the construction scheme, a first coordinate feature of each joint point of a target patient in each frame of gait image of a gait video is obtained through a feature extraction module; a second coordinate feature of each joint point of each patient in each frame of gait image of the gait video is obtained through another feature extraction module; and processing the joint point features of the two feature extraction modules through a target feature generation module to obtain target coordinate features of each joint point of the target patient in each frame of gait image of the gait video, and performing disease prediction on the target patient based on the target coordinate features of each joint point of the target patient. According to the disease prediction model, more accurate joint point coordinate information of the target patient can be obtained, and the performance of the disease prediction model is improved, so that the accuracy of a disease prediction result is improved.

Description

Technical Field

[0001] The present application relates to the medical field, and in particular to a method, model, device, and medium for constructing and applying a disease prediction model. Background Art

[0002] With the development of medical imaging technology and biometric recognition technology, gait analysis, as a non-contact biometric recognition method, has been widely used in the field of clinical medicine. By analyzing the patient's gait, doctors can diagnose certain neurological diseases, musculoskeletal diseases, and other health problems that affect walking patterns.

[0003] At present, relevant researchers have trained many disease prediction models using gait videos, aiming to assist doctors in disease diagnosis by performing gait analysis through trained disease prediction models, that is, using disease prediction models to obtain disease prediction results based on patients' gait videos, and using the disease prediction results to assist doctors in making more accurate medical diagnoses. However, the existing disease prediction models have the following problems when predicting diseases: when the existing disease prediction models process the joint point information of patients in gait videos, they generally encounter the problem of joint point recognition errors, such as mistaking the joint point features of the left leg for the joint point features of the right leg, and extracting wrong joint point information leads to unstable features, which leads to poor performance of the trained disease prediction model, affecting the accuracy and reliability of disease prediction; in addition, in a gait video containing multiple patients, the existing disease prediction models often find it difficult to accurately distinguish and extract the joint point coordinate information of each person, and cannot clearly identify which joint point belongs to a specific patient, which further increases the complexity and uncertainty of the model's prediction of diseases. Therefore, these problems not only limit the effect of the model in practical applications, but also hinder the widespread application of gait analysis technology in the medical field.

[0004] In summary, the existing disease prediction model has poor performance (i.e., when extracting the joint point information of the patient in the gait video, it is easy to obtain wrong joint point information), resulting in inaccurate disease prediction results. Therefore, there is an urgent need for a disease prediction model that can accurately extract the joint point information of the target patient from the gait video to improve the accuracy of the disease prediction results. Summary of the invention

[0005] The present application provides a method, model, device, and medium for constructing and applying a disease prediction model to solve the problem that the existing disease prediction model has poor performance, resulting in inaccurate disease prediction results.

[0006] The first aspect of the present application provides a method for constructing a disease prediction model, wherein the disease prediction model is used to predict a disease based on a patient's gait video, the method comprising:

[0007] Acquire a gait video set, wherein the gait video set includes multiple gait videos, each gait video includes multiple frames of gait images, each frame of gait image includes one or more patients, and the body structure of each patient includes multiple joint points;

[0008] Construct an initial disease prediction model, wherein the initial disease prediction model includes:

[0009] A first feature extraction module is used to take each gait video as a target gait video and a patient in the target gait video as a target patient, and perform a first feature extraction operation on each joint point of the target patient in each frame of gait image of the target gait video to obtain a first coordinate feature of each joint point of the target patient in each frame of gait image of the target gait video;

[0010] A second feature extraction module is used to perform a second feature extraction operation on each joint point of each patient in each frame of the gait image of the target gait video to obtain a second coordinate feature of each joint point of each patient in each frame of the gait image of the target gait video;

[0011] a target feature generation module, for obtaining a target coordinate feature of each joint point of the target patient in each gait image frame of the target gait video based on a first coordinate feature of each joint point of the target patient in each gait image frame of the target gait video and a second coordinate feature of each joint point of each patient in each gait image frame of the target gait video;

[0012] A disease prediction module, used for obtaining a disease prediction result corresponding to a target patient in a target gait video based on a target coordinate feature of each joint point of a target patient in each frame of a gait image of a target gait video;

[0013] Wherein, the target feature generation module is respectively connected to the first feature extraction module, the second feature extraction module, and the disease prediction module in communication;

[0014] The initial disease prediction model was trained multiple times using the gait video set until convergence.

[0015] In some embodiments of the present application, the first feature extraction module is configured to:

[0016] A top-down human posture estimation method is used to perform a first feature extraction operation on each joint point of the target patient in each frame of the gait image of the target gait video, so as to obtain a first coordinate feature of each joint point of the target patient in each frame of the gait image of the target gait video.

[0017] In some embodiments of the present application, the second feature extraction module is configured as follows:

[0018] A bottom-up human posture estimation method is used to perform a second feature extraction operation on each joint point of each patient in each frame of a gait image of a target gait video, so as to obtain a second coordinate feature of each joint point of each patient in each frame of a gait image of a target gait video.

[0019] In some embodiments of the present application, when each frame of the gait image includes a patient, the target feature generation module is configured to obtain the target coordinate feature of each joint point of the target patient in each frame of the gait image of the target gait video in the following manner:

[0020] The first coordinate feature of each joint point of the target patient in each frame of the gait image of the target gait video is fused with the second coordinate feature of the corresponding joint point of the target patient in the frame of the gait image to obtain the target coordinate feature of each joint point of the target patient in each frame of the gait image of the target gait video.

[0021] In some embodiments of the present application, when each frame of gait image includes multiple patients, the target feature generation module is configured to obtain the target coordinate feature of each joint point of the target patient in each frame of gait image of the target gait video in the following manner:

[0022] The first coordinate features of each joint point of the target patient in each frame of the gait image of the target gait video are respectively calculated with the second coordinate features of the corresponding joint points of each patient in the frame of the gait image to obtain multiple mean square error values, and the second coordinate feature of each joint point of the patient corresponding to the smallest mean square error value among the multiple mean square error values ​​is used as the target coordinate feature of each joint point of the target patient in each frame of the gait image of the target gait video.

[0023] In some embodiments of the present application, multiple mean square error values ​​are obtained in the following manner:

[0024]

[0025] Where m is the number of patients in each frame of gait image, n is the number of joint points of the patient, and MSE j represents the jth mean square error, a i Represents the first coordinate feature of the i-th joint point of the target patient in each frame of the gait image of the target gait video, Represents the second coordinate feature of the i-th joint point of the j-th patient in each frame of the gait image of the target gait video.

[0026] A second aspect of the present application provides a disease prediction model, which is used to predict a disease based on a gait video of a patient, wherein the gait video includes multiple frames of gait images, each frame of the gait image includes one or more patients, and the body structure of each patient includes multiple joint points. The model includes:

[0027] a first feature extraction module, used for taking a patient in the gait video as a target patient, and performing a first feature extraction operation on each joint point of the target patient in each frame of the gait image of the gait video, so as to obtain a first coordinate feature of each joint point of the target patient in each frame of the gait image of the gait video;

[0028] A second feature extraction module is used to perform a second feature extraction operation on each joint point of each patient in each frame of the gait image of the gait video to obtain a second coordinate feature of each joint point of each patient in each frame of the gait image of the gait video;

[0029] a target feature generation module, for obtaining a target coordinate feature of each joint point of the target patient in each gait image frame of the gait video based on a first coordinate feature of each joint point of the target patient in each gait image frame of the gait video and a second coordinate feature of each joint point of each patient in each gait image frame of the gait video;

[0030] A disease prediction module is used to obtain a disease prediction result corresponding to a target patient in a gait video based on a target coordinate feature of each joint point of the target patient in each frame of a gait video;

[0031] Among them, the target feature generation module is communicated with the first feature extraction module, the second feature extraction module, and the disease prediction module respectively.

[0032] The third aspect of the present application provides an application method of a disease prediction model, the method comprising:

[0033] Obtain a video of the patient's gait;

[0034] The disease prediction model in the above embodiment is used to obtain a disease prediction result corresponding to the patient in the gait video based on the patient's gait video.

[0035] The fourth aspect of the present application provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps of the method described in any one of the first and third aspects of the above embodiments are implemented.

[0036] The fifth aspect of the present application provides a computer-readable storage medium having program instructions stored thereon, and when the program instructions are executed by a processor, the method described in any one of the first aspect and the third aspect of the above embodiments is implemented.

[0037] The present application has the following beneficial effects: the disease prediction model constructed by the construction scheme of the disease prediction model of the present application obtains the first coordinate feature of each joint point of the target patient in each frame of the gait image of the gait video through a feature extraction module, and then obtains the second coordinate feature of each joint point of each patient in each frame of the gait image of the gait video through another feature extraction module, and then processes the joint point features of the two feature extraction modules through a target feature generation module to obtain the target coordinate feature of each joint point of the target patient in each frame of the gait image of the gait video, and predicts the disease of the target patient based on the target coordinate feature of each joint point of the target patient. Therefore, the disease prediction model of the present application extracts the joint point coordinate information of the target patient through one module, and then extracts the joint point coordinate information of each patient in the gait video through another module, and processes the joint point information of the patient extracted by the two modules, which can obtain more accurate joint point coordinate information of the target patient, thereby improving the performance of the disease prediction model and thus improving the accuracy of the disease prediction results. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] The drawings herein are incorporated into the specification and constitute a part of the specification. These drawings illustrate embodiments consistent with the present application and are used together with the specification to illustrate the technical solution of the present application.

[0039] Figure 1 This is a flow chart of an embodiment of a method for constructing a disease prediction model provided by the present application;

[0040] Figure 2 This is a schematic diagram of the framework of an embodiment of a disease prediction model provided by the present application;

[0041] Figure 3 is a schematic diagram of an example of a patient's joint point structure provided in this application;

[0042] Figure 4 This is a flow chart of an embodiment of an application method of a disease prediction model provided by the present application;

[0043] Figure 5 It is a schematic diagram of a framework of an embodiment of an electronic device provided by the present application;

[0044] Figure 6 It is a schematic diagram of a framework of an embodiment of a computer-readable storage medium provided by the present application. DETAILED DESCRIPTION

[0045] The scheme of the embodiment of the present application is described in detail below in conjunction with the drawings of the specification.

[0046] In the following description, for the purpose of explanation rather than limitation, specific details such as specific system structures, interfaces, and technologies are provided to facilitate a thorough understanding of the present application.

[0047] The term "and / or" in this article is only a description of the association relationship of associated objects, indicating that there may be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the objects associated before and after are in an "or" relationship. In addition, "many" in this article means two or more than two. In addition, the term "at least one" in this article means any combination of at least two of any one or more of a plurality of, for example, including at least one of A, B, and C, can mean including any one or more elements selected from the set consisting of A, B, and C.

[0048] As described in the background technology, the existing disease prediction model has poor performance (i.e., when extracting the joint point information of the patient in the gait video, it is easy to obtain wrong joint point information), resulting in inaccurate disease prediction results. Therefore, there is an urgent need for a disease prediction model that can accurately extract the joint point information of the target patient from the gait video to improve the accuracy of the disease prediction results.

[0049] In order to solve the above problems, the present application proposes a new disease prediction model construction scheme. The disease prediction model constructed by the construction scheme of the present application obtains the first coordinate feature of each joint point of the target patient in each frame of the gait image of the gait video through a feature extraction module, and then obtains the second coordinate feature of each joint point of each patient in each frame of the gait image of the gait video through another feature extraction module, and then processes the joint point features of the two feature extraction modules through a target feature generation module to obtain the target coordinate feature of each joint point of the target patient in each frame of the gait image of the gait video, and predicts the disease of the target patient based on the target coordinate feature of each joint point of the target patient. Therefore, the disease prediction model of the present application extracts the joint point coordinate information of the target patient through one module, and then extracts the joint point coordinate information of each patient in the gait video through another module, and processes the joint point information of the patient extracted by the two modules. The method can obtain more accurate joint point coordinate information of the target patient, improve the performance of the disease prediction model, and thus improve the accuracy of the disease prediction results.

[0050] The present application is described in detail below with reference to the accompanying drawings and specific embodiments.

[0051] According to one embodiment of the present application, Figure 1 As shown, the present application provides a method for constructing a disease prediction model, the method comprising: S1, obtaining a gait video set, wherein the gait video set includes multiple gait videos, each gait video includes multiple frames of gait images, each frame of gait image includes one or more patients, and the body structure of each patient includes multiple joints; S2, constructing an initial disease prediction model, wherein, as Figure 2 As shown, the initial disease prediction model of the present application includes: a first feature extraction module, which is used to take each gait video as a target gait video, and take a patient in the target gait video as a target patient, and perform a first feature extraction operation on each joint point of the target patient in each frame of gait image of the target gait video to obtain a first coordinate feature of each joint point of the target patient in each frame of gait image of the target gait video; a second feature extraction module, which is used to perform a second feature extraction operation on each joint point of each patient in each frame of gait image of the target gait video to obtain a second coordinate feature of each joint point of each patient in each frame of gait image of the target gait video; a target feature generation module, which is used to generate a target feature based on the target The first coordinate feature of each joint point of the target patient in each frame of the gait image of the target gait video and the second coordinate feature of each joint point of the patient in each frame of the gait image of the target gait video are used to obtain the target coordinate feature of each joint point of the target patient in each frame of the gait video; the disease prediction module is used to obtain the disease prediction result corresponding to the target patient in the target gait video based on the target coordinate feature of each joint point of the target patient in each frame of the gait image of the target gait video; wherein the target feature generation module is communicated and connected with the first feature extraction module, the second feature extraction module, and the disease prediction module respectively; S3, using the gait video set to perform multiple iterative training on the initial disease prediction model until convergence.

[0052] According to one embodiment of the present application, in each iterative training process, the parameters of the disease prediction model are updated according to the following loss function:

[0053]

[0054] Among them, y is the true value of the disease corresponding to the patient in the CT image, The disease prediction result corresponding to the patient in the target gait video.

[0055] It can be seen from the above embodiments that the true result of whether the patient has the corresponding disease is obtained based on the patient's CT image, and the parameters of the disease prediction model are updated based on the loss value between the true disease result and the disease prediction result obtained by the model, thereby improving the performance of the disease prediction model.

[0056] In order to more intuitively understand the functions performed by the first feature extraction module, the second feature extraction module, and the target feature generation module in the disease prediction model constructed by the construction scheme of the present application, the first feature extraction module, the second feature extraction module, and the target feature generation module of the disease prediction model are described in detail below.

[0057] 1. The first feature extraction module

[0058] The first feature extraction module is used to take each gait video as a target gait video, and take a patient in the target gait video as a target patient, and perform a first feature extraction operation on each joint point of the target patient in each frame of gait image of the target gait video to obtain a first coordinate feature of each joint point of the target patient in each frame of gait image of the target gait video.

[0059] After research, it was found that the technical feature of the top-down human posture estimation method is to preferentially capture the head shape in the center of the screen as an anchor point, and infer the joint coordinate points based on the anchor point in subsequent video frames. Therefore, the top-down human posture estimation method has good character tracking capabilities. According to one embodiment of the present application, the first feature extraction module of the present application is configured to: use the top-down human posture estimation method to perform a first feature extraction operation on each joint point of the target patient in each frame of the gait image of the target gait video, so as to obtain the first coordinate feature of each joint point of the target patient in each frame of the gait image of the target gait video. Among them, according to one embodiment of the present application, the top-down human posture estimation method is the Blazepose method. It should be noted that the Blazepose method is only one of the top-down human posture estimation methods used in the present application to extract the joint features of the target patient. With the continuous development of the human posture estimation method, the present application can also use other human posture estimation methods to extract the joint point features of the target patient, which is not specifically limited here.

[0060] It can be seen from the above embodiments that accurate joint features of the target patient can be extracted using a top-down human posture estimation method.

[0061] 2. Second Feature Extraction Module

[0062] The second feature extraction module is used to perform a second feature extraction operation on each joint point of each patient in each frame of the gait image of the target gait video to obtain a second coordinate feature of each joint point of each patient in each frame of the gait image of the target gait video.

[0063] It has been found through research that the technical feature of the bottom-up human posture estimation method is to capture all identifiable human posture joints in space, and merge highly correlated structures into target person information through a connection method. Therefore, the bottom-up human posture estimation method can still maintain stable reasoning ability in multi-person environments and complex environments, and return accurate coordinate information. According to one embodiment of the present application, the second feature extraction module of the present application is configured to: use the bottom-up human posture estimation method to perform a second feature extraction operation on each joint point of each patient in each frame of the gait image of the target gait video, so as to obtain the second coordinate feature of each joint point of each patient in each frame of the gait image of the target gait video. Among them, according to one embodiment of the present application, the bottom-up human posture estimation method is the YoLoV8 method. It should be noted that the YoLoV8 method is only one of the bottom-up human posture estimation methods used in the present application to extract the joint features of all patients in the image. With the continuous development of the human posture estimation method, the present application can also use other human posture estimation methods to extract the joint features of all patients in the image, which is not specifically limited here.

[0064] It can be seen from the above embodiments that accurate joint features of all patients in the image can be extracted using the bottom-up human posture estimation method.

[0065] 3. Target Feature Generation Module

[0066] The target feature generation module is used to obtain the target coordinate feature of each joint point of the target patient in each frame of the gait image of the target gait video based on the first coordinate feature of each joint point of the target patient in each frame of the gait image of the target gait video and the second coordinate feature of each joint point of each patient in each frame of the gait image of the target gait video.

[0067] The technical routes of human posture estimation methods are divided into top-down and bottom-up. The former first detects the head of the target and then derives the position of the torso; the latter marks all torsos within the visible range and then connects the related torsos. This allows top-down techniques (for example, the Blazepose method) to perform better in target tracking tasks in complex environments, while bottom-up techniques (for example, the YoloV8 method) can obtain more stable and accurate coordinate points.

[0068] The following are two ways of obtaining the target joint point features of the target patient by using two different human body pose estimation techniques to obtain the joint point features of the patient.

[0069] (1) Fusion method

[0070] According to one embodiment of the present application, when each frame of gait image includes a patient, the target feature generation module of the present application is configured to: fuse the first coordinate feature of each joint point of the target patient in each frame of gait image of the target gait video (i.e., obtained using top-down human posture estimation technology) with the second coordinate feature of the corresponding joint point of the target patient in the frame of gait image (i.e., obtained using bottom-up human posture technology) to obtain the target coordinate feature of each joint point of the target patient in each frame of gait image of the target gait video.

[0071] It can be seen from the above embodiments that when the gait image includes only one patient, the joint point features of the patient obtained by using two human posture estimation technologies are fused to obtain more accurate joint point features of the patient.

[0072] (2) Error calculation method

[0073] According to another embodiment of the present application, when multiple patients are included in each frame of the gait image, the target feature generation module of the present application is configured to: perform mean square error calculation on the first coordinate feature of each joint point of the target patient in each frame of the gait image of the target gait video and the second coordinate feature of the corresponding joint point of each patient in the frame of the gait image, respectively, to obtain multiple mean square error values, and use the second coordinate feature of each joint point of the patient corresponding to the smallest mean square error value among the multiple mean square error values ​​as the target coordinate feature of each joint point of the target patient in each frame of the gait image of the target gait video. According to one embodiment of the present application, the calculation method of multiple mean square error values ​​is as follows:

[0074]

[0075] Where m is the number of patients in each frame of gait image, n is the number of joint points of the patient, and MSE j represents the jth mean square error, a i Represents the first coordinate feature of the i-th joint point of the target patient in each frame of the gait image of the target gait video, Represents the second coordinate feature of the i-th joint point of the j-th patient in each frame of the gait image of the target gait video.

[0076] For example, in the combination of two human pose estimation methods, this application chooses to use the Blazepose method as a top-down human pose estimation method, and sets it to capture only the coordinate information of one person. According to tests and literature records, the middle position of the image has the characteristic of being captured first; this application chooses to use the YOLOV8 method as a bottom-up human pose estimation method, and sets it to capture the coordinate information of one or more people. Figure 3As shown, it shows the joint structure of the patient. One embodiment of the present application selects 6 joints (numbered 11, 12, 23, 24, 25, 26) in the patient's torso as a basis for judgment, and calculates the mean square error of the 6 joints 11, 12, 23, 24, 25, 26 of the target patient in each frame of the gait image of the target gait video and the second coordinate features of the 6 joints 11, 12, 23, 24, 25, 26 of each patient in the frame of the gait image to obtain a plurality of mean square error values, and the second coordinate features of the 6 joints 11, 12, 23, 24, 25, 26 of the patient corresponding to the smallest mean square error value among the plurality of mean square error values ​​are used as the target coordinate features of the 6 joints 11, 12, 23, 24, 25, 26 of the target patient in each frame of the gait image of the target gait video. It should be noted that selecting the patient's 11, 12, 23, 24, 25, and 26 joint points for calculating the mean square error value is only an example used in this application to more intuitively illustrate the mean square error calculation process, and the specific joint points selected are not specifically limited here.

[0077] It can be seen from the above embodiments that the target feature generation module combines the advantages of two human posture estimation technologies and can meet the needs of target positioning and grasping high-precision joint point coordinates under complex backgrounds.

[0078] In addition, the method for constructing a disease prediction model of the above-mentioned embodiment of the present application is adopted. According to one embodiment of the present application, the present application provides a disease prediction model, which is used to predict diseases based on a patient's gait video, wherein the gait video includes multiple frames of gait images, each frame of the gait image includes one or more patients, and the body structure of each patient includes multiple joint points. The disease prediction model includes: a first feature extraction module, which is used to take a patient in the gait video as a target patient, and perform a first feature extraction operation on each joint point of the target patient in each frame of the gait image of the gait video to obtain a first coordinate feature of each joint point of the target patient in each frame of the gait image of the gait video; a second feature extraction module, which is used to extract a first feature from each joint point of the target patient in each frame of the gait image of the gait video; A second feature extraction operation is performed on each joint point of each patient to obtain the second coordinate feature of each joint point of each patient in each frame of gait image of the gait video; a target feature generation module is used to obtain the target coordinate feature of each joint point of the target patient in each frame of gait image of the gait video based on the first coordinate feature of each joint point of the target patient in each frame of gait image of the gait video and the second coordinate feature of each joint point of each patient in each frame of gait image of the gait video; a disease prediction module is used to obtain the disease prediction result corresponding to the target patient of the gait video based on the target coordinate feature of each joint point of the target patient in each frame of gait image of the gait video; wherein the target feature generation module is communicatively connected with the first feature extraction module, the second feature extraction module, and the disease prediction module respectively.

[0079] To summarize, compared with the existing disease prediction models, when processing the joint point information of patients in gait videos, the problem of incorrect joint point recognition is generally encountered, such as mistaking the joint point features of the left leg for the joint point features of the right leg, extracting incorrect joint point information resulting in feature instability, and difficulty in accurately distinguishing and extracting each person's joint point coordinate information, and being unable to clearly identify which joint point belongs to a specific patient. The disease prediction model of the present application extracts the joint point coordinate information of the target patient through one module, and then extracts the joint point coordinate information of each patient in the gait video through another module, and processes the joint point information of the patient extracted by the two modules. This method can obtain more accurate joint point coordinate information of the target patient, thereby improving the performance of the disease prediction model and thus improving the accuracy of the disease prediction results.

[0080] In addition, according to the disease prediction model of the above embodiment, the present application provides an application method of the disease prediction model, such as Figure 4 As shown, the method includes: T1, obtaining a gait video of a patient; T2, using a disease prediction model as in the above embodiment based on the gait video of the patient to obtain a disease prediction result corresponding to the patient in the gait video.

[0081] For example, scoliosis is the most common spinal deformity among children and adolescents, and is the third largest "killer" of children's health after obesity and myopia. Its incidence and severity rate are significantly higher than the international average. Since the skeletal system of adolescents is in the growth and development stage, there is a risk of further aggravation of scoliosis along with the overall development of the spine. If it is not discovered and intervened early, as scoliosis continues to worsen, it will compress important organs such as the heart and lungs, affecting their normal development, causing abnormal functions of important organs such as the heart and lungs of adolescents, and making it impossible to complete daily activities normally. In mild cases, it will cause back pain, which will significantly reduce the patient's quality of life; in severe cases, it will shorten their lifespan. At the same time, patients often bear a profound psychological burden due to the obvious abnormalities in their appearance. Therefore, implementing "early detection" is the most critical link in the prevention and control of scoliosis.

[0082] Currently, the commonly used screening methods at home and abroad include visual inspection, Adams flexion test (physical examination), corrugated image examination, scoliosis measuring ruler, portable electronic scoliosis screening tool and X-ray measurement of Cobb angle. Their advantages and disadvantages can be seen in Table 1.

[0083] Table 1

[0084]

[0085]

[0086] From the analysis of Table 1, we can see that the current screening methods require students to take off their shirts and rely on professional doctor examinations and manual measurements. In addition to being inefficient, there is also a certain degree of subjective error, which can easily lead to a high false positive rate.

[0087] The current mainstream scoliosis screening and assessment methods have the following difficulties:

[0088] 1) The early symptoms of scoliosis are not obvious and are easily ignored. Many patients do not have obvious pain and functional impairment in the early stage, and the scoliosis angle is small and difficult to detect.

[0089] 2) Limitations of screening methods. Currently, physical examinations such as the Adam's flexion test are mainly used, which have limited sensitivity and specificity. Although X-rays are the gold standard, they are not suitable for routine screening due to radiation exposure.

[0090] 3) The selection of screening subjects is not uniform. Some are mainly school-age children, while others include adolescents. The incidence and characteristics of scoliosis in different age groups are different, which makes screening difficult.

[0091] 4) Low screening coverage and poor follow-up compliance. Due to lack of awareness, some students and parents do not pay enough attention to it, and there are difficulties in school organization, resulting in low screening coverage and high loss of follow-up rate for positive cases.

[0092] 5) Lack of unified diagnosis and treatment standards. The medical level varies from place to place, and there is a lack of standards for the diagnosis, treatment and follow-up of scoliosis, which will affect the triage and patient management after screening.

[0093] 6) The cost-effectiveness of screening needs to be weighed. Large-scale screening requires manpower and material resources, and requires a balance between input and output.

[0094] The disease prediction model of the present application uses human posture estimation technology (Pose estimation) to obtain the coordinates of the joint points during the human gait process to predict whether the patient suffers from scoliosis, without the need for the patient to undress or stick sensors on the human body surface. This technology is used to capture the dynamic biological information of the human body during movement within a time period, rather than the static information at a certain moment. Combining optical systems and computer vision technology, the patient's motion characteristics are identified and tracked based on RGB 2D images (i.e., videos taken by ordinary mobile phone cameras), and the characteristic data of the target dynamic movement are extracted. The disease prediction model of the present application can be used to more conveniently and accurately predict whether the patient has scoliosis, and the doctor can make a more accurate diagnosis based on the prediction results. It should be noted that the use of the disease prediction model constructed in the present application to predict whether the patient's spine is scoliotic is only an embodiment of the present application. The disease prediction model of the present application can also be used for other results that can predict whether the patient suffers from the corresponding disease based on the patient's joint point information. The application scenario of the disease prediction model here is not specifically limited.

[0095] Based on the inventive concept of the above embodiments, the present application further provides an electronic device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps of the method described in the above embodiments are implemented. Figure 5 Provide detailed explanation.

[0096] like Figure 5 As shown, it shows the electronic device of the present application, which may specifically include a processor 110 and a memory 120. The memory 120 is coupled to the processor 110.

[0097] The processor 110 is used to control the operation of the electronic device. The processor 110 may also be referred to as a CPU (Central Processing Unit). The processor 110 may be an integrated circuit chip having the ability to process signals. The processor 110 may also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components. A general-purpose processor may be a microprocessor or the processor 110 may also be any conventional processor, etc.

[0098] The memory 120 is used to store computer programs, and may be a RAM, a ROM, or other types of storage terminals. Specifically, the memory 120 may include one or more computer-readable storage media, which may be non-transitory or transient. The memory 120 may also include a high-speed random access memory, and a non-volatile memory, such as one or more disk storage terminals, flash memory storage terminals. In some embodiments, the non-transitory computer-readable storage medium in the memory 120 is used to store at least one program code.

[0099] The processor 110 is used to execute the computer program stored in the memory 120 to implement the methods described in the various method embodiments of the present application.

[0100] In some embodiments, the electronic device may further include: a peripheral terminal interface 130 and at least one peripheral terminal. The processor 110, the memory 120 and the peripheral terminal interface 130 may be connected via a bus or a signal line. Each peripheral terminal may be connected to the peripheral terminal interface 130 via a bus, a signal line or a circuit board. Specifically, the peripheral terminal includes: at least one of a radio frequency circuit 140, a display screen 150, an audio circuit 160 and a power supply 170.

[0101] The peripheral terminal interface 130 may be used to connect at least one peripheral terminal related to I / O (Input / output) to the processor 110 and the memory 120. In some embodiments, the processor 110, the memory 120, and the peripheral terminal interface 130 are integrated on the same chip or circuit board; in some other implementations, any one or two of the processor 110, the memory 120, and the peripheral terminal interface 130 may be implemented on a separate chip or circuit board, which is not limited in this embodiment.

[0102] The radio frequency circuit 140 is used to receive and transmit RF (Radio Frequency) signals, also known as electromagnetic signals. The radio frequency circuit 140 communicates with the communication network and other IoT devices through electromagnetic signals, and the radio frequency circuit 140 is the communication circuit of the electronic device. The radio frequency circuit 140 converts the electrical signal into an electromagnetic signal for transmission, or converts the received electromagnetic signal into an electrical signal. Optionally, the radio frequency circuit 140 includes: an antenna system, an RF transceiver, one or more amplifiers, a tuner, an oscillator, a digital signal processor, a codec chipset, an operator identity module card, and the like. The radio frequency circuit 140 can communicate with other terminals through at least one wireless communication protocol. The wireless communication protocol includes but is not limited to: the World Wide Web, a metropolitan area network, an intranet, various generations of mobile communication networks (2G, 3G, 4G and 5G), a wireless local area network and / or a WiFi (Wireless Fidelity) network. In some embodiments, the radio frequency circuit 140 may also include circuits related to NFC (Near Field Communication), which is not limited in this application.

[0103] The display screen 150 is used to display the UI (User Interface). The UI may include graphics, text, icons, videos, and any combination thereof. When the display screen 150 is a touch display screen, the display screen 150 also has the ability to collect touch signals on the surface or above the surface of the display screen 150. The touch signal can be input to the processor 110 as a control signal for processing. At this time, the display screen 150 can also be used to provide virtual buttons and / or virtual keyboards, also known as soft buttons and / or soft keyboards. In some embodiments, the display screen 150 can be one, set on the front panel of the electronic device; in other embodiments, the display screen 150 can be at least two, respectively set on different surfaces of the electronic device or in a folding design; in other embodiments, the display screen 150 can be a flexible display screen, set on a curved surface or a folding surface of the electronic device. Even, the display screen 150 can also be set to a non-rectangular irregular shape, that is, a special-shaped screen. The display screen 150 can be made of materials such as LCD (Liquid Crystal Display), OLED (Organic Light-Emitting Diode, organic light-emitting diode).

[0104] The audio circuit 160 may include a microphone and a speaker. The microphone is used to collect sound waves from the operator and the environment, and convert the sound waves into electrical signals and input them into the processor 110 for processing, or input them into the radio frequency circuit 140 to achieve voice communication. For the purpose of stereo acquisition or noise reduction, there can be multiple microphones, which are respectively arranged in different parts of the electronic device. The microphone can also be an array microphone or an omnidirectional acquisition microphone. The speaker is used to convert the electrical signal from the processor 110 or the radio frequency circuit 140 into sound waves. The speaker can be a traditional film speaker or a piezoelectric ceramic speaker. When the speaker is a piezoelectric ceramic speaker, it can not only convert the electrical signal into sound waves audible to humans, but also convert the electrical signal into sound waves inaudible to humans for purposes such as ranging. In some embodiments, the audio circuit 160 may also include a headphone jack.

[0105] The power supply 170 is used to power various components in the electronic device. The power supply 170 can be an alternating current, a direct current, a disposable battery, or a rechargeable battery. When the power supply 170 includes a rechargeable battery, the rechargeable battery can be a wired rechargeable battery or a wireless rechargeable battery. A wired rechargeable battery is a battery that is charged through a wired line, and a wireless rechargeable battery is a battery that is charged through a wireless coil. The rechargeable battery can also be used to support fast charging technology.

[0106] For a detailed description of the functions and execution processes of each functional module or component in the electronic device embodiment of the present application, reference can be made to the description in the above-mentioned method embodiments of the present application, which will not be repeated here.

[0107] In the several embodiments provided in the present application, it should be understood that the disclosed electronic devices and methods can be implemented in other ways. For example, the various embodiments of the electronic devices described above are only schematic. For example, the division of modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0108] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0109] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0110] Based on the inventive concept of the above embodiment, the present application also provides a computer-readable storage medium storing a computer program, which can be executed by a processor to implement the above projection area calculation method. Figure 6 The execution process of the above-mentioned embodiment in a computer-readable storage medium is described.

[0111] like Figure 6 As shown, it shows the computer-readable storage medium of the present application. If the above-mentioned integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium 200. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions / computer programs to enable an IoT device (which can be a personal computer, server, or network terminal, etc.) or a processor (processor) to perform all or part of the steps of each implementation method of the present application. The aforementioned storage medium includes: various media such as a USB flash drive, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a disk or an optical disk, and electronic terminals such as computers, mobile phones, laptops, tablet computers, cameras, etc. having the above-mentioned storage medium.

[0112] The description of the execution process of the program data in the computer-readable storage medium can refer to the description in the above-mentioned method embodiments of the present application, which will not be repeated here.

[0113] The above descriptions are merely embodiments of the present application and are not intended to limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present application.

[0114] Those skilled in the art will appreciate that, in the above method of specific implementation, the order in which the steps are written does not imply a strict execution order and does not constitute any limitation on the implementation process. The specific execution order of the steps should be determined by their functions and possible internal logic.

Claims

1. A method for constructing a disease prediction model, wherein the disease prediction model is used to predict a disease based on a patient's gait video, characterized in that: The method comprises: Acquire a gait video set, wherein the gait video set includes a plurality of gait videos, each of the gait videos includes a plurality of gait image frames, each of the gait image frames includes one or more patients, and the body structure of each patient includes a plurality of joint points; Constructing an initial disease prediction model, wherein the initial disease prediction model includes: A first feature extraction module is used to take each gait video as a target gait video and a patient in the target gait video as a target patient, and perform a first feature extraction operation on each joint point of the target patient in each frame of gait image of the target gait video to obtain a first coordinate feature of each joint point of the target patient in each frame of gait image of the target gait video; A second feature extraction module is used to perform a second feature extraction operation on each joint point of each patient in each frame of the gait image of the target gait video to obtain a second coordinate feature of each joint point of each patient in each frame of the gait image of the target gait video; a target feature generating module, configured to obtain a target coordinate feature of each joint point of the target patient in each gait image frame of the target gait video based on the first coordinate feature of each joint point of the target patient in each gait image frame of the target gait video and the second coordinate feature of each joint point of each patient in each gait image frame of the target gait video; A disease prediction module, used to obtain a disease prediction result corresponding to a target patient in the target gait video based on a target coordinate feature of each joint point of the target patient in each frame of the gait image of the target gait video; Wherein, the target feature generation module is respectively connected to the first feature extraction module, the second feature extraction module, and the disease prediction module in communication; The initial disease prediction model is trained multiple times iteratively using the gait video set until convergence.

2. The method for constructing a disease prediction model according to claim 1, characterized in that: The first feature extraction module is configured as follows: A top-down human posture estimation method is used to perform a first feature extraction operation on each joint point of the target patient in each frame of the gait image of the target gait video, so as to obtain a first coordinate feature of each joint point of the target patient in each frame of the gait image of the target gait video.

3. The method for constructing a disease prediction model according to claim 1, characterized in that: The second feature extraction module is configured as follows: A second feature extraction operation is performed on each joint point of each patient in each frame of the gait image of the target gait video using a bottom-up human posture estimation method to obtain second coordinate features of each joint point of each patient in each frame of the gait image of the target gait video.

4. The method for constructing a disease prediction model according to claim 1, characterized in that: When each frame of the gait image includes a patient, the target feature generation module is configured to obtain the target coordinate feature of each joint point of the target patient in each frame of the gait image of the target gait video in the following manner: The first coordinate feature of each joint point of the target patient in each frame of the gait image of the target gait video is fused with the second coordinate feature of the corresponding joint point of the target patient in the frame of the gait image to obtain the target coordinate feature of each joint point of the target patient in each frame of the gait image of the target gait video.

5. The method for constructing a disease prediction model according to claim 1, characterized in that: When each frame of the gait image includes multiple patients, the target feature generation module is configured to obtain the target coordinate feature of each joint point of the target patient in each frame of the gait image of the target gait video in the following manner: The first coordinate feature of each joint point of the target patient in each frame of the gait image of the target gait video is respectively calculated with the second coordinate features of the corresponding joint points of each patient in the frame of the gait image to obtain multiple mean square error values, and the second coordinate feature of each joint point of the patient corresponding to the smallest mean square error value among the multiple mean square error values ​​is used as the target coordinate feature of each joint point of the target patient in each frame of the gait image of the target gait video.

6. The method according to claim 5, characterized in that Multiple mean square error values ​​are obtained in the following way: Where m is the number of patients in each frame of gait image, n is the number of joint points of the patient, and MSE j represents the jth mean square error, a i The first coordinate feature of the i-th joint point of the target patient in each frame of the gait image of the target gait video, Represents the second coordinate feature of the i-th joint point of the j-th patient in each frame of the gait image of the target gait video.

7. A disease prediction model constructed using the method for constructing a disease prediction model according to any one of claims 1 to 6, wherein the disease prediction model is used to predict a disease based on a patient's gait video, wherein: The gait video includes multiple frames of gait images, each frame of gait image includes one or more patients, and the body structure of each patient includes multiple joint points, characterized in that the model includes: a first feature extraction module, configured to take a patient in the gait video as a target patient, and perform a first feature extraction operation on each joint point of the target patient in each frame of the gait image of the gait video, so as to obtain a first coordinate feature of each joint point of the target patient in each frame of the gait image of the gait video; A second feature extraction module is used to perform a second feature extraction operation on each joint point of each patient in each frame of the gait image of the gait video to obtain a second coordinate feature of each joint point of each patient in each frame of the gait image of the gait video; a target feature generation module, for obtaining a target coordinate feature of each joint point of the target patient in each gait image frame of the gait video based on a first coordinate feature of each joint point of the target patient in each gait image frame of the gait video and a second coordinate feature of each joint point of each patient in each gait image frame of the gait video; A disease prediction module, used for obtaining a disease prediction result corresponding to a target patient in the gait video based on a target coordinate feature of each joint point of the target patient in each frame of the gait video; Among them, the target feature generation module is communicatively connected with the first feature extraction module, the second feature extraction module, and the disease prediction module respectively.

8. A method for applying a disease prediction model, characterized in that: The method comprises: Obtain a video of the patient's gait; The disease prediction model as claimed in claim 7 is used based on the gait video of the patient to obtain a disease prediction result corresponding to the patient in the gait video.

9. An electronic device, characterized in that: The method comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the method according to any one of claims 1 to 6 and 8 when executing the computer program.

10. A computer-readable storage medium having program instructions stored thereon, characterized in that: When the program instructions are executed by a processor, the method described in any one of claims 1-6 and 8 is implemented.

Images