Rehabilitation evaluation system, electronic equipment and storage medium

Abstract

The invention relates to a rehabilitation evaluation system, electronic equipment and a storage medium, the system comprises a user side, a server side and a management side, and the user side comprises a data acquisition module used for acquiring an action video and personal information of a user; the key information identification module is used for obtaining key information of a person in each video frame based on the action video; the data desensitization module is used for desensitizing the action video to obtain a first video; the server side comprises an evaluation module used for obtaining a first evaluation result based on the first video, the key information and the personal information; the management end comprises a first display module for displaying a first evaluation result, a first video and a key point; the manual adjustment module responds to a received modification instruction and modifies the first evaluation result to obtain a second evaluation result; and the user side comprises a second display module for displaying the second evaluation result. The system can improve the medical resource utilization rate of a unit; the privacy of the user can be protected, and the evaluation result can be timely and accurately provided.

Description

Technical Field

[0001] The present disclosure relates to the field of rehabilitation technology, and in particular to a rehabilitation assessment system, an electronic device and a storage medium. Background Art

[0002] As the aging population continues to intensify, the incidence of various diseases continues to increase, and the demand for rehabilitation is growing year by year. During the rehabilitation process, due to the different physical fitness of patients, the rehabilitation conditions of each patient may be different, so it is necessary to conduct rehabilitation assessments and formulate rehabilitation plans for patients in stages. Moreover, the patient's illness is personal privacy, and the patient's privacy should be protected during the rehabilitation assessment process.

[0003] However, due to factors such as uneven and insufficient distribution of medical resources, inconvenience for patients to seek medical treatment, and the need for personalized rehabilitation guidance, patients cannot receive accurate rehabilitation assessments and rehabilitation plans in a timely manner while protecting their privacy. Summary of the invention

[0004] In view of this, the present disclosure proposes a rehabilitation assessment scheme.

[0005] According to one aspect of the present disclosure, a rehabilitation assessment system is provided, comprising: a user end, a server end, and a management end, wherein the user end is communicatively connected to the server end, and the server end is communicatively connected to the management end;

[0006] The user terminal includes: a data acquisition module, which is used to obtain the action video of the user and the personal information of the user; a key information identification module, which is used to estimate the action posture based on the action video, and obtain the key information of the person in each video frame, wherein the key information includes: key points and key point coordinates; a data desensitization module, which is used to perform desensitization processing on the action video, and obtain a first video that does not show the privacy information and retains the action form;

[0007] The server side includes: an evaluation module, configured to obtain a first evaluation result based on the first video, the key information, and the personal information;

[0008] The management terminal includes: a first display module, used to display the first evaluation result, the first video, and the key points; a manual adjustment module, used to modify the first evaluation result in response to a received modification instruction to obtain a second evaluation result;

[0009] The user terminal also includes: a second display module, and the second display module is used to display the second evaluation result.

[0010] In a possible implementation, the key information further includes: a first frame number corresponding to each key point, where the first frame number represents a video frame to which the key point belongs;

[0011] The management end includes: a restoration module;

[0012] The restoration module is used to add key points in each video frame of the first video according to the first frame number and the key point coordinates to obtain a second video;

[0013] The first display module is used to display the second video.

[0014] In a possible implementation, the key information further includes: a first key frame number;

[0015] The evaluation module is further used to identify a first key frame in the first video and record the first key frame number;

[0016] The management end includes: a restoration module;

[0017] The restoration module is used to determine the first key frame in the first video according to the first key frame number;

[0018] According to the first key frame number and the key point coordinates, the corresponding key point is loaded on the first key frame to obtain a third video;

[0019] The first display module is used to display the third video;

[0020] The second display module is used to display the first video; and / or,

[0021] Determine the first key frame in the first video according to the first key frame number;

[0022] According to the first key frame number and the key point coordinates, the corresponding key point is loaded on the first key frame to obtain the third video;

[0023] The third video is displayed.

[0024] In a possible implementation, the manual adjustment module is further used to, in response to the annotation of the first video, determine a second key frame in the first video, and record a second key frame number of the second key frame;

[0025] The management end includes: a restoration module;

[0026] The restoration module is used to:

[0027] Determine the second key frame in the first video according to the second key frame number;

[0028] According to the frame number and the key point coordinates, the key points of the corresponding points are loaded on the second key frame to obtain a fourth video;

[0029] The first display module is used to display the fourth video;

[0030] The second display module is used to display the first video; and / or,

[0031] Determine the second key frame in the first video according to the second key frame number;

[0032] According to the frame number and the key point coordinates, the key points of the corresponding points are loaded on the second key frame to obtain the fourth video;

[0033] The fourth video is displayed.

[0034] In a possible implementation, the key information includes: the confidence of the key point, and the evaluation module includes: a feature acquisition module and an artificial intelligence evaluation module.

[0035] The feature acquisition module is used to determine the motion feature according to the first video, the key point coordinates, and the confidence level;

[0036] The artificial intelligence evaluation module is used to determine the first evaluation result based on the motion features, the first video, and the key point coordinates.

[0037] In a possible implementation, the data desensitization module is further used to:

[0038] Identifying a face region and a body region in each video frame of the action video;

[0039] The face area and the body area are filled with a first color, and other areas except the face area and the body area are filled with a second color to obtain the first video.

[0040] In a possible implementation, the personal information includes: the disease information of the user, and the collection module is further used to:

[0041] Determining a standard action according to the disease information of the user;

[0042] When the user performs the standard action, obtaining the action video;

[0043] The second display module is used to display the standard action.

[0044] In a possible implementation, the data desensitization module is further used to:

[0045] Based on each video frame of the action video, determining coordinate data representing the boundaries of a face and a body in each video frame;

[0046] The restoration module is used to generate the first video according to the coordinate data.

[0047] According to another aspect of the present disclosure, an electronic device is provided, comprising: a processor; and a memory for storing processor-executable instructions; wherein the processor is configured to implement the functions of the above system when executing the instructions stored in the memory.

[0048] According to another aspect of the present disclosure, a non-volatile computer-readable storage medium is provided, on which computer program instructions are stored, wherein the computer program instructions implement the functions of the above system when executed by a processor.

[0049] According to another aspect of the present disclosure, a computer program product is provided, including a computer-readable code, or a non-volatile computer-readable storage medium carrying the computer-readable code. When the computer-readable code runs in a processor of an electronic device, the processor in the electronic device executes the above method.

[0050] In the disclosed embodiment, the user end can collect the user's action video, desensitize the action video, and obtain the first video. Except for the user end, the server end and the management end will not obtain the action video with the user's privacy, which well protects the user's privacy. The server end can evaluate the user's recovery condition based on the first video, key point information, and personal information to obtain a first evaluation result. Since the machine automatically evaluates and automatically generates the first evaluation result, the speed is fast. On the management end, the doctor can understand the user's condition or recovery condition based on the first video, key information, personal information, and reference to the first evaluation result, and give professional and more appropriate diagnosis and treatment opinions for the user's actual condition. In addition, the doctor can adjust the first evaluation result to obtain a second evaluation result with higher accuracy. In this way, under the condition of accurate evaluation, the time for evaluating the user's recovery condition can be saved and the utilization rate of the unit's medical resources can be improved. In addition, the user can obtain accurate evaluation results in a timely manner while protecting his privacy.

[0051] Further features and aspects of the present disclosure will become apparent from the following detailed description of exemplary embodiments with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] The accompanying drawings, which are incorporated in and constitute a part of the specification, illustrate exemplary embodiments, features, and aspects of the disclosure and, together with the description, serve to explain the principles of the disclosure.

[0053] Figure 1A schematic diagram of the structure of a rehabilitation assessment system provided in an embodiment of the present disclosure.

[0054] Figure 2 A schematic diagram of the structure of another rehabilitation assessment system provided in an embodiment of the present disclosure.

[0055] Figure 3 A schematic diagram of the structure of an electronic device for rehabilitation assessment provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0056] Various exemplary embodiments, features and aspects of the present disclosure will be described in detail below with reference to the accompanying drawings. The same reference numerals in the accompanying drawings represent elements with the same or similar functions. Although various aspects of the embodiments are shown in the accompanying drawings, the drawings are not necessarily drawn to scale unless otherwise specified.

[0057] The word “exemplary” is used exclusively herein to mean “serving as an example, example, or illustration.” Any embodiment described herein as “exemplary” is not necessarily to be construed as preferred or advantageous over other embodiments.

[0058] In addition, in order to better illustrate the present disclosure, numerous specific details are given in the following specific embodiments. It should be understood by those skilled in the art that the present disclosure can also be implemented without certain specific details. In some examples, methods, means, components and circuits well known to those skilled in the art are not described in detail in order to highlight the subject matter of the present disclosure.

[0059] Figure 1 A structural diagram of a rehabilitation assessment system provided by an embodiment of the present disclosure. The system includes: a user end, a server end, and a management end, wherein the user end is connected to the server end in communication, and the server end is connected to the management end in communication; the user end includes: a data acquisition module for acquiring a user's action video and the user's personal information; a key information identification module for performing action posture estimation based on the action video to obtain key information of a person in each video frame, wherein the key information includes: key points and key point coordinates; a data desensitization module for desensitizing the action video to obtain a first video that does not display privacy information and retains the action form; the server end includes: an evaluation module for obtaining a first evaluation result based on the first video, the key information, and the personal information; the management end includes: a first display module for displaying the first evaluation result, the first video, and the key points; a manual adjustment module for modifying the first evaluation result in response to a received modification instruction to obtain a second evaluation result; the user end also includes: a second display module, wherein the second display module is used to display the second evaluation result.

[0060] The user end can be an electronic device with communication, display, and data processing capabilities. For example, the user end can be a mobile phone, a tablet computer, a laptop computer, or other intelligent terminal devices with communication, display, and data processing capabilities. The user end can be operated and used by the user side. The server end can be a cloud server or a physical server. The server end has at least communication and data processing capabilities. The server end can be maintained and managed by the medical institution side, or by a third party. The management end can be an electronic device with communication, display, and data processing capabilities. For example, a laptop computer, a desktop computer, a medical intelligent terminal, etc. The user end can be operated and used by a medical service agency.

[0061] In the disclosed embodiment, the user end may include a data acquisition module, a key information identification module, a data desensitization module, and a second display module. The server end may include an evaluation module. The management end may include a first display module and a manual adjustment module.

[0062] The data acquisition module may include a camera, such as an RGB camera built into a mobile device, to collect action videos when the user performs an action. The data acquisition module may perform action recognition and start action video collection when the action performed by the user is a predetermined action or is similar to a predetermined action. Alternatively, the user may issue a collection instruction to the data acquisition module through voice or touch screen; the data acquisition module may start action video collection in response to receiving the collection instruction.

[0063] The data acquisition module can obtain the user's personal information in response to the user's input operation. Among them, personal information may include: the user's physiological information and disease information. Physiological information can be information that characterizes the user's own physiological characteristics, such as: name, identity identification code (identity card number or medical card number, mobile phone number, etc.), height, weight age, gender, etc. Disease information can be information that characterizes the user's condition, physical recovery, and diagnosis, such as: onset time, number of onsets, onset location, rehabilitation record. In another example, the data acquisition module can automatically obtain the user's personal information based on the identity identification code entered by the user. The disclosed embodiment does not limit the method for the data acquisition module to obtain personal information.

[0064] The key information recognition module can perform posture estimation on the action displayed in the action video based on the action video. For example, the RTMPose algorithm, DeepPose algorithm, etc. can be used for posture estimation. Through posture estimation, the key information module can obtain the key information of the characters in the video. Among them, the key information may include: key points, key point coordinates, etc. The key points can represent the points of the character's joints, and the coordinates of the key points can represent the positions of the character's joints. The key point coordinates can include the horizontal and vertical coordinates of the key points in the video frame. The key point coordinates can also include the frame number of the video frame.

[0065] The data desensitization module can be a process for removing privacy information from the action video while retaining the action form, and the first video can be obtained after the processing. The privacy information can be the user's facial information, body information, or information of other user-specified areas. For example: the data desensitization module can use the SAM2 algorithm to segment each video frame, obtain the area containing privacy information, and fill these areas with a preset pattern. The first video does not show privacy information, but can show the action form of the character, and the action form of the character in the first video can be consistent with the action form of the character in the action video.

[0066] The user end and the server end can communicate. The server end can obtain the first video, personal information, and key information. The evaluation module can be configured as a module for executing a neural network based on an attention mechanism, wherein the neural network can be a recurrent neural network, or a long short-term memory artificial neural network, etc. The evaluation module can be configured as a module for executing a decision tree algorithm. The evaluation module can capture the relationship between image feature vectors based on the first video and key information in the time dimension, and map the image feature vectors to the action score space in combination with personal information, and construct the relationship between the action and the score, thereby evaluating the character action in the first video and obtaining a first evaluation result. A motion scale can be built into the evaluation module, and an action score space can be obtained based on the motion scale. The action score space can characterize the various actions performed by users of different ages and conditions, as well as the scores of the action performance postures. The first evaluation result can include the scoring results of the user's actions and the rehabilitation plan.

[0067] The server side can communicate with the management side, and the management side can obtain the first video, personal information, key information, and the first evaluation result.

[0068] The first display module can display the first evaluation result, personal information, and the first video and key points. The doctor can determine whether the first evaluation result needs to be modified based on the first video, key points and personal information. If no modification is required, a confirmation instruction can be given by clicking a button; the manual adjustment module can respond to the received confirmation instruction and directly use the first evaluation result as the second evaluation result. If modification is required, a modification instruction can be issued to specify the modification location and the correct content after modification. The manual adjustment module can respond to the received modification instruction to modify the first evaluation result and obtain the second evaluation result.

[0069] The user end and the management end can communicate through the server end. The user end can obtain the second evaluation result. The second display module can display the second evaluation result.

[0070] In the disclosed embodiment, the user end can collect the user's action video, desensitize the action video, and obtain the first video. Except for the user end, the server end and the management end will not obtain the action video with the user's privacy, which well protects the user's privacy. The server end can evaluate the user's recovery condition based on the first video, key point information, and personal information to obtain a first evaluation result. Since the machine automatically evaluates and automatically generates the first evaluation result, the speed is fast. On the management end, the doctor can understand the user's condition or recovery condition based on the first video, key information, personal information, and reference to the first evaluation result, and give professional and more appropriate diagnosis and treatment opinions for the user's actual condition. In addition, the doctor can adjust the first evaluation result to obtain a second evaluation result with higher accuracy. In this way, under the condition of accurate evaluation, the time for evaluating the user's recovery condition can be saved and the utilization rate of the unit's medical resources can be improved. In addition, the user can obtain accurate evaluation results in a timely manner while protecting his privacy.

[0071] In a possible implementation, the key information also includes: a first frame number corresponding to each key point, the first frame number representing the video frame to which the key point belongs; the management end includes: a restoration module; the restoration module is used to add key points in each video frame of the first video according to the first frame number and the key point coordinates to obtain a second video; the first display module is used to display the second video.

[0072] The first frame number and the corresponding key point information can be obtained within the confirmed video frame range. Among them, the confirmed video frame range may include the video frames contained in the first video; the key point information may include the key point coordinates. The first frame number corresponding to the key point can represent the number of the video frame where the key point is located. Each key point can correspond to a first frame number and the key point coordinates of the key point in the corresponding video frame. The management end can determine the video frame where each key point is located based on the first frame number, and add each key point to the corresponding video frame according to the key point coordinates. The color of the key point can be a color that contrasts with the color of the human body area. For example: the color of the human body area can be white, and the color of the key point can be red. For the convenience of description, the first video with the key point added can be named the second video.

[0073] The second video can not only show the action form, but also show the relative positions of the character's joints in the video in conjunction with the action form, so that doctors can identify the user's recovery status and make accurate assessments, saving doctors' time and improving assessment efficiency.

[0074] In a possible implementation, the key information also includes: a first key frame number; the evaluation module is also used to identify the first key frame in the first video and record the first key frame number; the management end includes: a restoration module; the restoration module is used to determine the first key frame in the first video according to the first key frame number; load the corresponding key point on the first key frame according to the first key frame number and the key point coordinates to obtain the third video; the first display module is used to display the third video; the second display module is used to display the first video; and / or, determine the first key frame in the first video according to the first key frame number; load the corresponding key point on the first key frame according to the first key frame number and the key point coordinates to obtain the third video; display the third video.

[0075] In an embodiment of the present disclosure, the evaluation module can identify the first key frame in the first video based on the first video. As mentioned above, each video frame can correspond to a frame number, so the first key frame also corresponds to a frame number. For the convenience of description, the frame number corresponding to the first key frame is named the first key frame number. In addition, the evaluation module can use the first key frame number as key information. During the communication between the evaluation module and the management module, the management module can obtain the first key frame number. The restoration module can extract the first key frame from the first video based on the first key frame number to form a first temporary key frame video; and, according to the first key frame number, the key points corresponding to the first key frame number are added to the first key frame according to the key point coordinates. For the convenience of description, the first temporary key frame video with the key points added can be named the third video.

[0076] The first display module can display a third video. The third video is shorter than the first video or the second video, but is sufficient to display the relative position of the user's joints and the movement form, and does not affect the doctor's rehabilitation assessment. Therefore, the method in the disclosed embodiment can save the doctor's time in watching the user's movements, while helping the doctor to accurately assess, and further improve the assessment efficiency.

[0077] The user end can communicate with the server end. The user end can obtain the first key frame number. The second display module can extract the first key frame from the first video locally on the user end based on the first key frame number to form a second temporary key frame video; and, according to the first key frame number, add the key point corresponding to the first key frame number to the first key frame according to the key point coordinates to obtain the third video.

[0078] The second display module can display the first video or the third video, so that the user can watch his own actions and find the difference between his actions and the standard actions; and displaying the third video can save the user's viewing time and improve the user experience. Moreover, the user end only needs to obtain the first key frame number to obtain the third video, which reduces the data transmission volume between the server and the user end and saves communication resources.

[0079] In a possible implementation, the manual adjustment module is also used to determine the second key frame in the first video in response to the marking of the first video, and record the second key frame number of the second key frame; the management end includes: a restoration module; the restoration module is used to: determine the second key frame in the first video according to the second key frame number; load the key points of the corresponding points on the second key frame according to the frame number and the key point coordinates to obtain the fourth video; the first display module is used to display the fourth video; the second display module is used to display the first video; and / or, determine the second key frame in the first video according to the second key frame number; load the key points of the corresponding points on the second key frame according to the frame number and the key point coordinates to obtain the fourth video; display the fourth video.

[0080] In the disclosed embodiment, the doctor may annotate the key frame in the first video. The manual adjustment module may determine the second key frame in response to the annotating operation and record the frame number of the second key frame. For ease of description, the frame number of the second key frame may be named the second key frame number.

[0081] The restoration module can extract the second key frame from the first video based on the second key frame number to form a third temporary key frame video; and according to the second key frame number, the key point corresponding to the second key frame number is added to the second key frame according to the key point coordinates. For ease of description, the third temporary key frame video with the key point added can be named the fourth video.

[0082] The first display module can display the fourth video. The fourth video is shorter than the first video or the second video. Moreover, the fourth video is a key frame selected by the doctor based on the user's condition and diagnostic experience that can reflect the user's recovery condition, which is more accurate and fits the user's actual condition. Therefore, the method in the disclosed embodiment can improve the accuracy and efficiency of the doctor's evaluation.

[0083] The user end and the management end can communicate through the server end. The user end can obtain the second key frame number. The second display module can extract the second key frame from the first video locally on the user end based on the second key frame number to form a fourth temporary key frame video; and, according to the second key frame number, add the key point corresponding to the second key frame number to the second key frame according to the key point coordinates to obtain the fourth video.

[0084] The second display module can display the first video or the fourth video, so that users can watch their own movements and find the differences between their movements and standard movements. Since the fourth video is composed of the second key video frame selected by the doctor, it is more accurate to show the user's recovery from a medical professional perspective. In addition, displaying the fourth video can save users' viewing time and improve user experience. Moreover, the user end only needs to obtain the second key frame number to obtain the fourth video, which reduces the amount of data transmission between the server and the user end and saves communication resources.

[0085] In one possible implementation, the key information includes: the confidence of the key point, and the evaluation module includes: a feature acquisition module and an artificial intelligence evaluation module, the feature acquisition module is used to determine the motion feature based on the first video, the key point coordinates, and the confidence; the artificial intelligence evaluation module is used to determine the first evaluation result based on the motion feature, the first video, and the key point coordinates.

[0086] The key information identification module can determine the confidence of each key point. Each key point can correspond to a confidence level. The confidence level of the key point can represent the accuracy of the identified key point in representing the user's joint position.

[0087] The evaluation module may include a feature acquisition module and an artificial intelligence evaluation module. The feature acquisition module may use a neural network method to extract motion features based on the first video, key point coordinates, and confidence. Motion features may include: the speed of position change of each key point, the angle of change of the line connecting two adjacent key points; and the special state of the key point determined according to the speed of position change of the key point and the angle of change of the line connecting two adjacent key points. For example: In the second stage of the recovery period of disease A, if the movement speed of joint a is less than m mm / second (m is a positive number), it means that the third stage of the recovery period cannot be entered yet, and it is still necessary to stay in the second stage and the rehabilitation plan needs to be changed. At this time, the movement speed of joint point a can be recorded as less than m mm / second as a special state.

[0088] The artificial intelligence evaluation module can determine a first evaluation result based on the motion features, the first video, and the key point coordinates.

[0089] In the disclosed embodiment, motion features are extracted based on the first video, key point coordinates, and confidence, which makes up for the error of key information and improves the accuracy of motion features. In addition, the artificial intelligence evaluation module directly processes the features of the motion, saving feature extraction time, and then determines the first evaluation result in combination with the first video and key point coordinates, thereby improving the accuracy of the first evaluation result.

[0090] In a possible implementation, the data desensitization module is further used to: identify face areas and body areas in each video frame of the action video; fill the face areas and body areas with a first color, and fill other areas except the face areas and body areas with a second color to obtain the first video.

[0091] Since the first video needs to accurately show the user's actions. Therefore, the first video needs to retain information representing the range of the face area and the range of the human body area, but does not need to retain information within the face area and the human body area to protect the user's privacy. Thus, the data desensitization module can identify the face area and the human body area in each video frame of the action video. The face area and the human body area are filled with the first color, and the other areas except the face area and the human body area are filled with the second color. The first video can be a video composed of video frames filled with the face area and the human body area with the first color, and the other areas are filled with the second color. Among them, the first color can be white, and the second color can be black.

[0092] In the disclosed embodiment, only the face area and the body area need to be identified to reduce the workload of segmentation. In addition, the first video in the present application can be a binary image, which can reduce the storage space occupied by the first video and save data transmission resources. In this way, storage space and data transmission are no longer bottlenecks for remote rehabilitation assessment, which saves hardware investment on the medical institution side and improves the feasibility of remote rehabilitation assessment.

[0093] In a possible implementation, the personal information includes: the disease information of the user; the collection module is further used to: determine the standard action based on the disease information of the user; obtain the action video when the user performs the standard action; and the second display module is used to display the standard action.

[0094] As mentioned above, the disease information may be information characterizing the user's condition, physical recovery, and diagnosis, such as: onset time, number of onsets, onset location, and recovery records.

[0095] The client can communicate with the server or other databases. The client can obtain the standard action names and / or standard action videos from the server or other databases according to the disease information.

[0096] The second display module can display the standard action name and / or the standard action video so that the user can clearly understand the action he should perform.

[0097] Some diseases have a long recovery period and many recovery stages, and the actions required to evaluate the recovery status of each stage are complex or numerous. This puts a burden on the user. Using the method disclosed in the present invention, the user does not need to remember or search for the actions to be performed, but only needs to perform the actions displayed by the second display module, which reduces the burden on the user, makes the user action evaluation process simpler, and improves the user experience.

[0098] In a possible implementation, the data desensitization module is further used to: determine coordinate data representing the boundaries of a face and a body in each video frame based on each video frame of the action video; and the restoration module is used to generate the first video based on the coordinate data.

[0099] The coordinate data may include: coordinates of each face boundary point on the face region boundary, and coordinates of each body boundary point on the body region boundary. In the disclosed embodiment, the management terminal may obtain the coordinate data. The restoration module may determine the face region and body region of each video frame according to the coordinate data, and fill the face region and body region with a first color, and fill the region other than the face region and body region with a second color to generate a first video.

[0100] In the embodiment of the present disclosure, the first video does not need to be transmitted between the server and the management end, and only the coordinate data needs to be transmitted. The transmission resources occupied by the coordinate data are much smaller than those of the first video. Therefore, using the method of the embodiment of the present disclosure, transmission resources can be further saved.

[0101] In a possible implementation, the server side also includes: a cloud storage module, the cloud storage module is used to store the first video, key information, personal information, the first evaluation result, and the second evaluation result; the data desensitization module is used to delete the action video.

[0102] In the disclosed embodiment, the server side includes a cloud storage module. The cloud storage module can store the first video, key information, personal information, the first evaluation result, and the second evaluation result. The cloud storage module can also store the first key frame number, the second key frame number, and coordinate data. This can reduce the storage pressure on the user side and the management side. The data desensitization module can delete the action video after determining the first video, further reducing the storage pressure on the user side.

[0103] Figure 2A structural diagram of another rehabilitation assessment system provided for an embodiment of the present disclosure. The system includes: a user end, a server end, and a management end. The user end includes: a data acquisition module, a key information identification module, a data desensitization module, and a second display module. The server end includes an evaluation module. The evaluation module includes: a feature acquisition module and an artificial intelligence evaluation module. The management end includes: a restoration module, a manual adjustment module, and a first display module.

[0104] The user can input an evaluation instruction at the user end. In response to the received evaluation instruction, the data acquisition module displays the form for collecting personal information through the second display module. The user can input identity information in the form. The data acquisition module can obtain the standard action name and the standard action video according to the identity information. The second display module displays the standard action name and the standard action video. The user can perform actions according to the standard action video. The user can issue a collection instruction to the data acquisition module through voice or touch screen; the data acquisition module can start action video collection in response to receiving the collection instruction and obtain the action video.

[0105] The key information recognition module can perform posture estimation based on the action video to obtain key information. The data desensitization module can identify the face area and the body area in each video frame of the action video, and fill the face area and the body area with white. Fill other areas except the face area and the body area with black to obtain the first video. And obtain coordinate data.

[0106] The cloud storage module can store the first video, key information, personal information, and coordinate data.

[0107] The feature acquisition module can extract motion features based on the first video, key point coordinates, and confidence. The artificial intelligence evaluation module can determine a first evaluation result based on the motion features, the first video, and key point coordinates. And the first evaluation result is stored in the cloud storage module.

[0108] The restoration module can determine the face area and the body area of ​​each video frame according to the coordinate data, and fill the face area and the body area with white, and fill the area other than the face area and the body area with black to generate the first video.

[0109] The first display module can display the first evaluation result and the first video. The doctor can annotate the key frames in the first video. The manual adjustment module can determine the second key frame in response to the annotation operation and record the second key frame number. The restoration module can extract the second key frame from the first video based on the second key frame number to form a third temporary key frame video; and, according to the second key frame number, add the key point corresponding to the second key frame number to the corresponding second key frame according to the key point coordinates to obtain the fourth video. The first display module displays the fourth video.

[0110] The first display module can also display key information and personal information. The doctor can determine whether the first evaluation result needs to be modified in combination with the fourth video, personal information, and key information. If no modification is required, a confirmation instruction can be given by clicking a button; the manual adjustment module can respond to the received confirmation instruction and directly use the first evaluation result as the second evaluation result. If modification is required, a modification instruction can be issued to specify the modification location and the correct content after modification. The manual adjustment module can respond to the received modification instruction to modify the first evaluation result and obtain the second evaluation result.

[0111] The cloud storage module can store the second evaluation result and the second key frame number. The second display module can extract the second key frame from the first video locally on the user end based on the second key frame number to form a fourth temporary key frame video; and according to the second key frame number, add the key point corresponding to the second key frame number to the corresponding second key frame according to the key point coordinates to obtain the fourth video. The second display module can display the second evaluation result and the fourth video.

[0112] The embodiment of the present disclosure also provides a computer-readable storage medium on which computer program instructions are stored, and the computer program instructions implement the above method when executed by a processor. The computer-readable storage medium can be a volatile or non-volatile computer-readable storage medium.

[0113] An embodiment of the present disclosure further proposes an electronic device, comprising: a processor; and a memory for storing instructions executable by the processor; wherein the processor is configured to implement the above method when executing the instructions stored in the memory.

[0114] The embodiments of the present disclosure also provide a computer program product, including a computer-readable code, or a non-volatile computer-readable storage medium carrying the computer-readable code. When the computer-readable code runs in a processor of an electronic device, the processor in the electronic device executes the above method.

[0115] Figure 3The electronic device 1900 is a schematic diagram of a structure of an electronic device for rehabilitation assessment provided by an embodiment of the present disclosure. For example, the electronic device 1900 may be provided as a server or a terminal device. Figure 3 , the electronic device 1900 includes a processing component 1922, which further includes one or more processors, and a memory resource represented by a memory 1932 for storing instructions executable by the processing component 1922, such as an application. The application stored in the memory 1932 may include one or more modules, each corresponding to a set of instructions. In addition, the processing component 1922 is configured to execute instructions to perform the above method.

[0116] The electronic device 1900 may also include a power supply component 1926 configured to perform power management of the electronic device 1900, a wired or wireless network interface 1950 configured to connect the electronic device 1900 to a network, and an input / output interface 1958 (I / O interface). The electronic device 1900 may operate based on an operating system stored in the memory 1932, such as Windows Server 2003. TM , Mac OS X TM , Unix TM ,Linux TM , FreeBSD TM or similar.

[0117] In an exemplary embodiment, a non-volatile computer-readable storage medium is also provided, such as a memory 1932 including computer program instructions, which can be executed by the processing component 1922 of the electronic device 1900 to perform the above method.

[0118] The present disclosure may be a system, a method and / or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for causing a processor to implement various aspects of the present disclosure.

[0119] A computer-readable storage medium may be a tangible device that can hold and store instructions used by an instruction execution device. A computer-readable storage medium may be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. More specific examples of computer-readable storage media (a non-exhaustive list) include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disk read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device, such as a punch card or a raised structure in a groove on which instructions are stored, and any suitable combination of the foregoing. As used herein, a computer-readable storage medium is not to be interpreted as a transient signal per se, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagating through a waveguide or other transmission medium (e.g., a light pulse through a fiber optic cable), or an electrical signal transmitted through a wire.

[0120] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, optical fiber transmissions, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in the computer-readable storage medium in each computing / processing device.

[0121] The computer program instructions for performing the operation of the present disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-related instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages, such as Smalltalk, C++, etc., and conventional procedural programming languages, such as "C" language or similar programming languages. Computer-readable program instructions may be executed completely on a user's computer, partially on a user's computer, as an independent software package, partially on a user's computer, partially on a remote computer, or completely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., using an Internet service provider to connect via the Internet). In some embodiments, an electronic circuit, such as a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), may be customized by utilizing the state information of the computer-readable program instructions, and the electronic circuit may execute the computer-readable program instructions, thereby realizing various aspects of the present disclosure.

[0122] Various aspects of the present disclosure are described herein with reference to the flowcharts and / or block diagrams of the methods, devices (systems) and computer program products according to the embodiments of the present disclosure. It should be understood that each box in the flowchart and / or block diagram and the combination of each box in the flowchart and / or block diagram can be implemented by computer-readable program instructions.

[0123] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processor of the computer or other programmable data processing device, a device that implements the functions / actions specified in one or more boxes in the flowchart and / or block diagram is generated. These computer-readable program instructions can also be stored in a computer-readable storage medium, and these instructions cause the computer, programmable data processing device, and / or other equipment to work in a specific manner, so that the computer-readable medium storing the instructions includes a manufactured product, which includes instructions for implementing various aspects of the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0124] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operating steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0125] The flow chart and block diagram in the accompanying drawings show the possible architecture, function and operation of the system, method and computer program product according to multiple embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a part of a module, program segment or instruction, and a part of the module, program segment or instruction includes one or more executable instructions for realizing the specified logical function. In some alternative implementations, the function marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two continuous square boxes can actually be executed substantially in parallel, and they can sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs the specified function or action, or can be implemented with a combination of special hardware and computer instructions.

[0126] The embodiments of the present disclosure have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and changes will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The selection of terms used herein is intended to best explain the principles of the embodiments, practical applications, or technical improvements in the market, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein.

Claims

1. A rehabilitation assessment system, characterized in that: include: A user end, a server end, and a management end, wherein the user end is connected to the server end for communication, and the server end is connected to the management end for communication; The user terminal includes: a data acquisition module, which is used to obtain the action video of the user and the personal information of the user; a key information identification module, which is used to estimate the action posture based on the action video, and obtain the key information of the person in each video frame, wherein the key information includes: key points and key point coordinates; a data desensitization module, which is used to perform desensitization processing on the action video, and obtain a first video that does not show the privacy information and retains the action form; The server side includes: an evaluation module, configured to obtain a first evaluation result based on the first video, the key information, and the personal information; The management terminal includes: a first display module, used to display the first evaluation result, the first video, and the key points; a manual adjustment module, used to modify the first evaluation result in response to a received modification instruction to obtain a second evaluation result; The user terminal also includes: a second display module, and the second display module is used to display the second evaluation result.

2. The system according to claim 1, characterized in that The key information further includes: a first frame number corresponding to each key point, the first frame number representing the video frame to which the key point belongs; The management end includes: a restoration module; The restoration module is used to add key points in each video frame of the first video according to the first frame number and the key point coordinates to obtain a second video; The first display module is used to display the second video.

3. The system according to claim 1, characterized in that The key information also includes: the first key frame number; The evaluation module is further used to identify a first key frame in the first video and record the first key frame number; The management end includes: a restoration module; The restoration module is used to determine the first key frame in the first video according to the first key frame number; According to the first key frame number and the key point coordinates, the corresponding key point is loaded on the first key frame to obtain a third video; The first display module is used to display the third video; The second display module is used to display the first video; and / or, Determine the first key frame in the first video according to the first key frame number; According to the first key frame number and the key point coordinates, the corresponding key point is loaded on the first key frame to obtain the third video; The third video is displayed.

4. The system according to claim 1, characterized in that The manual adjustment module is further used to, in response to the annotation of the first video, determine a second key frame in the first video and record a second key frame number of the second key frame; The management end includes: a restoration module; The restoration module is used to: Determine the second key frame in the first video according to the second key frame number; According to the frame number and the key point coordinates, the key points of the corresponding points are loaded on the second key frame to obtain a fourth video; The first display module is used to display the fourth video; The second display module is used to display the first video; and / or, Determine the second key frame in the first video according to the second key frame number; According to the frame number and the key point coordinates, the key points of the corresponding points are loaded on the second key frame to obtain the fourth video; The fourth video is displayed.

5. The system according to claim 1, characterized in that The key information includes: the confidence of the key points, and the evaluation module includes: a feature acquisition module and an artificial intelligence evaluation module. The feature acquisition module is used to determine the motion feature according to the first video, the key point coordinates, and the confidence level; The artificial intelligence evaluation module is used to determine the first evaluation result based on the motion features, the first video, and the key point coordinates.

6. The system according to claim 1, characterized in that The data desensitization module is further used for: Identifying a face region and a body region in each video frame of the action video; The face area and the body area are filled with a first color, and other areas except the face area and the body area are filled with a second color to obtain the first video.

7. The system according to claim 1, characterized in that The personal information includes: the disease information of the user, and the collection module is further used to: Determining a standard action according to the disease information of the user; When the user performs the standard action, obtaining the action video; The second display module is used to display the standard action.

8. The system according to claim 1, characterized in that The data desensitization module is further used for: Based on each video frame of the action video, determining coordinate data representing the boundaries of a face and a body in each video frame; The restoration module is used to generate the first video according to the coordinate data.

9. An electronic device, characterized in that: include: processor; a memory for storing processor-executable instructions; Wherein, the processor is configured to implement the functions of the system described in any one of claims 1 to 8 when executing the instructions stored in the memory.

10. A non-volatile computer-readable storage medium having computer program instructions stored thereon, characterized in that: When the computer program instructions are executed by a processor, the functions of the system described in any one of claims 1 to 8 are realized.

Images