Portable cerebral apoplexy auxiliary evaluation system and method based on RGB-D camera

Abstract

The invention provides a portable cerebral apoplexy auxiliary evaluation system and method based on an RGB-D camera, and the system comprises an interactive input unit which is used for receiving an operation instruction, and collecting and outputting patient data in real time; the patient data comprises user interaction data and limb movement data; the limb movement data comprises video data and corresponding depth image data; the calculation unit is used for receiving the patient data collected by the interactive input unit and calculating to obtain time sequence motion characteristics; outputting an evaluation result according to the time sequence motion characteristics; and the data output unit is used for integrated display of the time sequence motion characteristics and the evaluation result. According to the method, the RGB-D depth camera and the calculation unit are utilized, through image examples and voice guidance, NIHSS scale limb movement rapid scoring can be completed within one minute, rapid screening capable of being operated by ordinary users by themselves is achieved, and the method has the advantages of being efficient and convenient to use.

Description

Technical Field

[0001] The present invention relates to the technical field of data processing, and in particular to a portable stroke auxiliary assessment system and method based on an RGB-D camera. Background Art

[0002] Stroke has the characteristics of high disability and high mortality. Stroke screening data presents the characteristics of high-dimensional and unbalanced data sets. In the existing technology, stroke screening mainly relies on the examination and clinical scoring of professional medical personnel, and its main disadvantages are as follows:

[0003] 1. Dependence on professionals: Professional medical personnel are required for evaluation, and it is difficult for ordinary users to complete the screening on their own.

[0004] 2. Complex operation: The screening procedures are complicated and usually need to be carried out in medical institutions.

[0005] 3. It takes a long time: The traditional method requires professional medical personnel to perform NIHSS physical examinations on each patient one by one, which takes about 10 to 15 minutes for a single patient. It greatly occupies professional medical personnel and consumes medical resources, making it difficult to quickly screen a large number of people. Summary of the invention

[0006] In view of the defects in the prior art, the object of the present invention is to provide a portable stroke auxiliary assessment system and method based on an RGB-D camera.

[0007] A portable stroke auxiliary assessment system based on an RGB-D camera provided by the present invention comprises:

[0008] An interactive input unit, used to receive operation instructions, collect and output patient data in real time; the patient data includes user interaction data and limb movement data; the limb movement data includes video data and corresponding depth image data;

[0009] A calculation unit, used for receiving the patient data collected by the interactive input unit, calculating the time-series motion characteristics; and outputting the evaluation results according to the time-series motion characteristics;

[0010] Data output unit for integrated display of temporal motion characteristics and evaluation results.

[0011] Preferably, the interactive input unit includes a screen interaction module and a computer vision input module; the computer vision input module includes using an RGB-D camera to collect limb movement data and then output it to a real-time display module as input to the calculation unit and direct data input for the NIHSS limb rapid score of stroke; the screen interaction module is used to display an interactive interface to the user and manage and store corresponding data.

[0012] Preferably, the computing unit comprises a microcomputer and a built-in processing algorithm; the built-in processing algorithm comprises a time series three-dimensional human body posture estimation algorithm, an algorithm for extracting features from time series three-dimensional human body posture, and an algorithm for preprocessing time series features;

[0013] The time-series 3D human posture estimation algorithm includes a deep neural network that extracts key points of a person's skeleton from an RGB or RGB-D image; connects and processes the patient's skeleton for each frame of the image recorded in the video stream;

[0014] The algorithm for extracting features from the time-series three-dimensional human posture includes extracting explicit features from the three-dimensional skeleton; the explicit features include the joint angles of the upper and lower limbs;

[0015] The algorithm for preprocessing time series features includes preprocessing the feature sequence; the preprocessing includes noise reduction and difference resampling.

[0016] Preferably, the computing unit also includes an artificial intelligence evaluation module; the artificial intelligence evaluation module includes a deep neural network, which takes the preprocessed time series features as input and outputs a corresponding score.

[0017] Preferably, the data output unit includes:

[0018] The data integration module is used to integrate the preprocessed time-series motion features with the scoring results output by the artificial intelligence evaluation module; integrate the physical examination data and corresponding evaluation results of the same patient, draw a medical record curve, and output the time-series features as reference information;

[0019] The real-time display module is used to display the physical examination video recorded by the computer vision input module and the timing characteristics of the calculation unit output in real time, and to display the evaluation score and related basis after the evaluation is completed.

[0020] A portable stroke auxiliary assessment method based on an RGB-D camera provided by the present invention comprises:

[0021] Step S1: enabling the interactive input unit to receive operation instructions, collect and output patient data in real time;

[0022] The patient data includes user interaction data and limb movement data; the limb movement data includes video data and corresponding depth image data;

[0023] Step S2: Instruct the calculation unit to receive the patient data collected by the interactive input unit, calculate the time series motion characteristics; and output the evaluation results according to the time series motion characteristics;

[0024] Step S3: Using a data output unit to integrate and display the temporal motion characteristics and the evaluation results.

[0025] Preferably, the interactive input unit includes a screen interaction module and a computer vision input module; the computer vision input module includes using an RGB-D camera to collect limb movement data and then output it to a real-time display module as input to the calculation unit and direct data input for the NIHSS limb rapid score of stroke; the screen interaction module is used to display an interactive interface to the user and manage and store corresponding data.

[0026] Preferably, the computing unit comprises a microcomputer and a built-in processing algorithm; the built-in processing algorithm comprises a time series three-dimensional human body posture estimation algorithm, an algorithm for extracting features from time series three-dimensional human body posture, and an algorithm for preprocessing time series features;

[0027] The time-series 3D human posture estimation algorithm includes a deep neural network that extracts key points of a person's skeleton from an RGB or RGB-D image; connects and processes the patient's skeleton for each frame of the image recorded in the video stream;

[0028] The algorithm for extracting features from the time-series three-dimensional human posture includes extracting explicit features from the three-dimensional skeleton; the explicit features include the joint angles of the upper and lower limbs;

[0029] The algorithm for preprocessing time series features includes preprocessing the feature sequence; the preprocessing includes noise reduction and difference resampling.

[0030] Preferably, the computing unit also includes an artificial intelligence evaluation module; the artificial intelligence evaluation module includes a deep neural network, which takes the preprocessed time series features as input and outputs a corresponding score.

[0031] Preferably, the data output unit includes:

[0032] The data integration module is used to integrate the preprocessed time-series motion features with the scoring results output by the artificial intelligence evaluation module; integrate the physical examination data and corresponding evaluation results of the same patient, draw a medical record curve, and output the time-series features as reference information;

[0033] The real-time display module is used to display the physical examination video recorded by the computer vision input module and the timing characteristics of the calculation unit output in real time, and to display the evaluation score and related basis after the evaluation is completed.

[0034] Compared with the prior art, the present invention has the following beneficial effects:

[0035] 1. The present invention utilizes an RGB-D depth camera and a computing unit, and through image examples and voice guidance, can complete the rapid scoring of NIHSS scale limb movement within 1 minute, realizing rapid screening that ordinary users can operate by themselves, and has the advantages of high efficiency and convenience.

[0036] 2. The system of the present invention adopts a miniaturized design, which is easy to carry and use; it utilizes human posture estimation and depth image analysis to provide accurate limb movement analysis results and calculate scientific and effective NIHSS scores, making the results more convincing and providing better decision-making advice for doctors.

[0037] 3. The present invention can perform rapid self-screening in various scenarios without relying on professionals, and has good flexibility and practicality.

[0038] Other beneficial effects of the present invention will be explained in the specific implementation manner through the introduction of specific technical features and technical solutions. Through the introduction of these technical features and technical solutions, those skilled in the art should be able to understand the beneficial technical effects brought about by the technical features and technical solutions. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Other features, objects and advantages of the present invention will become more apparent from the detailed description of non-limiting embodiments made with reference to the following drawings:

[0040] Figure 1 It is a system block diagram of the present invention.

[0041] Figure 2 The figure is a flow chart of the method of the present invention. DETAILED DESCRIPTION

[0042] The present invention is described in detail below in conjunction with specific embodiments. The following embodiments will help those skilled in the art to further understand the present invention, but are not intended to limit the present invention in any form. It should be noted that, for those of ordinary skill in the art, several changes and improvements can also be made without departing from the concept of the present invention. These all belong to the protection scope of the present invention.

[0043] Example 1

[0044] Reference Figure 1 As shown, a portable stroke auxiliary assessment system based on RGB-D camera includes:

[0045] Interactive input unit, calculation unit and data output unit.

[0046] Among them, the interactive input unit is used to accept operation instructions, collect patient video data in real time, display real-time data collection results, and visualize stroke assessment reports. It includes a screen interaction module and a computer vision input module.

[0047] The computer vision input module includes an RGB-D camera to collect video data and corresponding depth image data when the patient completes the physical examination. The data is then output to the real-time display module for user reference and serves as input to the calculation unit as direct data input for the NIHSS limb rapid score for stroke.

[0048] The screen interaction module includes a touch screen for presenting an interactive interface to the user, accepting user instructions, displaying video data collected by the RGB-D camera in real time, and presenting visualization results; it also includes a user interaction interface and a corresponding data management system, through which the user can control the start and end time points of the evaluation and select the data storage management method.

[0049] The computing unit is used to process the signal input of the interactive input unit, including the signal input of the user interaction interface and the RGB-D video signal of the computer vision input module. It includes a computing motherboard that can realize parallel computing and a built-in data processing algorithm.

[0050] The computing motherboard refers to a microcomputer with parallel computing capabilities, such as the Nvidia Orin edge computing board or a microcomputer equipped with a GPU, which can realize signal processing and related data storage. The data processing algorithm includes a time-series 3D human posture estimation algorithm, an algorithm for extracting features from time-series 3D human posture, an algorithm for preprocessing the above time-series features, and an artificial intelligence evaluation module.

[0051] The temporal 3D human pose estimation algorithm includes a pose estimation algorithm and a temporal smoothing step.

[0052] The pose estimation algorithm can be a 2D or 3D pose estimation algorithm. If it is a 2D pose estimation algorithm, the depth information is used to calculate the corresponding 3D pose. The 2D or 3D pose estimation algorithm is a deep neural network that extracts the key points of the human skeleton from the RGB or RGB-D image, including but not limited to open source solutions such as Mediapipe, OpenPose or a unique algorithm trained by oneself.

[0053] Temporal smoothing is used to smoothly connect the patient skeleton of each frame image recorded in the video stream, making the 3D posture sequence smooth and natural and reducing noise. The algorithm for extracting features from the temporal three-dimensional human posture is a method for extracting explicit features from the three-dimensional skeleton, including the joint angles of the upper and lower limbs. The algorithm for preprocessing temporal features includes denoising, smoothing algorithms, and interpolation resampling algorithms for feature sequences. Noise reduction and smoothing algorithms include Chebyshev filters, moving window functions, etc. Interpolation resampling methods include linear interpolation and spline interpolation, etc.

[0054] The AI ​​evaluation module includes a deep neural network that takes preprocessed time series features as input and outputs corresponding scores. This network needs to be trained in advance using healthy population data and clinical patient data.

[0055] The data output unit includes a data integration module and a real-time display module.

[0056] The data integration module integrates the preprocessed time-series motion features with the final evaluation results output by the artificial intelligence evaluation module, and saves them according to the patient's basic information as an index according to clinical usage habits; it integrates the previous physical examination data and corresponding evaluation results of the same patient, draws a medical record curve, and provides the time-series features as reference information to clinicians.

[0057] The real-time display module actually displays the physical examination video recorded by the computer vision input module, the timing characteristics output by the calculation unit, and the evaluation score and related basis after the evaluation is completed. It also provides a data search function, which can trace back the past physical examination records of the same patient and view the corresponding timing characteristics as a reference for clinical diagnosis.

[0058] Example 2

[0059] Reference Figure 1 and Figure 2 As shown, the interactive input unit includes a screen interaction module for collecting user interaction data; and a computer vision input module for collecting limb movement data of the subject.

[0060] The computing unit includes a microcomputer and built-in algorithms for processing and analyzing the data collected by the visual input module, calculating the temporal motion characteristics, such as the joint angles of the limbs, etc., and performing NIHSS evaluation after processing.

[0061] The data output unit is used to display the operation interface and guide the subjects to complete corresponding actions.

[0062] The user clicks the start button on the screen to start the assisted evaluation system. The computer vision input module starts working and collects the object RGB-D video stream.

[0063] The screen displays images and guides the subjects to complete the upper and lower limb examination movements in the NIHSS score. The RGB-D depth camera collects image data of the subjects completing the movements and transmits the data to the computing unit, while the screen displays the real-time collected data.

[0064] The computing unit uses the human posture estimation module to find the joint points of the subject and uses the depth image to obtain the time-series three-dimensional key points.

[0065] The computing unit calculates the temporal human motion features from the temporal three-dimensional human key points, and inputs the smoothed temporal motion features into the deep neural network classifier for classification and evaluation.

[0066] The calculation unit outputs the NIHSS score result and displays it on the data output unit, while taking the temporal motion characteristics as an auxiliary evaluation basis.

[0067] Compared with the prior art, the portable rapid stroke diagnosis and screening system of the present invention, by utilizing an RGB-D depth camera and a computing unit, can complete the rapid scoring of NIHSS scale limb movement within 1 minute through image examples and voice guidance, thus realizing a portable rapid screening that can be operated by ordinary users themselves. It effectively solves the problems that traditional methods greatly occupy professional medical personnel, consume medical resources, and are difficult to quickly screen a large number of people.

[0068] Those skilled in the art know that, in addition to realizing the system and its various devices, modules, and units provided by the present invention in a purely computer-readable program code, it is entirely possible to realize the same functions in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers by logically programming the method steps. Therefore, the system and its various devices, modules, and units provided by the present invention can be considered as a hardware component, and the devices, modules, and units included therein for realizing various functions can also be regarded as structures within the hardware component; the devices, modules, and units for realizing various functions can also be regarded as both software modules for realizing the method and structures within the hardware component.

[0069] In the description of the present application, it should be understood that the terms "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside", etc., indicating orientations or positional relationships, are based on the orientations or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present application and simplifying the description, and do not indicate or imply that the referred device or element must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be understood as a limitation on the present application.

[0070] The above describes the specific embodiments of the present invention. It should be understood that the present invention is not limited to the above specific embodiments, and those skilled in the art can make various changes or modifications within the scope of the claims, which does not affect the essence of the present invention. In the absence of conflict, the embodiments of the present application and the features in the embodiments can be combined with each other arbitrarily.

Claims

1. A portable stroke auxiliary assessment system based on RGB-D camera, characterized in that: include: An interactive input unit, used to receive operation instructions, collect and output patient data in real time; The patient data includes user interaction data and limb movement data; The limb movement data includes video data and corresponding depth image data; A calculation unit, used for receiving the patient data collected by the interactive input unit, and calculating the time-series motion characteristics; Output the evaluation results according to the time-series motion characteristics; Data output unit for integrated display of temporal motion characteristics and evaluation results.

2. The portable stroke auxiliary assessment system based on RGB-D camera according to claim 1 is characterized in that: The interactive input unit includes a screen interaction module and a computer vision input module; The computer vision input module includes using an RGB-D camera to collect limb movement data and then output it to a real-time display module as input to a calculation unit and direct data input for the NIHSS rapid limb score for stroke; the screen interaction module is used to display an interactive interface to the user and manage and store corresponding data.

3. The portable stroke auxiliary assessment system based on RGB-D camera according to claim 2 is characterized in that: The computing unit includes a microcomputer and a built-in processing algorithm; the built-in processing algorithm includes a time series three-dimensional human body posture estimation algorithm, an algorithm for extracting features from time series three-dimensional human body postures, and an algorithm for preprocessing time series features; The time-series 3D human posture estimation algorithm includes a deep neural network that extracts key points of a person's skeleton from an RGB or RGB-D image; connects and processes the patient's skeleton for each frame of the image recorded in the video stream; The algorithm for extracting features from the time-series three-dimensional human posture includes extracting explicit features from the three-dimensional skeleton; the explicit features include the joint angles of the upper and lower limbs; The algorithm for preprocessing time series features includes preprocessing the feature sequence; the preprocessing includes noise reduction and difference resampling.

4. The portable stroke auxiliary assessment system based on RGB-D camera according to claim 3 is characterized in that: The computing unit also includes an artificial intelligence evaluation module; the artificial intelligence evaluation module includes a deep neural network, which takes the preprocessed time series features as input and outputs a corresponding score.

5. The portable stroke auxiliary assessment system based on RGB-D camera according to claim 4 is characterized in that: The data output unit comprises: The data integration module is used to integrate the preprocessed time-series motion features with the scoring results output by the artificial intelligence evaluation module; integrate the physical examination data and corresponding evaluation results of the same patient, draw a medical record curve, and output the time-series features as reference information; The real-time display module is used to display the physical examination video recorded by the computer vision input module and the timing characteristics of the calculation unit output in real time, and to display the evaluation score and related basis after the evaluation is completed.

6. A portable stroke auxiliary assessment method based on RGB-D camera, characterized in that: include: Step S1: enabling the interactive input unit to receive operation instructions, collect and output patient data in real time; The patient data includes user interaction data and limb movement data; the limb movement data includes video data and corresponding depth image data; Step S2: Instruct the calculation unit to receive the patient data collected by the interactive input unit and calculate the time series motion characteristics; Output the evaluation results according to the time-series motion characteristics; Step S3: Using a data output unit to integrate and display the temporal motion characteristics and the evaluation results.

7. The portable stroke auxiliary assessment method based on RGB-D camera according to claim 6 is characterized in that: The interactive input unit includes a screen interaction module and a computer vision input module; The computer vision input module includes using an RGB-D camera to collect limb movement data and then output it to a real-time display module as input to a calculation unit and direct data input for the NIHSS rapid limb score for stroke; the screen interaction module is used to display an interactive interface to the user and manage and store corresponding data.

8. The portable stroke auxiliary assessment method based on RGB-D camera according to claim 7 is characterized in that: The computing unit includes a microcomputer and a built-in processing algorithm; the built-in processing algorithm includes a time series three-dimensional human body posture estimation algorithm, an algorithm for extracting features from time series three-dimensional human body postures, and an algorithm for preprocessing time series features; The time-series 3D human posture estimation algorithm includes a deep neural network that extracts key points of a person's skeleton from an RGB or RGB-D image; connects and processes the patient's skeleton for each frame of the image recorded in the video stream; The algorithm for extracting features from the time-series three-dimensional human posture includes extracting explicit features from the three-dimensional skeleton; the explicit features include the joint angles of the upper and lower limbs; The algorithm for preprocessing time series features includes preprocessing the feature sequence; the preprocessing includes noise reduction and difference resampling.

9. The portable stroke auxiliary assessment method based on RGB-D camera according to claim 8, characterized in that: The computing unit also includes an artificial intelligence evaluation module; the artificial intelligence evaluation module includes a deep neural network, which takes the preprocessed time series features as input and outputs a corresponding score.

10. The portable stroke auxiliary assessment method based on RGB-D camera according to claim 9, characterized in that: The data output unit comprises: The data integration module is used to integrate the preprocessed time-series motion features with the scoring results output by the artificial intelligence evaluation module; integrate the physical examination data and corresponding evaluation results of the same patient, draw a medical record curve, and output the time-series features as reference information; The real-time display module is used to display the physical examination video recorded by the computer vision input module and the timing characteristics of the calculation unit output in real time, and to display the evaluation score and related basis after the evaluation is completed.

Images