Method for determining coma degree and medium

Abstract

The invention discloses a coma degree determination method and a medium. The method comprises the following steps: acquiring historical health data of a to-be-diagnosed patient and an intake state of the to-be-diagnosed patient for at least one specified substance; based on the historical health data and the intake state, determining a pre-diagnosis strategy for the to-be-diagnosed patient; processing collected data for the patient to be diagnosed based on the pre-diagnosis strategy to obtain a processing result; wherein the collected data is at least associated with body parts included in the pre-diagnosis strategy; and determining the coma degree of the to-be-diagnosed patient based on the processing result. Through the method, automatic, self-adaptive and intelligent determination of the coma degree of the to-be-diagnosed patient is realized, so that the determination efficiency of the coma degree can be improved, a first-aid scene with a large number of to-be-diagnosed patients can be met, and the requirement of efficient pre-hospital first-aid pre-diagnosis of the to-be-diagnosed patient can be met.

Description

Technical Field

[0001] The present application relates to the field of smart medical technology, and in particular to a method and medium for determining the degree of coma. Background Art

[0002] In practical applications, the patient's level of consciousness is usually determined by the Glasgow Coma Scale Score (GCS), the Glasgow Leige Scale (GLS) and other evaluation systems, and whether the patient is in a coma is determined based on the level of consciousness. However, the above schemes take a long time and are inefficient when there are a large number of patients; moreover, the above schemes cannot be applied to pre-hospital emergency scenarios. Summary of the invention

[0003] Based on the above problems, the embodiments of the present application provide a method and medium for determining the degree of coma.

[0004] The technical solution provided by the embodiment of the present application is as follows:

[0005] The present application embodiment provides a method for determining the degree of coma, the method comprising:

[0006] Acquiring historical health data of a patient to be diagnosed and the patient's intake status of at least one designated substance;

[0007] Determining a pre-diagnosis strategy for the patient to be diagnosed based on the historical health data and the intake status;

[0008] Processing the collected data for the patient to be diagnosed based on the pre-diagnosis strategy to obtain a processing result; wherein the collected data is at least associated with a body part included in the pre-diagnosis strategy;

[0009] The coma degree of the patient to be diagnosed is determined based on the processing result.

[0010] An embodiment of the present application further provides a computer-readable storage medium, wherein a computer program is stored in the storage medium; when the computer program is executed by a processor of an electronic device, the method for determining the degree of coma as described above can be implemented.

[0011] The method for determining the degree of coma provided in the embodiment of the present application can determine the health level of the patient to be diagnosed in a historical period through the historical health data of the patient to be diagnosed; and, by obtaining the intake status of at least one substance of the patient to be diagnosed, can determine the degree of negative impact of the intake status of at least one substance on the current patient state and historical health level of the patient to be diagnosed; at the same time, based on the historical health data and the intake status, determine the pre-diagnosis strategy for the patient to be diagnosed, so that the pre-diagnosis strategy can take into account the mutual influence between the historical health level of the patient to be diagnosed and the intake status of at least one specified substance, thereby improving the pertinence and accuracy of the pre-diagnosis strategy; on this basis, the collected data is at least consistent with the body parts included in the pre-diagnosis strategy On the other hand, the collected data for the patient to be diagnosed is processed based on the pre-diagnosis strategy to obtain the processing result, which can not only improve the pertinence and efficiency of the processing of the collected data, but also improve the consistency between the processing result and the patient status of the patient to be diagnosed; on the basis of the above process, the coma degree of the patient to be diagnosed is determined based on the processing result, which can not only improve the accuracy of the coma degree, but also realize the automatic, adaptive and intelligent determination of the coma degree of the patient to be diagnosed, thereby not only improving the efficiency of determining the coma degree, but also being able to cope with the emergency scenarios with a large number of patients to be diagnosed, and also meeting the needs of efficient pre-hospital emergency pre-diagnosis for the patients to be diagnosed. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 A schematic diagram of a flow chart of a method for determining the degree of coma provided in an embodiment of the present application;

[0013] Figure 2 A schematic diagram of a modularized process for determining a prognostic strategy associated with age information provided in an embodiment of the present application;

[0014] Figure 3A A schematic diagram of the structure of eye video data segmentation provided in an embodiment of the present application;

[0015] Figure 3B A schematic diagram of the structure of recognizing eye movements provided in an embodiment of the present application;

[0016] Figure 3C A schematic diagram of the eye state recognition process provided in an embodiment of the present application;

[0017] Figure 3D A schematic diagram of a process for obtaining an eye recognition model provided in an embodiment of the present application;

[0018] Figure 3E A schematic diagram of the process of eye-opening ability detection provided in an embodiment of the present application;

[0019] Figure 3FAnother schematic diagram of a process for detecting the eye-opening ability provided in an embodiment of the present application;

[0020] Figure 3G A schematic diagram of the eye marking process provided in an embodiment of the present application;

[0021] Figure 4A A schematic diagram of the flow of language barrier detection and five sense organs detection provided in the embodiment of the present application;

[0022] Figure 4B A schematic diagram of a speech detection process for adults provided in an embodiment of the present application;

[0023] Figure 4C A schematic diagram of a speech detection process for non-adults provided in an embodiment of the present application;

[0024] Figure 5A A schematic diagram of a process for collecting hand motion data provided in an embodiment of the present application;

[0025] Figure 5B A schematic diagram of a process for marking and scoring based on action scoring rules provided in an embodiment of the present application;

[0026] Figure 5C A schematic diagram of the structure of a data acquisition device provided in an embodiment of the present application;

[0027] Figure 5D A schematic diagram of a process for obtaining a depth image provided in an embodiment of the present application;

[0028] Figure 5E A schematic diagram of the structure of a control data acquisition device provided in an embodiment of the present application;

[0029] Fig. 5F A structural diagram of the data recognition capability provided by the data acquisition device provided in the embodiment of the present application;

[0030] Figure 5G A schematic diagram of a depth image provided in an embodiment of the present application;

[0031] Figure 5H A schematic diagram of the structure of depth image pixel data provided in an embodiment of the present application;

[0032] Fig.5I A schematic diagram showing the effect of mapping a skeleton image to a background binary image provided in an embodiment of the present application;

[0033] Figure 5J A schematic diagram of the structure of the joint points of the skeleton sample data and the hand sample data provided in the embodiment of the present application;

[0034] Figure 5K This is a schematic diagram of the statistical results of the node distance feature set;

[0035] Figure 5L A schematic diagram of the action recognition process provided by an embodiment of the present application;

[0036] Fig. 6A A schematic diagram of the structure of a device for determining the degree of coma provided in an embodiment of the present application;

[0037] Figure 6B It is a structural schematic diagram of a monitoring device provided in the related art;

[0038] Figure 6C A schematic diagram of the development status of the auxiliary diagnosis robot provided in an embodiment of the present application. DETAILED DESCRIPTION

[0039] The technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application.

[0040] It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0041] In the clinical diagnosis process, the patient's level of consciousness is usually actively judged through GCS, GLS or other assessment systems. In addition, before judging the patient's level of consciousness, it may take 5 minutes to detect the patient's injuries, and the accuracy rate may be around 60%.

[0042] Taking the conventional consciousness assessment to determine whether the patient is in a state of drowsiness, lethargy, or coma as an example, Table 1 shows the statistical results of the time required for consciousness assessment in three different hospitals. As shown in Table 1, the consciousness assessment time in the three different hospitals all exceeded 1 minute. Therefore, the method of assessing the patient's consciousness level in the related art is inefficient.

[0043] Table 1

[0044] Hospital Name First Hospital Second Hospital The Third Hospital Mental status assessment time 2 minutes 1.5 minutes 1.8 minutes

[0045] Related technologies also provide methods for monitoring the patient's state of consciousness, auditory sensory system, movement, wakefulness, muscle activity, eye movement, eye opening, tension and anxiety level, vital signs parameters, and auditory-visual recall using sleep analysis, electroencephalography (EEG) bispectral analysis, bi-coherence, and auditory evoked potential (AEP) in an integrated manner. However, the above methods cannot be applied to emergency scenarios before patients are admitted to the hospital, and are heavily dependent on professional monitoring equipment and professional medical personnel.

[0046] Based on the above problems, the embodiments of the present application provide a method and medium for determining the degree of coma.

[0047] The method for determining the degree of coma provided in the embodiment of the present application can be implemented by a processor of an electronic device. The processor may include at least one of an application specific integrated circuit (ASIC), a digital signal processor (DSP), a digital signal processing device (DSPD), a programmable logic device (PLD), a field programmable gate array (FPGA), a central processing unit (CPU), a controller, a microcontroller (MCU), and a microprocessor.

[0048] Exemplarily, the electronic device may be a physical device or a virtual machine device.

[0049] Exemplarily, the electronic device may be a computer device.

[0050] Exemplarily, the electronic device may include a robotic device, wherein the robot may be an auxiliary diagnosis robot.

[0051] Figure 1 A schematic diagram of a method for determining the degree of coma provided in an embodiment of the present application is shown in FIG. Figure 1 As shown, the process may include the following steps:

[0052] Step 101: Obtain historical health data of a patient to be diagnosed and the patient's intake status of at least one designated substance.

[0053] In one embodiment, the patients awaiting diagnosis may include patients who have not seen a doctor, have not been admitted to a doctor, and have not been hospitalized.

[0054] In one embodiment, the historical health data may include at least one of medical treatment data, medication data, and physical therapy data of the patient to be diagnosed within a historical period.

[0055] In one embodiment, historical health data may be obtained by any of the following methods:

[0056] The data output by the accompanying persons of the patients to be diagnosed are integrated to obtain historical health data.

[0057] Identify the identity information of the patient to be diagnosed, and obtain the historical medical history, historical medication record, historical surgery record and other data of the patient to be diagnosed based on the identity information, and then determine at least one of the above data as historical health data; illustratively, the identity information can be obtained by identifying the fingerprint, voiceprint and identity identification document of the patient to be diagnosed; wherein the identity identification document may include an identity card and a driver's license, etc.

[0058] In one embodiment, the designated substance may include a substance that causes physical or mental stimulation to the patient to be diagnosed to a degree greater than a preset degree; illustratively, the designated substance may include at least one of alcohol, sedatives, and stimulants.

[0059] In one embodiment, the designated substance may include a liquid and / or an object.

[0060] In one embodiment, the intake status may include at least one of whether a specified substance is ingested, the intake amount, the intake time, and the intake method.

[0061] In one implementation, the ingestion status may be obtained by any of the following methods:

[0062] The intake status is determined by the intake data provided by the accompanying person of the patient to be diagnosed.

[0063] The robot device is used to perform a designated substance intake test on the patient to be diagnosed, thereby obtaining the intake status; wherein the intake test may include, for example, an alcohol test.

[0064] Step 102: Determine a pre-diagnosis strategy for the patient to be diagnosed based on historical health data and intake status.

[0065] In one embodiment, the pre-diagnosis strategy may include test items, test procedures, test conditions, and test data processing methods included in the automated preliminary test of the patient to be diagnosed.

[0066] In one embodiment, the prognostic strategy can be determined by:

[0067] The degree of negative impact of at least one designated substance on the physical state and / or mental state represented by the historical health data is determined according to the intake status, and a prognostic strategy is determined according to the degree of negative impact.

[0068] Step 103: Process the collected data for the patient to be diagnosed based on the pre-diagnosis strategy to obtain a processing result.

[0069] The collected data is at least associated with a body part included in the pre-diagnosis strategy.

[0070] In one embodiment, the body parts included in the pre-diagnosis strategy may include at least one body part targeted by the detection items included in the pre-diagnosis strategy.

[0071] In one embodiment, the collected data may include at least one of video data, image data, audio data, body temperature data, pulse data, blood pressure data, and blood oxygen data of the above-mentioned body parts.

[0072] In one embodiment, the above-mentioned collected data can be obtained through a data acquisition device; illustratively, the data acquisition device may include an image acquisition device, an audio acquisition device, and a sensor device integrated in the robot device; wherein the sensor device can realize the collection of data such as blood pressure, heart rate, blood oxygen, and pulse.

[0073] In one implementation, the processing result may be obtained by any of the following methods:

[0074] The collected data is analyzed based on the data analysis process included in the pre-diagnosis strategy, and the obtained analysis result is determined as the processing result; illustratively, the processing result at this time may include whether the vital signs of the person to be diagnosed in at least one dimension are greater than the health indicator index.

[0075] Based on the data analysis method included in the pre-diagnosis strategy, features are extracted from various types of data in the collected data, and the results of the feature extraction are integrated to obtain processing results; illustratively, the processing results at this time can represent a comprehensive health level assessment for the person to be diagnosed.

[0076] Step 104: Determine the coma degree of the patient to be diagnosed based on the processing result.

[0077] In one embodiment, the coma level may indicate whether the patient to be diagnosed is in a coma.

[0078] In one embodiment, the coma degree may characterize the coma level of the patient to be diagnosed when the patient is already in a coma state; illustratively, the coma degree may include deep coma and shallow coma.

[0079] In one embodiment, the degree of coma can be determined by:

[0080] The treatment results are classified according to medical diagnosis standards and the classification results are determined as the degree of coma.

[0081] The processing results are comprehensively analyzed through an artificial intelligence model or a neural network model, and the results of the comprehensive analysis are determined as the degree of coma.

[0082] From the above, it can be seen that the method for determining the degree of coma provided in the embodiment of the present application can determine the health level of the patient to be diagnosed in a historical period through the historical health data of the patient to be diagnosed; and, by obtaining the intake status of at least one substance of the patient to be diagnosed, it can determine the degree of negative impact of the intake status of at least one substance on the current patient state and historical health level of the patient to be diagnosed; at the same time, based on the historical health data and the intake status, determine the pre-diagnosis strategy for the patient to be diagnosed, so that the pre-diagnosis strategy can take into account the mutual influence between the historical health level of the patient to be diagnosed and the intake status of at least one specified substance, thereby improving the pertinence and accuracy of the pre-diagnosis strategy; on this basis, the collected data is at least consistent with the pre-diagnosis strategy included On the other hand, the collected data for the patient to be diagnosed is associated with the body parts, thereby improving the effectiveness of the collected data; on the other hand, the collected data for the patient to be diagnosed is processed based on the pre-diagnosis strategy to obtain the processing result, which can not only improve the pertinence and efficiency of the processing of the collected data, but also improve the consistency between the processing result and the patient status of the patient to be diagnosed; on the basis of the above process, the coma degree of the patient to be diagnosed is determined based on the processing result, which can not only improve the accuracy of the coma degree, but also realize the automatic, adaptive and intelligent determination of the coma degree of the patient to be diagnosed, thereby not only improving the efficiency of determining the coma degree, but also being able to cope with emergency scenarios with a large number of patients to be diagnosed, and also meeting the needs of efficient pre-hospital emergency pre-diagnosis of patients to be diagnosed.

[0083] Based on the above embodiments, in the method for determining the degree of coma provided in the embodiments of the present application, the pre-diagnosis strategy for the patient to be diagnosed is determined based on the historical health data and the intake status, which can be achieved in the following ways:

[0084] Determine the age information of the patient to be diagnosed; determine the target weight associated with the target data; and determine the pre-diagnosis strategy based on the target weight and the target data.

[0085] The target data includes at least one of the age data, historical health data and intake status of the patient to be diagnosed.

[0086] In one implementation, the age information may include the actual age of the patient to be diagnosed, and may also include the age stage of the patient to be diagnosed, such as adolescence, middle age, and old age.

[0087] In one implementation, the age information may be determined by any of the following methods:

[0088] The age information is determined based on the recognition result of the identification certificate of the patient to be diagnosed.

[0089] The age information is determined based on the age data provided by the accompanying person of the patient to be diagnosed.

[0090] Age recognition is performed based on the patient's appearance and / or clothing features to predict age information.

[0091] In one implementation, the target weight may be greater than zero.

[0092] In one embodiment, the target weight may be determined by any of the following methods:

[0093] The target weight is determined according to the differentiated needs of pre-diagnosis; wherein the differentiated needs may include parameters, processes or conditions that can reflect the degree of differentiation between different patients to be diagnosed. For example, if the differentiated needs include an age differentiation parameter, then when determining the coma state, a weight of 0.5 can be set for the age information, and a weight less than 0.5 can be set for the data in the target data excluding the age information, wherein the target weight may include the age weight as well as other weights.

[0094] The target data is analyzed through a neural network to determine the target weight.

[0095] The target weight is determined by a preset association relationship and target data; wherein the association relationship may include a relationship of correlation between a set of target data and a set of target weights.

[0096] In one embodiment, the prognostic strategy may be determined by:

[0097] The weight parameters of the pre-trained prognostic template or model are adjusted to the target weights, and the prognostic methods and processes contained in the prognostic template or model after the weight parameters are adjusted are determined as the prognostic strategy; illustratively, the weight parameters corresponding to different value ranges of historical health data can be adjusted to 0.8, 1, and 1.2, etc.; in this way, when the target data is historical health data, the target weight can be determined from the above weight parameters according to the value range of the actual historical health data of the patient to be diagnosed, and the weight parameters of the prognostic template or model are adjusted to the target weight.

[0098] Figure 2 A schematic diagram of a modularized process for determining a pre-diagnosis strategy associated with age information provided in an embodiment of the present application. Figure 2 In the present invention, medical staff 201, auxiliary diagnosis robot 202 and patient to be diagnosed 203 cooperate with each other to determine the pre-diagnosis strategy; wherein, the auxiliary diagnosis robot 202 can be a robot device or a coma degree determination device.

[0099] like Figure 2As shown, the medical staff 201 performs a power-on operation on the auxiliary diagnosis robot 202. After responding to the power-on operation, the auxiliary diagnosis robot 202 can perform initialization operations including automatic stimulation module reset, camera gimbal reset, and battery self-test; exemplarily, the auxiliary diagnosis robot 202 can respond to the handle control operation of the medical staff 201, so as to collect a preview image containing the patient to be diagnosed 203, and determine the position of the patient to be diagnosed 203 based on the pixel features of the above preview image, and can also calibrate the patient to be diagnosed 203 in the preview image, and can locate the patient to be diagnosed 203 by capturing the eyes of the medical staff 201.

[0100] Exemplarily, the auxiliary diagnosis robot 202 can perform an alcohol test on the patient 203 through the alcohol detection module 204. If the result of the alcohol test is that the alcohol content exceeds the limit, the test is terminated and a prompt message indicating that the alcohol content exceeds the limit is output to the medical staff.

[0101] Exemplarily, if the alcohol test result shows that the alcohol content is within the limit, the auxiliary diagnosis robot 202 can obtain the historical health data and age information of the patient to be diagnosed 203 through the identity data determination module 205; wherein, the information of the patient to be diagnosed can be obtained through face recognition library, identity card and fingerprint recognition, thereby reading the age information, medical history and historical physical examination information of the patient to be diagnosed 203.

[0102] Exemplarily, the auxiliary diagnosis robot 202 can analyze the medical records and historical physical examination information through the abnormal medical history detection unit 206. If the above analysis results indicate that the patient to be diagnosed has a history of epilepsy, or a record of reuse of sedatives within 24 hours, or a 24-hour alcohol record, the detection is terminated, otherwise a speech disorder history detection is performed.

[0103] Exemplarily, if the abnormal medical history detection unit 206 detects that the patient 203 has a history of speech disorders based on the medical records and historical physical examination information, the marked language response is not implemented. If it is determined that the language expression of the patient 203 is normal, it is recorded normally.

[0104] For example, if the age information fails to be read through the previous step, the age information can be predicted through face prediction, and then the pre-diagnosis strategy corresponding to the age is loaded, and the formal pre-diagnosis process is entered.

[0105] Exemplarily, the pre-diagnosis strategies corresponding to alcohol, sedatives, etc. may also be loaded, and the formal pre-diagnosis process may be entered.

[0106] Through the above process, the auxiliary diagnosis robot can automatically and intelligently perform alcohol testing, obtain the age information and medical records of the patient to be diagnosed, and automatically load the prognostic strategy for the corresponding age, thereby simplifying the process of determining the prognostic strategy and improving the efficiency of determining the prognostic strategy.

[0107] As can be seen from the above, in the method for determining the degree of coma provided in the embodiment of the present application, the age information of the patient to be diagnosed is determined, and the target weight associated with at least one of the age information, historical health data, and intake status is determined, and the pre-diagnosis strategy is determined based on the target weight and the target data. In this way, by adjusting the value of the target weight and the data category associated with it, the diversification of the pre-diagnosis strategy can be improved, thereby meeting the pre-diagnosis needs of a variety of patients to be diagnosed; and, because the pre-diagnosis strategy is directly associated with the target weight and the target data, the accuracy of the pre-diagnosis strategy can be improved.

[0108] Based on the above embodiments, in the method for determining the degree of coma provided in the embodiments of the present application, the pre-diagnosis strategy for the patient to be diagnosed is determined based on the historical health data and the intake status, which can also be achieved in the following ways:

[0109] The historical health data and intake status are analyzed to determine the health status of the patient to be diagnosed; based on the health status, a pre-diagnosis strategy is determined from multiple strategies included in the strategy set.

[0110] Among them, the strategies in the strategy set are associated with the health level.

[0111] In one embodiment, the health level may represent whether the patient to be diagnosed is healthy, or the health level of the patient to be diagnosed.

[0112] In one embodiment, health status can be determined by:

[0113] An initial health level is determined based on historical health data, and the initial health level is corrected based on the degree of influence of the intake status on the initial health level, thereby obtaining a health level.

[0114] In one embodiment, the multiple strategies included in the strategy set may include a process and method for pre-diagnosis of at least one of multiple patient types, multiple health levels, and multiple specified substance intakes.

[0115] In one embodiment, multiple strategies included in the strategy set can be associated with multiple health levels. In this way, based on the degree of match between the health level of the patient to be diagnosed and the health level in the strategy set, a pre-diagnosis strategy corresponding to the health level of the patient to be diagnosed can be determined from the strategy set.

[0116] In one embodiment, the strategies in the strategy set may include prognostic templates or models; illustratively, multiple different prognostic templates or models may be pre-trained, and different prognostic templates or models are respectively associated with different health levels, so that after determining the health level of the patient to be diagnosed, the prognostic template or model corresponding to the patient to be diagnosed can be determined from the multiple prognostic templates or models.

[0117] From the above, it can be seen that in the method for determining the degree of coma provided in the embodiment of the present application, after analyzing the historical health data and the intake status to determine the health degree of the patient to be diagnosed, the pre-diagnosis strategy is determined from the multiple strategies included in the strategy set based on the health degree. In this way, since the health degree is determined based on the historical health data and the intake status, the health degree can accurately reflect the actual health state of the patient to be diagnosed; and the pre-diagnosis strategy is determined based on the health degree, so that the pre-diagnosis strategy can accurately correspond to the actual health state of the patient to be diagnosed, then, through the pre-diagnosis strategy, targeted, comprehensive and accurate pre-diagnosis detection of the patient to be diagnosed can be achieved.

[0118] Based on the foregoing embodiments, in the method for determining the degree of coma provided in the embodiments of the present application, the collected data includes eye video data of the patient to be diagnosed; the processing results include eye movement data; and the pre-diagnosis strategy includes an eye pre-diagnosis strategy.

[0119] In one embodiment, the eye video data may include video data at least including the eye of the patient to be diagnosed.

[0120] In one implementation, the eye movement data may indicate the switching movement between closing and opening the eyes of the patient to be diagnosed, and may also indicate that the eyes of the patient to be diagnosed are closed and cannot be opened.

[0121] In one embodiment, the eye pre-diagnosis strategy may include a process of how to collect eye video data, how to extract features from the eye video data, and how to identify eye features and eye movements.

[0122] Accordingly, the collected data for the patient to be diagnosed is processed based on the pre-diagnosis strategy to obtain the processing result, which can be achieved in the following ways:

[0123] If the eye video data indicates that the eye of the patient to be diagnosed does not contain an open wound, the eye video data is processed based on the eye pre-diagnosis strategy to obtain eye movement data; if the eye video data indicates that the eye of the patient to be diagnosed contains an open wound, the first information is output to prompt medical personnel to treat the open wound.

[0124] The first information includes at least the location information of the patient to be diagnosed and the information that the eye contains an open wound.

[0125] In one embodiment, the eye movement data may be obtained by:

[0126] The eye video data is segmented to obtain at least one frame of eye image, and then based on the recognition method included in the eye pre-diagnosis strategy, the at least one frame of eye image is recognized, and the recognition result is determined as eye movement data.

[0127] In one embodiment, after the medical staff finishes treating the open wound, the eye video data can be re-collected and processed based on the eye pre-diagnosis strategy to obtain eye motion data.

[0128] In one implementation, the first information may be output in the form of voice broadcast, display screen output, and light flashing.

[0129] In one embodiment, the location information may include the location of the patient to be diagnosed relative to at least one reference object; wherein the reference object may include a building, a floor, a room, an entrance, etc., and the reference object may also include medical facilities of a hospital.

[0130] From the above, it can be seen that in the method for determining the degree of coma provided in the embodiment of the present application, if the eye video data of the patient to be diagnosed indicates that the eye of the patient to be diagnosed does not contain an open wound, the eye video data is processed based on the eye pre-diagnosis strategy to obtain eye movement data. In this way, not only the effectiveness of the eye video data and the eye movement data can be improved, but also strict control over the operation of processing the eye video data can be achieved; and, if the eye video data indicates that the eye of the patient to be diagnosed contains an open wound, the first information is output to prompt the medical staff to handle the open wound, thereby weakening the negative impact of the open wound on the eye of the patient to be diagnosed; on the other hand, since the first information contains the position information of the patient to be diagnosed and the information that the eye contains an open wound, the output of the first information can achieve accurate prompts for the medical staff, thereby improving the efficiency of handling the open wound of the eye of the patient to be diagnosed.

[0131] Based on the foregoing embodiments, in the method for determining the degree of coma provided in the embodiments of the present application, the eye pre-diagnosis strategy includes an eye recognition model.

[0132] In one embodiment, the eye recognition model may include a neural network capable of realizing eye movement recognition function; illustratively, the eye recognition model may include at least two feature extraction modules, each for extracting different eye features, and by integrating the eye features outputted by at least two feature extraction modules, the eye movement data of the patient to be diagnosed can be determined.

[0133] In one embodiment, the eye recognition model may include a deep convolution neural network expression recognition model (Deep Convolution Neural Network Empression, DCNNE).

[0134] Accordingly, processing the eye video data based on the eye pre-diagnosis strategy to obtain the eye movement data can be achieved by the following steps:

[0135] Step A1: segment the eye video data to obtain at least one frame of eye image.

[0136] Figure 3A A schematic diagram of the structure of eye video data segmentation provided in an embodiment of the present application.

[0137] like Figure 3A As shown, the facial key point detection module 301 can detect the image frames in the eye video data, and determine the image containing the facial key points as the eye image to be selected; the eye image to be selected can be processed by the eye regression unit 303 in the eye positioning module 302 to determine the center position of the eye; at the same time, the eye image to be selected can be processed by the eye side length regression unit 304 to obtain the side length of the eye area; the eye center position and the side length of the eye area are integrated by the eye image output module 305 to obtain at least one frame of eye image.

[0138] Exemplarily, the facial key point detection module 301 may be a face detector based on single shot detection (Single Shot Detector, SSD) that is pre-trained based on the Wider face detection dataset.

[0139] Exemplarily, after detecting the face area, the eye regression unit 303 can mark the positions of the two eyes according to the coordinates of the face area, thereby determining the center position of the eyes; at the same time, the eye side length regression unit 304 uses the leftmost and rightmost coordinates of the eyes to extract the eye contour and adjust its size, thereby obtaining the side length of the eye area. The eye image output module 305 integrates the center position of the eyes and the side length of the eye area to obtain a 32×32 eye image.

[0140] pass Figure 3A The structure shown can achieve simultaneous determination of the side length of the eye area and the center position of the eye, thereby improving the efficiency of determining the eye image; and, since the eye image is determined based on the side length of the eye area and the center position of the eye, the eye image can fully include the eye features of the patient to be diagnosed.

[0141] Step A2: determining eye edge features based on at least one frame of eye image.

[0142] In one embodiment, the eye edge feature may include an extended trajectory of the eye edge region.

[0143] In one implementation, feature extraction may be performed on at least one frame of eye image to determine eye edge features.

[0144] Step A3: Determine a set of primitive attributes of the eye.

[0145] In one embodiment, the primitive attribute set may include a reflectance attribute and a histogram attribute of the eye.

[0146] Step A4: Process the eye edge features and primitive attribute set through the eye recognition model to obtain eye movement data.

[0147] In one embodiment, the eye recognition model may include a first module and a second module, wherein the first module and the second module may respectively perform feature extraction on eye edge features and a set of primitive attributes to obtain two extraction results, and then integrate and judge the two extraction results to obtain eye movement data.

[0148] In one embodiment, the first module may include a convolution layer, a pooling layer, a global average pooling layer, a fully connected layer, and a classifier; exemplarily, the first module may include 6 convolution layers, 3 maximum pooling layers, a global average pooling layer (GAP) layer, two fully connected layers, and a softmax classifier; compared with the traditional Flatten layer, the number of parameters of the GAP layer in the first module can be reduced, thereby overcoming the model overlap problem; exemplarily, the first module may non-regressively transform the features between the convolution layer and the fully connected layer through the ReLU activation function. Linear transformation; illustratively, after each convolution layer and fully connected layer, batch normalization and dropout layers with a probability of 0.2 can be used; wherein, the first convolution layer can use 32 convolution kernels with a size of 5×5, and the convolution kernel size used by the remaining 5 convolution layers can be 3×3, and the maximum pooling layer can use a 2×2 pooling window with a pooling span of 2 pixels, thereby reducing the spatial dimension of the feature map obtained by the convolution layer to half of the original, and the number of neurons contained in the first fully connected layer and the second fully connected layer can be 128 and 64 respectively.

[0149] In one embodiment, the second module may include a convolution layer, a maximum pooling layer, a GAP layer and a fully connected layer; wherein the second module may include 7 convolution layers, 3 maximum pooling layers, 1 GAP layer and 2 fully connected layers; exemplarily, the input image of the second module may be a 32×32 grayscale image; exemplarily, similar to the first module, the nonlinear conversion between different convolution layers and fully connected layers in the second module may be achieved by a (ReLU) activation function, and batch normalization and dropout layers are used after each convolution and fully connected layer to alleviate the overfitting problem.

[0150] In one embodiment, the first module and the second module can respectively extract different eye appearance features, and input the different eye appearance features into a softmax classifier through two fully connected layers to determine whether the eye action data is eyes open or closed.

[0151] Figure 3B A schematic diagram of the structure of recognizing eye movements provided in an embodiment of the present application, such as Figure 3B As shown, feature recognition is performed on the eye image 306 to determine the edge attribute 307 and the primitive attribute set 308 of the eye; the edge attribute 307 and the primitive attribute set 308 are respectively input into the eye state recognition network 309, and after being processed by the eye state recognition network 309, the eye movement 310 can be obtained.

[0152] The eye state recognition network 309 may be the eye recognition model in the aforementioned embodiment, and the eye movement 310 may be the eye movement data in the aforementioned embodiment.

[0153] Figure 3C The following is a flow chart of eye state recognition provided in the embodiment of the present application. Figure 3C As shown, the process may include the following steps:

[0154] Step 311: Input image.

[0155] Exemplarily, at least one frame of eye image may be input into the facial key point detection module.

[0156] Step 312: facial key point detection.

[0157] Exemplarily, facial key points may be detected by a facial key point detection module to obtain facial key points.

[0158] Step 313: Eye positioning.

[0159] For example, since the facial key points include the estimated positions of the left eye, the right eye, and the nose, there is a deviation between the estimated positions and the actual positions of the eyes, so the deviation can be calibrated by eye positioning.

[0160] Exemplarily, the facial key points may be detected by an eye positioning module to obtain the center position of the eye and the side length of the eye region.

[0161] Step 314: Eye state recognition.

[0162] Exemplarily, an eye state recognition operation may be performed through an eye state recognition network to obtain eye movement data.

[0163] pass FIG. 3B to FIG. 3C The process shown can achieve accurate positioning of the eye area of ​​the patient to be diagnosed and accurate recognition of the eye movements, thereby improving the accuracy of the coma degree of the patient to be diagnosed.

[0164] From the above, it can be seen that in the method for determining the degree of coma provided in the embodiment of the present application, after the eye video data is segmented to obtain at least one frame of eye image, the eye edge features are determined based on the at least one frame of eye image, so that the accuracy of the eye edge can be improved; and after determining the primitive attribute set of the eye, the eye edge features and the primitive attribute set are processed by the eye recognition model, so that the eye movement data obtained through the above process can fully and comprehensively present the eye movement of the patient to be diagnosed, thereby improving the accuracy and completeness of the eye movement data.

[0165] Based on the above embodiments, the method for determining the degree of coma provided in the embodiments of the present application may further include the following steps:

[0166] Obtain eye sample data; and train an eye recognition model in an initial state based on the eye sample data to obtain an eye recognition model.

[0167] The eye sample data is at least associated with a first aid scenario; the eye sample data includes at least an eye image containing stains and a covered eye image; the stains include at least blood stains and dust.

[0168] In one embodiment, the first aid scenario may include emergency rescue scenarios related to natural disasters, and may also include emergency rescue scenarios associated with accidents; wherein natural disasters may include volcanoes, earthquakes, mudslides, and floods, etc., and accidents may include traffic accidents, etc.

[0169] In one embodiment, the stain-containing eye image may include an open eye image accompanied by blood stain and a closed eye image.

[0170] In one embodiment, the stain-containing eye image may include an open eye image accompanied by ground dust and makeup stains, and a closed eye image.

[0171] In one embodiment, the covered eye image may include an image in which at least one eye is blocked or covered, such as an image of an eye blocked or covered by gauze or sunglasses.

[0172] In one implementation, the eye recognition model in the initial state may be trained by a supervised or unsupervised method to obtain the eye recognition model.

[0173] In one embodiment, during the training of the initial state eye recognition model based on the eye sample data, the intermediate data can be processed by batch normalization and dropout layer to speed up the training and alleviate overfitting.

[0174] In one embodiment, the training process of the initial state eye recognition model may include pre-training and fine-tuning training; wherein, the pre-training may include training the initial state eye recognition model based on the FER dataset to obtain an intermediate state eye recognition model; the fine-tuning training may further adjust the parameters of the intermediate state eye recognition model based on the eye sample data; exemplarily, during the pre-training process, the number of classifiers included in the output layer of the initial state eye recognition model may be greater than the number of classifiers included in the output layer during the fine-tuning training process, for example, during the pre-training process, the output layer may include a 7-way softmax classifier, and during the fine-tuning training process, the above classifier is changed to a 2-way softmax.

[0175] Figure 3D A schematic diagram of a process for obtaining an eye recognition model provided in an embodiment of the present application is shown in FIG. Figure 3D As shown, the process may include the following steps:

[0176] Step 315: construct an eye recognition model in an initial state.

[0177] Exemplarily, the eye recognition model in the initial state may include a DCNNE in the initial state.

[0178] Step 316: Obtain an eye picture of the first aid scene and annotate the eye movements.

[0179] Exemplarily, the eye movements represented by the above eye pictures can be annotated by using a deep learning annotation tool.

[0180] Step 317: Generate eye sample data.

[0181] Exemplarily, the above-mentioned eye pictures with eye movements annotated may be integrated to obtain eye sample data.

[0182] Step 318: Train the eye recognition model in the initial state based on the eye sample data to obtain an eye recognition model.

[0183] Exemplarily, the eye recognition model can be obtained by the method provided in the aforementioned embodiment.

[0184] From the above, it can be seen that in the method for determining the degree of coma provided in the embodiment of the present application, the number of eye samples obtained is at least associated with the first aid scene, so that the eye sample data can fully carry the data characteristics of the first aid scene; and the eye sample data at least includes an eye image containing stains and a covered eye image, and the stains at least include blood stains and dust, so that the eye sample data can carry diverse and refined features of the eyes in the first aid scene; on this basis, the eye recognition model in the initial state is trained based on the eye sample data, so that the eye recognition model obtained through the above process can realize accurate recognition of diverse eye images in the first aid scene.

[0185] Based on the foregoing embodiments, in the method for determining the degree of coma provided in the embodiments of the present application, the eye pre-diagnosis strategy at least includes stimulation parameters for the patient to be diagnosed.

[0186] In one embodiment, the stimulation parameters may include at least one of the type of stimulation performed on the patient, the stimulation site, the stimulation intensity, and the duration of the stimulation.

[0187] Accordingly, the method for determining the degree of coma provided in the embodiment of the present application may further include the following steps:

[0188] In the process of outputting the audio data included in the stimulation parameters, eye video data is collected, and / or, in the process of performing supraorbital nerve compression based on the eye compression parameters included in the stimulation parameters, expression video data and eye video data of the patient to be diagnosed are collected.

[0189] In one embodiment, when the patient's eyes are not opened naturally, stimulation operations including sound stimulation and physical pressure stimulation may be performed on the patient based on stimulation parameters to prompt the patient to open his eyes.

[0190] In one implementation, the audio data included in the stimulation parameters may be used to prompt the patient to open both eyes. For example, the audio data may be “Hello, please open your eyes.”

[0191] In one implementation, the output volume of the audio data may be greater than or equal to a first threshold value to increase the probability that the patient to be diagnosed can hear the audio data.

[0192] In one embodiment, if the eye video data collected during the process of outputting the audio data included in the stimulation parameters indicates that the patient to be diagnosed did not open his eyes during the output of the above audio data, the supraorbital nerve pressing operation can be performed based on the eye pressing parameters included in the stimulation parameters.

[0193] In one embodiment, during the process of performing supraorbital nerve compression, the facial expression of the patient to be diagnosed can be synchronously collected to obtain facial expression video data; illustratively, the facial expression video data can also be analyzed to obtain the conscious reaction state of the patient to the supraorbital nerve compression.

[0194] From the above, it can be seen that in the method for determining the degree of coma provided in the embodiment of the present application, the eye pre-diagnosis strategy at least includes stimulation parameters for the patient to be diagnosed, and the stimulation parameters include audio data and supraorbital nerve pressing operations. In this way, based on the audio data and supraorbital nerve pressing operations included in the stimulation parameters, diversified stimulation of the patient to be diagnosed to perform eye movements is achieved, which can increase the probability of the patient to be diagnosed performing eye movements, thereby increasing the number and types of eye movement features carried in the eye video data.

[0195] Based on the above embodiments, in the method for determining the degree of coma provided in the embodiments of the present application, the eye video data is processed based on the eye pre-diagnosis strategy to obtain the eye movement data, which can also be implemented in the following ways:

[0196] If the eye video data indicates that the eye is covered by an occluding object, identify the occluding object, remove the occluding object from the eye video data, and obtain a first removed image; process the first removed image based on the eye pre-diagnosis strategy to obtain eye movement data.

[0197] Exemplarily, if the eye video data indicates that the eye is not covered by an occluding object, the operation of identifying the occluding object may not be performed.

[0198] In one implementation, the occluding object may be removed by using an image digital recognition algorithm, thereby obtaining a first removed image; illustratively, the number of images in the first removed image may be at least one frame.

[0199] In one implementation, if the area of ​​the eyes blocked in the first removed image is greater than or equal to the second threshold, a prompt message indicating that the eyes are covered by the blocked object may be output to the medical staff so that the medical staff can remove the blocked object in time.

[0200] In one implementation, after the medical staff removes the obstructing object, they can continue to collect eye video data, and process the eye video data using the method provided in the aforementioned embodiment to obtain eye movement data.

[0201] Figure 3E A schematic diagram of the flow of eye-opening ability detection provided in the embodiment of the present application is shown in FIG. Figure 3EAs shown, after receiving the eye movement data confirmation instruction issued by the medical staff 201, the auxiliary diagnosis robot 202 can determine the position of the patient to be diagnosed and start the data collection and lighting compensation functions, where the data collection includes video collection and audio collection. If it is detected that the light intensity of the environment where the patient to be diagnosed is located is less than the intensity threshold, the ambient light brightness is compensated through the lighting compensation function.

[0202] Exemplarily, the auxiliary diagnosis robot 202 can also plan a moving route, and maintain a low-decibel working state during the movement to the patient 203 to be diagnosed, so as to reduce the impact on the patient 203 to be diagnosed.

[0203] Exemplarily, the auxiliary diagnosis robot 202 can call the eye pre-examination module 319 and continuously identify the eye video data collected by it, so as to determine the eye movement of the patient 203 to be diagnosed during the movement of the auxiliary diagnosis robot 202.

[0204] Exemplarily, when the eye pre-examination module 319 detects that the patient 203 opens his eyes naturally, it records “eyes opened automatically”, at which time the detection ends and is marked and scored.

[0205] Exemplarily, if the eye pre-examination module 319 fails to detect that the patient 203 opens his eyes naturally within 10 seconds, it outputs audio data, where the audio data may be "Hello, please open your eyes", and collects eye video data. If it is detected that the patient 203 opens his eyes, it records "eye opening is detected", the detection ends, and it is marked and scored.

[0206] For example, if the patient is not detected to open his eyes within 10 seconds after the audio data is output, the arm will perform air compression on the supraorbital nerve, and collect eye video data and facial expression video data at the same time. After the above operations are completed, the arm will be reset. In the above process, if the eye-opening action is recognized, it will record "pain to open eyes" and the detection is completed at this time, and the score will be marked. If the eye-opening action is not recognized, it will record "cannot open eyes", and the marking is completed at this time. The score is marked. At this point, the eye-opening ability test is completed.

[0207] Exemplarily, the above-mentioned marking scoring can be performed based on the eye scoring rule:

[0208] 4 points: spontaneous eye opening (spontaneous); 3 points: eyes will open when called (to speech); 2 points: eyes will open when stimulated or in pain (to pain); 1 point: no response to stimulation (none); C point: if the eyes cannot be opened due to eye swelling, fracture, etc., it should be indicated as "C" (closed).

[0209] Among them, in the process of determining the type of eye opening due to stimulation or pain, the patient can be patted or shaken first. If the patient does not respond, strong stimulation can be output to the patient. For example, the tip of a pen can be used to stimulate the outside of the second or third finger of the patient, and the stimulation can be increased to the maximum within 10 seconds. If the patient opens his eyes under the strong stimulation, 2 points can be scored. If he only frowns, closes his eyes, or has a painful expression, he cannot be scored 2 points.

[0210] Figure 3F Another schematic diagram of the process of eye-opening ability detection provided in the embodiment of the present application. Figure 3F In the present invention, after the eye pre-inspection module 319 detects at least one eye center and eye area, if an obvious mass or open fracture is detected in the eye, the trauma treatment unit is activated to perform trauma pre-grading based on the eye video data, and the medical staff 201 is notified. The notification content may include the level of trauma pre-grading, the location of the patient to be diagnosed, and the specific location of the open fracture, etc., and then the standby mode is entered.

[0211] Exemplarily, after the medical staff 201 finishes treating the open fracture, if at most one eye is detected to have visible trauma through the eye video data, the eye numbers for priority capture are marked; if both eye areas are injured, the detection is completed and marked for scoring.

[0212] Exemplarily, if both eyes cannot be located, the eye covering object recognition is performed, and it is determined whether the covering object can be removed by an image algorithm; if the covering object can be removed by an image algorithm, the covering object is removed; if the covering object cannot be removed, the medical staff 201 is reminded to remove the covering object in time and enter the standby mode.

[0213] For example, after the medical staff 201 removes the above-mentioned covering object, the following steps may be performed: Figure 3E The process shown continues to collect eye image data and complete the eye opening ability test.

[0214] Exemplarily, during the movement of the auxiliary diagnosis robot 202, the eye-opening detection module can be called to analyze the eye-opening state during the movement.

[0215] For example, in FIG. 3E to FIG. 3F In the embodiment, the series of operations and processes performed by the eye pre-examination module may be the eye pre-examination strategy in the aforementioned embodiment.

[0216] pass FIG. 3E to FIG. 3F As shown in the process, the auxiliary diagnosis robot can stably collect eye image data under various conditions, thereby realizing the automated, stable and accurate detection of the patient's eye movement data, i.e., the ability to open eyes.

[0217] As can be seen from the above, in the coma state determination method provided by the embodiment of the present application, if the eye video data indicates that the eye is covered by an occluding object, the occluding object is identified, and the occluding object is removed from the eye video data to obtain a first removed image, and then the first removed image is processed based on the eye pre-diagnosis strategy to obtain eye movement data. In this way, when the eye is covered by an occluding object, stable processing of the eye video data is achieved, thereby improving the stability and effectiveness of the eye movement data.

[0218] Figure 3G Schematic diagram of the eye marking process provided in the embodiment of the present application. Figure 3G As shown, the process may include the following steps:

[0219] Step 320, start.

[0220] Step 321: Collect eye video data in real time.

[0221] Step 322: Video segmentation.

[0222] Exemplarily, the eye video data may be segmented into image frames, and the image frames may be processed to obtain eye images.

[0223] Step 323: Determine whether there is an abnormal state.

[0224] For example, the eye image can be identified to determine whether the patient's eye has abnormal conditions such as a mass, an open fracture, etc.

[0225] Exemplarily, if an abnormal state exists, step 324 is executed; if an abnormal state does not exist, steps 325 to 326 or 327 to 328 are executed.

[0226] Step 324, marking ends.

[0227] Exemplarily, a prompt message may be output to medical personnel so that the medical personnel can handle the abnormal state. After the abnormal state is handled, step 321 may continue to be executed.

[0228] Step 325: Output stimulation.

[0229] Exemplarily, based on the audio data or eye pressure parameters included in the stimulation parameters in the aforementioned embodiments, audio data may be output or a supraorbital nerve pressure operation may be performed, thereby outputting stimulation to the patient to be diagnosed.

[0230] Exemplarily, during the execution of step 325 , step 327 may be executed synchronously.

[0231] Step 326: Summarize the stimuli.

[0232] For example, the types, intensities, durations, etc. of various stimuli may be summarized.

[0233] Step 327: Reaction detection.

[0234] Exemplarily, the reaction data of the patient to be diagnosed under the audio data or the supraorbital nerve pressing operation can be detected in real time; exemplary, the above reaction data can include eye video data and expression video data.

[0235] Step 328: Summarize the eyes and facial expressions.

[0236] Exemplarily, feature extraction can be performed on the eye video data and the expression video data according to the type and duration of stimulation output by the stimulation output module, and eye movements and expression movements can be obtained respectively, and the eye movements and expression movements can be summarized.

[0237] Step 329: Eye marking scoring.

[0238] Exemplarily, eye marking scoring may be performed by the eye scoring rules in the aforementioned embodiments.

[0239] Step 330: End of eye marking.

[0240] For example, the degree of eye reaction of the patient to be diagnosed can be determined based on the result of the eye mark scoring.

[0241] As can be seen from the above, the embodiment of the present application can automatically collect eye video data, and based on the eye video data and the output stimulation, detect the eyes and facial expressions of the patient to be diagnosed, so as to achieve eye marking and scoring. In this way, not only an automated and intelligent eye marking method is provided, but also the dependence on medical personnel and professional medical equipment can be reduced, and the efficiency of eye marking can be improved.

[0242] Based on the foregoing embodiments, in the method for determining the degree of coma provided in the embodiments of the present application, the collected data includes voice data output by the patient to be diagnosed; the processing result includes a voice recognition result for the voice data; and the pre-diagnosis strategy includes a voice pre-diagnosis strategy.

[0243] In one implementation, the speech recognition result may indicate whether the speech data contains specified semantics, and may also indicate the semantics represented by the speech data.

[0244] In one implementation, the speech pre-diagnosis strategy may include at least one of a speech data recognition method, process, condition, and a speech recognition result judgment rule.

[0245] Accordingly, the collected data for the patient to be diagnosed is processed based on the pre-diagnosis strategy to obtain the processing result, which can also be achieved in the following ways:

[0246] Output a first question list; collect voice data output when the patient to be diagnosed answers at least one question in the first question list; process the voice data based on the voice pre-diagnosis strategy to obtain a voice recognition result.

[0247] In one embodiment, the first question list may include at least one question; illustratively, the at least one question may be related to at least one of the personal information, historical dietary status, health status, and personal feelings of the patient to be diagnosed.

[0248] In one embodiment, the first list of questions may be pre-set; illustratively, the first list of questions may be determined based on at least one of the historical health data, intake status, and age stage of the patient to be diagnosed.

[0249] In one embodiment, the kth question in the first question list can be outputted through an audio output device or a video output device, and after the kth voice data of the patient answering the kth question is collected through the data collection device, the k+1th question is outputted; wherein the audio output device can output the kth question in the form of voice broadcast, and the video output device can output the kth question in the form of text display, image display or video playback; k can be an integer greater than or equal to 1.

[0250] In one implementation, the speech recognition result may be obtained by any of the following methods:

[0251] Based on the voice data integration strategy included in the voice pre-diagnosis strategy, the first question list and the voice data are integrated to obtain an integration result, and then feature extraction and feature recognition are performed on the integration result to obtain a voice recognition result.

[0252] Based on the speech pre-diagnosis strategy, the speech recognition method and target data are determined, and then the speech data is recognized by the speech recognition method to obtain recognition data, and the degree of matching between the recognition data and the target data is determined as the speech recognition result; wherein the target data may include the correct answers to the first question list.

[0253] From the above, it can be seen that in the method for determining the degree of coma provided in the embodiment of the present application, after outputting the first list of questions, by collecting the voice data output by the patient to be diagnosed when answering at least one question in the first list of questions, the probability of valid information contained in the voice data can be increased; on this basis, the effectiveness of the voice recognition results obtained by processing the voice data based on the voice pre-diagnosis strategy can be improved.

[0254] Based on the above embodiment, in the method for determining the degree of coma provided in the embodiment of the present application, outputting the first question list can also be implemented in the following manner:

[0255] Based on historical health data, the speech expression ability of the patient to be diagnosed is determined; if the speech expression ability indicates that the patient to be diagnosed is not in a state of speech expression disorder, a first question list is output.

[0256] For example, if the language expression ability indicates that the patient to be diagnosed is in a state of language expression disorder, the first question list may not be output, and the process of processing voice data based on the voice pre-diagnosis strategy may be terminated.

[0257] In one implementation, language expression ability may indicate whether the patient to be diagnosed has the ability to express himself in language, and may also indicate whether the patient to be diagnosed has the ability to correctly or clearly express subjective consciousness and objective facts.

[0258] From the above, it can be seen that in the method for determining the degree of coma provided in the embodiment of the present application, by determining the language expression ability of the patient to be diagnosed based on historical health data, the state and level of the patient to be diagnosed in terms of language expression can be accurately determined; and if the language expression ability indicates that the patient to be diagnosed is not in a state of language expression disorder, the first question list is output. In this way, not only can the probability that the patient to be diagnosed cannot output voice data after outputting the first question list be reduced, but also the probability of successfully obtaining voice data can be increased, thereby improving the efficiency of pre-diagnosis.

[0259] Based on the above embodiment, in the method for determining the degree of coma provided in the embodiment of the present application, outputting the first question list can also be implemented in the following manner:

[0260] If it is detected that the auditory organ of the patient to be diagnosed is in a damaged state, the coma degree determination device is controlled to move to the side of the auditory organ where the damage degree is less than the damage threshold, and a first question list is output.

[0261] For example, if it is detected that the auditory organ of the patient to be diagnosed is not in a damaged state, the first question list can be directly output.

[0262] For example, the first question list may be output when the patient to be diagnosed is not in a state of language expression disorder and his / her auditory organ is not in a damaged state.

[0263] In one embodiment, the auditory organ is in an injured state, which may include that the auditory organ contains an open wound, for example, the auricle of the patient to be diagnosed contains an open wound.

[0264] In one embodiment, facial video data may be collected to determine whether the auditory organ of the patient to be diagnosed is in a damaged state, and feature extraction may be performed on the facial video data using a feature extraction method to determine the degree of damage to the auditory organ.

[0265] In one embodiment, when it is determined based on historical health data that the auditory organ of the patient to be diagnosed is not hearing impaired, the coma degree determination device can be controlled to move to the side of the auditory organ where the degree of damage is less than the damage threshold, and a first question list can be output.

[0266] As can be seen from the above, in the method for determining the degree of coma provided in the embodiment of the present application, if it is detected that the auditory organ of the patient to be diagnosed is in a damaged state, the coma degree determination device is controlled to move to the side of the auditory organ where the degree of damage is less than the damage threshold, and the first list of questions is output. In this way, by detecting the damage state of the auditory organ, the probability that the patient to be diagnosed cannot identify the first list of questions due to the arbitrary output of the first list of questions can be reduced; and the probability that the patient to be diagnosed correctly identifies the first list of questions can be increased by controlling the coma degree determination device to move to the side of the auditory organ where the degree of damage is less than the damage threshold, and outputting the first list of questions.

[0267] Based on the above embodiments, in the method for determining the degree of coma provided in the embodiments of the present application, outputting the first question list can be achieved in the following manner:

[0268] Identify whether a designated part of the patient to be diagnosed includes an open wound; if the designated part does not include an open wound, output a first question list.

[0269] The designated part is at least associated with the pronunciation organ of the patient to be diagnosed.

[0270] Exemplarily, if the designated area does not include an open wound, the first question list is not output.

[0271] Exemplarily, the first question list may be output when the patient to be diagnosed does not suffer from a language expression disorder, his / her auditory organs are not damaged, and the designated part does not contain an open wound.

[0272] In one embodiment, the pronunciation organs may include respiratory organs, vocal organs, articulation organs, resonance organs, and the like.

[0273] In one embodiment, the designated part may include at least one part whose influence on the pronunciation process of the pronunciation organ is greater than or equal to a degree threshold; illustratively, the designated part may include facial features, arterial parts, and larynx, etc.

[0274] In one embodiment, the designated part does not include an open wound, which may indicate that the patient to be diagnosed is able to identify and answer at least one question in the first question list.

[0275] Accordingly, the method for determining the degree of coma provided in the embodiment of the present application may further include the following steps:

[0276] If the designated part includes an open wound, a second message is output to prompt medical personnel to treat the open wound.

[0277] The second information includes information that the designated part includes an open wound and location information of the patient to be diagnosed.

[0278] For example, after the medical staff has finished treating the open wound at the designated site, a first list of questions may be output and voice data may be collected.

[0279] From the above, it can be seen that in the method for determining the degree of coma provided in the embodiment of the present application, if the designated part of the patient associated with the vocal organ does not contain an open wound, a first list of questions is output, thereby increasing the probability that the patient to be diagnosed answers the questions in the first list of questions; and, if the designated part contains an open wound, second information is output. Since the second information contains the location information of the patient to be diagnosed and the information that the designated part contains an open wound, by outputting the second information, the probability of medical personnel rescuing the patient to be diagnosed in time can be increased, thereby reducing the danger level of the patient to be diagnosed caused by the designated part containing an open wound.

[0280] Based on the above embodiment, in the method for determining the degree of coma provided in the embodiment of the present application, outputting the first question list can also be implemented in the following manner:

[0281] Identify the facial video data containing the facial parts of the patient to be diagnosed, and obtain the facial coverage status of the patient to be diagnosed; if the facial coverage status indicates that the facial parts of the patient to be diagnosed are not in a covered state, output the first question list.

[0282] Exemplarily, if the facial features coverage status indicates that the facial features of the patient to be examined are covered, the first question list is not output.

[0283] In one embodiment, the facial features coverage status may include whether at least part of the facial features of the patient to be diagnosed are covered, and may also include the degree or area of ​​at least part of the facial features being covered.

[0284] In one implementation, feature extraction and facial feature recognition may be performed on facial feature video data to determine the facial feature coverage status.

[0285] Accordingly, the method for determining the degree of coma provided in the embodiment of the present application may further include the following steps:

[0286] If the facial features coverage status indicates that the facial features of the patient to be diagnosed are in a covered state, the facial features video data is processed by an image processing algorithm to remove the covering objects covering the facial features to obtain a second removed image; if the covering objects in the second removed image at least partially cover the facial features, a third information is output to prompt medical personnel to deal with the covering objects.

[0287] The third information includes information about the five sense organs covered by the covering object and location information of the patient to be diagnosed.

[0288] Exemplarily, if the facial features coverage status indicates that the facial features are not in a covered state, a first question list may be output.

[0289] Exemplarily, if the covered object in the second removed image no longer covers the facial features, or does not cover the auditory organs, visual organs, and mouth of the patient to be diagnosed, the first question list can be output.

[0290] Exemplarily, the first list of questions may be output when the patient does not have a language disorder, his / her auditory organs are not damaged, the designated part does not contain an open wound, and his / her facial features are not covered.

[0291] In one embodiment, the image processing algorithm may have facial features recognition and obstacle recognition functions.

[0292] In one implementation, the second removed image may be obtained by:

[0293] The facial features video data is intercepted to obtain at least one frame of facial features image, and then the at least one frame of facial features image is recognized through an image processing algorithm to determine the positions of the facial features and the size of the obstacle objects, and at least part of the obstacle objects covering the facial features are removed to obtain a second removed image.

[0294] In one embodiment, after the medical staff processes the covering object, a first list of questions is outputted if the patient to be examined is not in a state of speech expression disorder, his / her auditory organ is not in a state of damage, and the designated part does not contain an open wound.

[0295] From the above, it can be seen that in the method for determining the degree of coma provided in the embodiment of the present application, after identifying the facial video data containing the facial features of the patient to be diagnosed, the facial coverage status of the patient to be diagnosed is obtained. In this way, the facial coverage status can accurately reflect whether the facial features of the patient to be diagnosed are covered by the covering object; and, if the facial coverage status indicates that the facial features of the patient to be diagnosed are not in a covered state, a first list of questions is output, thereby increasing the probability that the patient to be diagnosed recognizes the questions in the first list of questions; and, if the facial coverage status indicates that the facial features of the patient to be diagnosed are in a covered state, the facial video data is processed by an image processing algorithm to remove the covering objects covering the facial features to obtain a second removed image, thereby realizing the automated and intelligent removal of the covering objects in the facial video data, and reducing the dependence on medical personnel; at the same time, if the covering object in the second removed image at least partially covers the facial features, a third information is output to prompt the medical personnel that the covering object covers the facial features and the position information of the patient to be diagnosed, thereby shortening the processing time of the covered objects.

[0296] Figure 4A A flow chart of the language barrier detection and the five sense organs detection provided in the embodiment of the present application is as follows: Figure 4A As shown, the auxiliary diagnosis robot 202 can locate the positions of ears and mouth; illustratively, the auxiliary diagnosis robot 202 can collect video data of the five sense organs and perform feature extraction on the video data of the five sense organs to determine the positions of ears and mouth.

[0297] Exemplarily, the auxiliary diagnosis robot 202 can analyze historical health data through the language disorder detection module 401 to obtain the language expression ability of the patient to be diagnosed to determine whether the patient to be diagnosed is in a state of language expression disorder. If the patient to be diagnosed is in a state of language expression disorder, the detection ends. If the patient to be diagnosed is not in a state of language expression disorder, subsequent detection can be performed through the facial features detection module 402.

[0298] Exemplarily, the facial features detection module 402 first detects whether the facial features are covered by a covering object. If a covering object is detected, it attempts to process the facial features video data through an image processing algorithm to remove the covering object to obtain a second removed image. Exemplarily, if the facial features are at least partially covered by the covering object in the second removed image, the medical staff is notified of the location information, requesting cooperation in processing the covering object, and entering standby mode to wait for orders from the medical staff.

[0299] Exemplarily, if no covering object is detected, it is detected whether the designated part of the patient contains an open wound, wherein the designated part may include vocal organs and arteries, etc.; exemplarily, if arterial bleeding is detected, the medical staff is notified of the location information and the bleeding location, and the device enters standby mode, waiting for the medical staff's command.

[0300] For example, if no open wound is detected in the designated area, or after the medical staff has finished treating the open wound in the designated area, it can be detected whether the facial area is in a state of trauma. If trauma is detected in the visible facial area, the trauma can be pre-graded through the facial video data, and the medical staff can be notified to request cooperation in treating the trauma, and then the system can enter standby mode and wait for orders from the medical staff.

[0301] For example, if the medical staff has finished treating the open wound at the designated site or has not detected any visible trauma to the facial area, the tracheal status can be detected; if tracheal injury (including incision) / intubation is detected, the detection ends.

[0302] Illustratively, the trachea and artery may be designated parts in the aforementioned embodiments.

[0303] Exemplarily, if the trachea is not in an injured or intubated state, both ears are detected. If the initial bilateral surgical examination shows no abnormality, the microphone is moved to the front of the patient to be examined and a first list of questions is output.

[0304] Exemplarily, if the initial surgical examination of a single ear is normal, the microphone is moved to the side of the normal ear and a first list of questions is output.

[0305] Exemplarily, if both ear surgical tests are abnormal, the medical staff is notified of the abnormalities in both ears, and the device enters standby mode, waiting for the medical staff's instructions.

[0306] From the above, it can be seen that in the embodiment of the present application, the auxiliary diagnostic robot can perform comprehensive, automated and accurate detection of the patient's language expression ability, facial coverage status, arterial bleeding status, facial area trauma, trachea and ears respectively through the language disorder detection module and the facial sense detection module, and can also terminate the detection in real time if the conditions for language detection are not met, thereby realizing automated and accurate comprehensive language ability detection and control of the patient, and improving the intelligence level of language detection.

[0307] Based on the foregoing embodiments, in the method for determining the degree of coma provided in the embodiments of the present application, the speech pre-diagnosis strategy includes a speech recognition model.

[0308] In one implementation, the speech recognition model may have semantic extraction and semantic category determination functions.

[0309] In one implementation, the speech recognition model may include a speech recognition module and a semantic matching module.

[0310] Exemplarily, the speech recognition model can be implemented by a convolutional neural network (CNN), which can process speech data using a mean normalization algorithm, and verify its recognition effect on the Spoken Arabic Digit dataset of the UCI machine learning library and a self-recorded keyword dataset. The keywords contained in the speech data can be identified by CNN, and the confidence probability features of the keywords can be determined, and then the speech recognition results can be determined based on the confidence probability features.

[0311] Exemplarily, the speech recognition model may include a long short-term memory network (Long Short-Term Memory, LSTM) and a recurrent neural network (Recurrent Neural Network, RNN).

[0312] Exemplarily, the speech recognition model can also be Spectral Subtraction Log-spectrum Normalization Dynamic Time Warping (SLN-DTW), where SLN-DTW is applied to the key information recognition of the examinee's answering speech based on unsupervised keyword detection technology, which can reduce the tedious workload corresponding to the collection, organization and annotation of large-scale large-vocabulary corpus, and related experiments have verified the effectiveness of SLN-DTW's speech keyword detection method and the effectiveness of the key information complete features calculated based on the detection results.

[0313] Exemplarily, the semantic matching module may include an Attention open information extraction (OIE) model based on deep learning, wherein Attention-OIE may include a bidirectional LSTM neural network as an input sentence encoder, and a multi-head Attention for extracting BIO (Beginning Inside Outside) labels for each sentence; wherein the above labels can be used as the reasoning basis for constructing sentence triples.

[0314] Exemplarily, the corpus used to train the semantic matching module may be open-ended sentences, and the output of the semantic matching module may be a BIO tag.

[0315] Accordingly, processing the speech data based on the pre-diagnosis strategy to obtain the speech recognition result can be achieved in the following ways:

[0316] Determine the clarity of the speech data; process the speech data through a speech recognition model to determine the degree of match between the speech data and the questions in the first question list; and determine the speech recognition result based on the clarity and the degree of match.

[0317] In one implementation, before determining the clarity, an adaptive Wiener noise reduction technology may be used to perform noise reduction processing on the speech data with respect to the background noise contained in the speech data, thereby removing most of the background noise contained in the speech data.

[0318] In one embodiment, the clarity level may include the degree to which data of basic semantic granularity contained in the speech data can be distinguished.

[0319] In one implementation, the clarity can be determined by a speech recognition module in a speech recognition model; illustratively, the speech data can be processed by SLN-DTW to determine the clarity, and the completeness and fluency of the semantic information in the speech data can also be determined.

[0320] In an embodiment of the present application, fluency can also be an evaluation parameter in a language pre-diagnosis strategy; illustratively, a speech segmentation technique based on a double threshold of short-time energy and zero-crossing rate can be used to segment speech data into voiced segments and silent segments, and the average pause feature calculated using the segmentation result can be used to determine the fluency of the pronunciation of the patient to be diagnosed.

[0321] In one implementation, the matching degree may include semantic information corresponding to the speech data, and whether the speech data completely matches the questions in the first question list, or whether the speech data partially matches the questions.

[0322] In one embodiment, the speech data is processed by a speech recognition model to extract semantic information contained in the speech data, and then the semantic information is matched with keywords contained in the questions in the first question list to determine the matching degree.

[0323] In one implementation, the first triplet of the voice information and the second triplet included in the question in the first question list can be obtained, and based on the correlation score between the first triplet and the second triplet, the similarity of the word sequence between the two triples is calculated by a greedy matching method, and the matching degree is determined by the similarity. The similarity can be calculated by formula (1) to formula (2):

[0324]

[0325]

[0326] Among them, TM(t Q ,t A ) indicates t Q With t A The final similarity is calculated between Q represents the correct answer sequence of the question, t A represents the collected speech data, ∑w Q w A Represents the sequence t Q and t A Greedy matching process, w is the corresponding matching sequence segment, cos_sim() represents the matching sequence that meets the cosine similarity.

[0327] Exemplarily, the voice data can also be matched with the correct answers to the questions in the first question list to determine the degree of matching; for example, if the voice data and the correct answer have the same meaning, which means that their triplet sequences are almost the same, their matching scores are close to 1; exemplarily, in the above matching process, several thresholds can be set in advance, and the degree of matching can be determined based on the relationship between the matching scores and the thresholds. For example, the several thresholds can include m1 and m2, wherein m2 can be less than m1, and m1 can be less than 1; if the matching score is greater than or equal to m1 and less than or equal to 1, the degree of matching can be determined to be 5 points; if the matching score is greater than or equal to m2 and less than m1, it can be determined that the patient to be diagnosed can respond, but the answer is not related to the question, and the degree of matching can be 4 points at this time; when the matching score is greater than or equal to 0 and less than m2, it can be determined that the patient to be diagnosed has the ability to express in a single character, and the degree of matching can be determined to be 3 points at this time.

[0328] In one embodiment, the speech recognition result may include a comprehensive scoring result of the patient's language expression ability, language organization ability, and problem identification ability.

[0329] In one implementation, a language scoring model may be pre-constructed, and the clarity and matching levels may be integrated and processed through the language scoring model to determine the speech recognition result.

[0330] Exemplarily, the correlation coefficient between the scoring features corresponding to the extracted speech sample data and the original scores can be calculated in advance, and the scoring features and the original scores can be linearly regressed using regression analysis to construct a language scoring model; wherein the original scores can include the scoring results of the speech sample data.

[0331] For example, the language scoring model can integrate the clarity and matching degree based on a pre-set language scoring rule. The language scoring rule can be as follows:

[0332] 5 points: Speak in an organized manner (oriented), with correct orientation ability, and can clearly express his or her name, city of residence or current location, year and month of the year; 4 points: Can answer, but there are situations where the answer is not related to the question (confused): disorientation and wrong answers; 3 points: Can say single words (inappropriate words): completely unable to have a conversation, can only say short sentences or single words; 2 points: Can make sounds (unintelligible sounds): can only make meaningless sounds in response to painful stimuli; 1 point: No response (none); T points: Unable to make normal sounds due to endotracheal intubation or incision, represented by "T" (tube); D points: There is a history of speech disorders, represented by "D" (dysphasic).

[0333] For example, the language scoring model can integrate the clarity and matching degree of the language scoring rules according to the age stage of the patient to be diagnosed. The language scoring rules corresponding to the age stage can be shown in Table 2. Figure 4B This is a flow chart of speech detection for adults provided in the embodiment of the present application. Figure 4B As shown, in the case where the patient 203 to be diagnosed is judged to be an adult, the speech detection module 403 can output a year question, such as asking what year this year is, and collect the voice data output by the patient 203 to be diagnosed in response to the question. If the patient to be diagnosed outputs a correct answer, the detection is completed, and based on the language scoring rules provided in the aforementioned embodiment, it is marked with a score of 5; if the patient to be diagnosed answers the year incorrectly, the detection is completed, and the score is marked with 4; if the patient to be diagnosed indicates that the question is unclear or cannot be heard, the medical staff is asked to continue the detection and reset; if the patient to be diagnosed speaks incoherently, the detection is completed, and the score is marked with 3; if the patient to be diagnosed has difficult speech, the detection is completed, and the score is marked with 2; if the patient to be diagnosed has no response, the detection is completed, and the score is marked with 1; if the patient to be diagnosed is a child, the detection can be completed by Figure 4C The flowchart shown is a flow chart for executing the speech detection process for non-adults.

[0334] Table 2

[0335] Age group 0-2 years old 2-5 years old More than 5 years old 5 Sound positioning, interaction Clear speech Have judgment and be able to talk 4 Crying and noisy, can be comforted Slurred speech Non-judgmental, able to talk 3 Groaning, inconsolable Crying Slurred speech 2 Snoring Snoring Slurred speech 1 No response No response No response

[0336] Figure 4C This is a flow chart of speech detection for non-adults provided in the embodiment of the present application. Figure 4C As shown, after the auxiliary diagnosis robot 202 reads the age, the age determination module 404 determines that the age is greater than 5 years old, and then executes Figure 4B The speech detection process is shown.

[0337] Exemplarily, if the age is determined to be between 2 and 5 years old, the child language scoring model 405 is used to perform speech detection on the patient to be diagnosed, and questions are asked to the child; if the patient speaks clearly, the detection is ended and the score is marked as 5; if he cries, the detection is ended and the score is marked as 4; if snoring is collected, the detection is ended and the score is marked as 2; if there is no response, the detection is ended and the score is marked as 1.

[0338] Exemplarily, if the age is determined to be less than 2 years old, the patient is subjected to speech detection through the infant language scoring model 406, and questions are asked to the infant; if the patient's voice can be located and can interact, the detection is ended and marked with a score of 5; if the crying can be comforted, the detection is ended and marked with a score of 4; if the child is moaning / inconsolable, the detection is ended and marked with a score of 3; if snoring is collected, the detection is ended and marked with a score of 2; if there is no response, the detection is ended and marked with a score of 1.

[0339] Through the above process, the embodiment of the present application can perform speech detection on patients of different age stages, thereby achieving targeted speech pre-diagnosis.

[0340] From the above, it can be seen that in the method for determining the degree of coma provided in the embodiment of the present application, the language organization ability and language expression ability of the patient to be diagnosed can be accurately characterized by the clarity of the voice data; and, the voice data is processed by the voice recognition model to determine the degree of match between the voice data and the questions in the first question list, thereby, through the degree of match, the sensitivity of the auditory organ of the patient to be diagnosed and the thinking state of the thinking organ can be indirectly reflected; on this basis, based on the clarity and the degree of match, the voice recognition result is determined, which can not only improve the accuracy of the voice recognition result, but also improve the display of the consciousness state of the patient to be diagnosed in at least one aspect.

[0341] Based on the foregoing embodiments, in the method for determining the degree of coma provided in the embodiments of the present application, the pre-diagnosis strategy includes a limb movement pre-diagnosis strategy; the collected data includes movement video data of the limb parts of the patient to be diagnosed; and the processing result includes a movement recognition result.

[0342] In one embodiment, the limb part may include a part of the patient to be diagnosed that can currently move or perform actions; exemplarily, the limb part may include a limb part, or at least a part of a limb part; exemplarily, the limb part may include at least one body part on the non-hemiplegic side of the patient to be diagnosed.

[0343] In one embodiment, the body movement prediction strategy may include processes, steps, and conditions for identifying movement video data and obtaining movement recognition results.

[0344] In one embodiment, the motion video data may include collected video data of a limb of the patient to be diagnosed performing a motion process.

[0345] In one implementation, the action recognition result may include the number or name of the action obtained by extracting features from the action video data.

[0346] Accordingly, the collected data for the patient to be diagnosed is processed based on the pre-diagnosis strategy to obtain the processing result, which can be achieved by the following steps:

[0347] Step B1: If at least one of the limb parts is not in a state of trauma of a specified degree, output a second question list.

[0348] For example, if all parts of the limbs are in a state of trauma of a specified degree, the second list of questions may not be output.

[0349] In one embodiment, the designated degree may be determined according to hemiplegia rules or injury levels in clinical medicine.

[0350] In one embodiment, historical health data of the patient to be diagnosed may be analyzed to determine whether at least one of the limb parts is in a state of injury of a specified degree.

[0351] In one embodiment, the at least one site that is not in a state of trauma of a specified degree may include a site that does not include an open wound or a non-hemiplegic side.

[0352] In one embodiment, the questions in the second question list may include questions that require the patient to control a limb part to perform at least one action; exemplarily, the questions in the second question list may include at least one of the name of the action, the part of the body associated with the action, the strength of the action, and the length of time for performing the action.

[0353] Step B2: collecting the actions performed by the patient to be diagnosed to control at least one part of the body in response to at least one question in the second question list to obtain action video data.

[0354] In one embodiment, when outputting at least one question in the second question list, the position of at least one part can be determined, and a video acquisition device can be started to acquire action video data.

[0355] Step B3: Process the action video data based on the action pre-diagnosis strategy to obtain the action recognition result.

[0356] In one implementation, the action recognition model included in the action pre-diagnosis strategy may be used to extract features from the action video data to obtain an action recognition result.

[0357] In one implementation, the action recognition result may include a classification result or a scoring result of the action performed by the patient.

[0358] In one implementation, the action recognition result may include a matching degree between the action performed by the patient to be diagnosed and at least one question in the second question list.

[0359] From the above, it can be seen that in the method for determining the degree of coma provided in the embodiment of the present application, if at least one of the limb parts of the patient to be diagnosed is not in a state of trauma of a specified degree, a second list of questions is output, thereby achieving strict control over the operation of outputting the second list of questions; and, the action of the patient to be diagnosed to control at least one part to perform in response to at least one question in the second list of questions is collected to obtain action video data, which not only improves the degree of correlation between the action video data and the second list of questions, but also provides a basis for judging the actions contained in the action video data; on this basis, after the action video data is processed based on the action pre-diagnosis strategy to obtain the action recognition result, the action recognition result can not only determine the activity level of at least one part of the patient to be diagnosed, but also judge the activity level of the patient's hearing, vision and thinking organs.

[0360] Based on the foregoing embodiment, in the method for determining the degree of coma provided by the embodiment of the present application, at least one question in the second question list includes a finger movement question.

[0361] In one embodiment, the finger motion question may include a question instructing at least one finger of the patient to be diagnosed to perform at least one specified finger motion.

[0362] In one implementation, the number of questions in the finger motion question may be at least one.

[0363] In one embodiment, the finger movement problem may be associated with at least one hand of the patient being examined.

[0364] Accordingly, collecting the action performed by the patient to be diagnosed to control at least one part of the body in response to at least one question in the second question list to obtain action video data can be achieved by the following steps:

[0365] Step C1: collecting hand movements performed by the patient for finger movement problems to obtain hand movement data.

[0366] In one implementation, the hand motion data may include at least one frame of image data including hand motion.

[0367] In one embodiment, the hand motion data may include a hand motion video including a continuous execution process of the hand motion.

[0368] Step C2: if the hand motion data indicates that at least one finger does not perform a designated finger motion, a pressure stimulation operation is performed on at least one body part of the patient to be diagnosed.

[0369] Exemplarily, if the hand motion data indicates that at least one finger performs a specified finger motion, then a specified pressure stimulation operation may not be performed on at least one body part of the patient to be diagnosed.

[0370] In one implementation, the at least one finger may include a finger specified in a finger motion question.

[0371] In one implementation, specifying the hand motion may include specifying the hand motion in a finger motion question.

[0372] In one embodiment, at least one body part may include a body part that is not in a state of trauma to a specified degree; exemplarily, at least one body part may include a part on which a pressure stimulation operation is performed so that the patient to be diagnosed can perform a response action, but the degree of damage to the patient to be diagnosed is less than an injury threshold; exemplarily, in the case where the visual organ of the patient to be diagnosed does not contain an open wound, at least one body part may include the supraorbital nerve.

[0373] In one embodiment, at least one of the type, strength, area, and duration of the pressure stimulation operation may be preset in the motion prognosis strategy.

[0374] Step C3: During the pressure stimulation operation, a reaction action performed by at least one body part is collected to obtain stimulation action data.

[0375] In one embodiment, at least one body part can be located, and an image acquisition device can be controlled to acquire a reaction action performed by at least one body part, thereby obtaining stimulation action data.

[0376] Step C4: Determine that the hand motion data or stimulation motion data is motion video data.

[0377] In one implementation, if the hand motion data indicates that the finger performs a specified finger motion, stimulation motion data may not be collected. In this case, the hand motion data may be determined as motion video data.

[0378] In one implementation, if the hand motion data indicates that at least one finger does not perform a designated finger motion, stimulation motion data may be collected and determined as motion video data.

[0379] From the above, it can be seen that in the method for determining the degree of coma provided in the embodiment of the present application, the hand movements performed by the patient to be diagnosed in response to the finger movement problem are collected to obtain hand movement data. In this way, the flexibility and controllability of the hand of the patient to be diagnosed can be accurately reflected through the hand movement data; and if the hand movement data indicates that at least one finger has not performed the specified finger movement, a pressure stimulation operation is performed on at least one body part of the patient to be diagnosed, and during the execution of the pressure stimulation operation, the reaction movement performed by at least one body part is collected to obtain stimulation movement data. Therefore, when the hand of the patient to be diagnosed cannot perform or does not perform the specified finger movement, the stimulation movement data is obtained through the above-mentioned pressure stimulation operation, which can achieve all-round tracking of the stimulation reaction of the patient to be diagnosed; on this basis, determining the hand movement data or stimulation movement data as action video data can increase the probability of valid movements contained in the action video data.

[0380] Based on the above embodiment, in the method for determining the degree of coma provided in the embodiment of the present application, outputting the second question list can be implemented in the following manner:

[0381] Identify whether the limb part is in a specified state; if the limb part is not in the specified state, output a second question list.

[0382] The specified state includes at least one of an open wound, heavy pressure, and swelling on the limb part.

[0383] In one implementation, limb video data may be collected and feature extraction may be performed on the limb video data to determine whether a limb part is in a specified state.

[0384] In one embodiment, the designated status may be determined based on clinically diagnosed medical risks and rules.

[0385] Accordingly, the method for determining the degree of coma provided in the embodiment of the present application may further include the following steps:

[0386] If the limb part is in the specified state, fourth information indicating that the limb part is in the specified state is output to prompt medical personnel to handle the limb part in the specified state.

[0387] The fourth information at least includes the location information of the patient to be diagnosed.

[0388] Figure 5A A schematic diagram of a process for collecting hand motion data provided in an embodiment of the present application, such as Figure 5AAs shown, if the auxiliary diagnostic robot 202 determines that there is trauma, obvious mass, open fracture or compression by external objects in the limbs, the immediate trauma processing module 501 collects the limb video data including the limb parts, and analyzes the limb video data, so as to perform trauma pre-grading and notify the medical staff to cooperate in treating the trauma; after the medical staff is in place, the countdown is executed, waiting for the medical staff to treat the trauma, and then the robot enters the standby mode and waits for the medical staff's order to continue detection / reset.

[0389] Exemplarily, if the medical staff has finished treating the trauma, or has determined that there is no trauma, obvious mass or open fracture on the limbs, or compression by external objects, the question can be output through the question output module 502, and the finger position can be identified at the same time, and then a voice prompt "extend 3 fingers" is given, and a video of the finger movement is collected synchronously; if local fingers are detected, a voice prompt "please make a fist" is given, and a video of the fist-making movement is collected synchronously; if the specified action cannot be detected, the medical staff is notified to handle it, and the mode is entered into standby mode, waiting for the medical staff's command to continue detection / resetting, and the response of the patient to be diagnosed can also be continued to be captured; wherein, the specified action may include extending three fingers and making a fist.

[0390] Through the above process, the hand movements of the patient to be diagnosed can be detected with the help of the question output module. In practical applications, the movements performed by the patient to be diagnosed can also be scored with the help of the movement scoring rules. Among them, the movement scoring rules can be as follows:

[0391] 6 points: obey commands: complete 2 different actions according to commands; 5 points: localize the pain when stimulated: the patient can move the limbs to try to remove the stimulation when stimulated. The gold standard for pain stimulation is to press the supraorbital nerve; 4 points: the limbs will withdraw in response to pain stimulation; 3 points: the limbs will bend in response to pain stimulation (decorticate flexion): in a "decerebrate rigidity" posture; 2 points: the limbs will extend in response to pain stimulation (decerebrate extension): in a "decerebrate rigidity" posture; 1 point: no response (noresponse).

[0392] Figure 5B A schematic diagram of a process for marking and scoring based on action scoring rules provided in an embodiment of the present application, such as Figure 5BAs shown, if the auxiliary diagnosis robot 202 detects the specified finger movement, the detection is completed and the mark is scored 6; if the specified finger movement is not detected, the auxiliary diagnosis robot presses the supraorbital nerve and detects whether the patient to be diagnosed can locate the pain position; if the pain position is detected, the detection is completed and the mark is scored 5; if the pain position of the patient to be diagnosed is not detected, the finger of the patient to be diagnosed is touched, and if the patient to be diagnosed avoids it, the robot stops immediately, wherein the strength of the finger contacting the patient to be diagnosed can be gradually increased within a safe range, and the longest can be, for example, 10 seconds, so as to stimulate the patient to be diagnosed to feel a tingling pain; at the same time, it is detected whether the patient to be diagnosed avoids the tingling pain, if it is detected that the patient avoids the tingling pain, the detection is completed and the mark is scored 4; if a flexion reaction is detected, the detection is completed and the mark is scored 3; if a hyperextension reaction is detected, the detection is completed and the mark is scored 3; if there is no movement, the detection is completed and the mark is scored 3.

[0393] Exemplarily, after the marking and scoring process for one hand of the patient is completed, the hand can be changed to repeat the above process, and the patient can be prompted to perform the test again using the right hand or the left hand; exemplarily, after the above process is completed, the scores can be summarized to obtain the action score corresponding to the action recognition result.

[0394] Exemplarily, the above-mentioned scoring and marking process can be performed according to the action scoring rules.

[0395] Therefore, through the above process, the patient's reaction to the tingling pain applied to his finger by the auxiliary diagnosis robot can be detected, and different tingling reaction actions can be quantitatively scored, thereby not only realizing multi-dimensional tracking and detection of the patient's reaction actions, but also realizing accurate quantification of the patient's tingling pain reaction.

[0396] From the above, it can be seen that the method for determining the degree of coma provided in the embodiment of the present application can determine whether to output the second list of questions based on whether the limb part is in a specified state, thereby realizing precise control over the operation of outputting the second list of questions; and, when the limb part is in a specified state, the fourth information can be output so that medical personnel can promptly treat the limb part of the patient to be diagnosed, thereby reducing the health risks of the patient to be diagnosed in real time.

[0397] Based on the foregoing embodiments, in the method for determining the degree of coma provided in the embodiments of the present application, the motion video data includes depth video data containing motion; and the limb motion pre-diagnosis strategy includes a motion recognition model.

[0398] In one embodiment, the depth video data may include a distance parameter between at least one body part of the patient to be diagnosed and the coma degree determination device; for example, the depth video data may be obtained by Figure 5C The data acquisition device shown is used to obtain the data.

[0399] Figure 5C The data acquisition device 503 can realize the dual functions of image acquisition and audio acquisition, and can be integrated in a coma degree determination device or an auxiliary diagnosis robot.

[0400] Exemplarily, the infrared camera 504 can project infrared rays, the red, green, blue (RGB) camera 505 can collect visible light, and the depth camera 506 is used to analyze infrared rays and jointly create depth image data or depth video data within the visible range with the RGB camera 505.

[0401] Exemplarily, the motor 507 with a rotation function can control the rotation angle of the image acquisition device during the image acquisition or audio acquisition process; the microphone array 508 can realize automatic acquisition and recognition of voice.

[0402] Figure 5D A schematic diagram of a process for obtaining a depth image provided in an embodiment of the present application is shown in FIG. Figure 5D As shown: the data acquisition device 503 controls the infrared camera 504 to emit infrared rays. When the infrared rays are projected onto the patient to be diagnosed, the patient to be diagnosed reflects the infrared rays. At this time, the depth camera 506 receives the infrared rays reflected by the patient to be diagnosed, and based on the coding mechanism it contains, encodes the infrared spectrum of the infrared rays reflected by the patient to be diagnosed to obtain a spot image; the image generation module 509 can perform depth processing on the spot image to generate a depth image or depth video data.

[0403] Figure 5E A schematic diagram of the structure of the control data acquisition device provided in the embodiment of the present application. In practical applications, the data acquisition device has powerful data acquisition and processing functions. Therefore, through the API of the natural user interface (Natural User Interface, NUI) provided by the application programming interface (Application Programming Interface, API) providing module 509 and the data requirements provided by the user development module 510, the transmission of the depth image data stream, color image data stream and skeleton data stream between the data acquisition device 503 and the user development module 510 can be realized, thereby providing conditions for the user development module 510 to perform data acquisition and data recognition.

[0404] Fig. 5F A schematic diagram of the structure of the data recognition capability provided by the data acquisition device provided in the embodiment of the present application. Fig. 5FAs shown, the API set 511 may include a color data API for realizing the transmission of RGB image data or video data. The data transmitted through the API may include image features required for face tracking 512, lip reading recognition 513, and eye recognition; the depth data API in the API set is used to realize the transmission of depth image or depth video data. These data can serve as the basis for sign language recognition 514 and motion recognition 515, wherein sign language recognition 514 can be realized by finger tracking, and motion recognition 515 can be realized by skeleton tracking; the audio data API in the API set is used to transmit audio data for the user module to realize the voice recognition 516 function.

[0405] In practical applications, the data acquisition device can be controlled to illuminate the spatial scene where the patient to be diagnosed is located with a laser speckle light source. Since laser speckle is highly random, the generated diffraction spots can be converted into different patterns as the distance changes. Therefore, this structured light only needs to be projected in a specific spatial scene, and objects at different distances can produce different patterns. The speckle image can be processed by calibrating the light source system, thereby obtaining the position information of at least one body part and bone node of the patient to be diagnosed.

[0406] pass FIG. 5D to FIG. 5F The structure shown, the powerful data acquisition function of the image acquisition device can be flexibly controlled by the user module in various situations, thereby providing data basis for determining the degree of coma.

[0407] In one implementation, the action recognition model can identify the type and name of the action performed by the patient to be diagnosed, and determine whether the action performed by the patient to be diagnosed is a target action corresponding to at least one question in the second question list.

[0408] Accordingly, based on the action pre-diagnosis strategy, the action video data is processed to obtain the action recognition result, which can be achieved in the following ways:

[0409] Segment the depth video data to obtain a depth image; identify the depth image to obtain the bone node positions of the limb parts; identify the bone node positions through the action recognition model to obtain the action recognition results.

[0410] Among them, the action recognition model at least includes a cascaded CNN and LSTM.

[0411] In one implementation, the depth image may include at least one frame of image in the depth video data.

[0412] In one embodiment, after segmenting the depth video data to obtain at least one frame of image, the human body contour in the at least one frame of image can be segmented by edge detection and feature extraction, thereby obtaining a depth of field image after background removal, and determining the depth of field image as a depth image.

[0413] Figure 5G A schematic diagram of a depth image provided in an embodiment of the present application. Figure 5G The depth image shown is different from a two-dimensional image. In this image, each pixel is not an independent grayscale value, but consists of 16-bit binary data.

[0414] Figure 5H This is a schematic diagram of the structure of the depth image pixel data provided by the embodiment of the present application. Figure 5H As shown, the upper 13 bits of the pixel are the depth value of the pixel (Depth), the unit of which can be millimeters, and the lower 3 bits are the user index number (Player Index Bitmask, PIB)

[0415] Exemplarily, a higher brightness of a pixel indicates a greater distance between the pixel and the data acquisition device, while a lower brightness indicates a closer distance between the pixel and the data acquisition device.

[0416] Exemplarily, the setting position and / or angle of the data acquisition device can be controlled to adjust the viewing angle so that the limb parts of the patient to be diagnosed can be included in the depth image; if the viewing angle is too small, the relevant image features of the limb parts may not be obtained, and if the viewing angle is too large, the feature accuracy in the depth image may be insufficient.

[0417] In one embodiment, the bone node positions may include the spatial positions of the bone nodes of the limb parts of the patient to be diagnosed.

[0418] In one implementation, the positions of the skeleton nodes may be obtained by processing the depth image using a skeleton recognition model.

[0419] Exemplarily, a skeleton recognition model can be trained through a machine learning algorithm so that the skeleton recognition model can predict the possible actions that various parts of the human body may perform, and determine for each body part the region category with the highest probability that it belongs to a human body part. For example, if the result of action recognition indicates that the probability of the arm region is the highest, then the part where the action is performed is determined to be the arm region.

[0420] Exemplarily, after determining the above-mentioned region categories, the joint points of the body parts corresponding to the above-mentioned region categories may be determined, and then the three-dimensional coordinate information of the joint points, ie, the positions of the bone nodes, may be determined according to the pixel grayscale.

[0421] Exemplarily, after separating the outline of the patient to be diagnosed from the depth image, the bone data of different parts of the human body can be determined in the above manner, and a bone image can be generated, and then the positions of the joint points in the bone image can be determined as the positions of the bone nodes.

[0422] Fig.5I A schematic diagram of the effect of mapping a skeleton image to a background binary image provided in an embodiment of the present application. Fig.5I The skeleton diagram shown is similar to Figure 5G Corresponding to, where the background part is binarized as Fig.5I The white area in Fig.5I The black area in the figure may be the outline area of ​​the patient to be diagnosed, and the white lines in the outline area may be the bone data of the patient to be diagnosed, which is used to characterize the bone structure of the patient to be diagnosed.

[0423] In an embodiment of the present application, the action recognition model can be obtained by training with skeleton sample data.

[0424] For example, the action recognition model may be trained by separately training CNN and LSTM.

[0425] For example, CNN and two multi-layer perceptrons (MLP) may be first connected to pre-train CNN; in the above training process, T may be set as the number of continuous time steps corresponding to the action or gesture contained in the sample data.

[0426] Exemplarily, the first convolution layer of CNN can have 20 convolution kernels, and the convolution kernel size can be 3×3×3, which is used to filter T×20×3 skeleton sample data or T×22×3 hand sample data, and output 20 (T-2)×18×1 data or (T-2)×20×1 data respectively; where T can be an integer greater than 2.

[0427] Figure 5J This is a schematic diagram of the structure of the joint points of the skeleton sample data and the hand sample data provided in the embodiment of the present application. Figure 5J As shown, the skeleton sample data may include 20 joint points, and the hand sample data may include 22 joint points.

[0428] For example, the pooling layer of a CNN can perform a non-overlapping maximum operation in a 2×2×1 neighborhood, thereby halving the original dimensionality of the time step and the number of skeleton samples to (T-2) / 2×9×20 data, and corresponding to the hand sample data, the number of bits is halved to (T-2) / 2×10×20.

[0429] For example, the second convolution layer of CNN uses 50 convolution kernels of size 2×2×20, and is connected to a 2×2×1 max pooling layer, thereby reducing the data size to (T-4) / 4×4×50 elements. The last convolution layer filters the output of the previous convolution layer with 100 kernels of size 3×3×50, and then performs a 2×2×1 max pooling operation again to reduce the data dimension, and connects the output of the data size (T-12) / 8×1×100 to two fully connected MLPs with hidden layers of 300 and 100 neurons respectively, and then outputs the action category vector through a softmax layer.

[0430] Exemplarily, after the CNN pre-training is completed, its connection with the MLP layer is disconnected and cascaded to the LSTM. At this time, the number of weights learned by the CNN and LSTM in the cascade state can be 132 and 130.

[0431] In the above training process, the joint point set △ of the skeleton sample data and the hand sample data can be expressed by formula (3):

[0432] △={P j ,1≤j≤J} (3)

[0433] Among them, P j represents the jth joint point included in the skeleton sample data and the hand sample data, and J is the total number of joint points included in the skeleton sample data and the hand sample data.

[0434] For example, in formula (3), each joint point P j The three-dimensional coordinates P that can pass through j ={x j ,y j ,z j} indicates that, where P j ∈R 3 .

[0435] Exemplarily, the sample data input to the CNN may include skeleton sample data within a time step T or a plurality of joint point sets contained in hand sample data.

[0436] In one implementation, the action recognition model may also be a feedforward neural network (Back Propagation, BP).

[0437] Exemplarily, in order to improve the accuracy of BP action recognition, the model structure can be optimized and the model parameters can be adjusted according to the sample data, wherein the above-mentioned optimization of the model structure and adjustment of the model parameters can be achieved by adjusting or optimizing the number of network layers, the number of neuron nodes in different layers, the initial weights of neurons, thresholds and at least one of the BP optimization methods.

[0438] Exemplarily, when optimizing the BP network structure, the number of neuron nodes in the input layer and the output layer can be set first. Since the characteristics of action recognition are based on the Euclidean distances of 14 groups of joint points, that is, there are 14 distance features for an action, the input data is fourteen-dimensional. Therefore, the number of neurons in the input layer can be 14.

[0439] It should be noted that in order to improve the training efficiency of dynamic BP and the recognition accuracy of BP, the sample data can include the Euclidean distances between fourteen groups of joint points with relatively obvious differences; among which, Table 3 is a summary of the information of the fourteen groups of joint points:

[0440] Table 3

[0441] Serial number Feature joint group Serial number Feature joint group 1 Right hand - head 8 Right hand - right hip 2 Left hand - head 9 Left hand-left hip 3 Right hand - right knee 10 Right elbow-right hip 4 Left hand - left knee 11 Left elbow-left hip 5 Right hand - right foot 12 Right elbow-left knee 6 Left hand - left foot 13 Left elbow-right knee 7 Right hand - Left hand 14 Right foot - left foot

[0442] For example, the coordinates of the two joint points are (x 1 ,y 1 ,z 1 ) and (x 2 ,y 2 ,z 2 ) as an example, the distance between the two joint points can be calculated by formula (4):

[0443]

[0444] in, It can be the bone position weight, and its value can be 1000 to strengthen the bone joint features during training.

[0445] Table 4 shows the statistical results of some distance features obtained by calculating the Euclidean distance of the joint points in Table 3: Figure 5K Schematic diagram of the set statistical results of node distance features.

[0446] Exemplarily, the output layer neurons can be symmetrical with the independent dimension of a vector and are used to characterize the recognition results of the action feature data. Therefore, the number of neurons in the output layer is set to the number of action categories to be recognized. For example, if the number of action categories to be recognized is 5, the number of neurons in the output layer can also be set to 5; wherein the above vector can be as shown in Table 5.

[0447] Table 4

[0448]

[0449] For example, when the loss error value Emax of BP is set to 0.05 and the maximum number of training times is set to 1000 times, it can be determined from the training process that the loss error value of the hidden layer of the three-layer structure is around 0.5, which is quite different from Emax, and the recognition rate is only 82%. Therefore, the hidden layer can be set to a four-layer structure.

[0450] Exemplarily, the number of nodes in the two hidden layers of BP can be set to the same number. Through the training process, it can be determined that when the number of neurons is less than 10 and the number of training times is 1000, the degree of loss error of BP is significantly greater than Emax; when the number of neurons contained in the hidden layer reaches more than 10, the number of training times of BP at the maximum loss error decreases with the increase of the number of neuron nodes; when the number of neurons contained in the hidden layer reaches more than 20, the number of training times when BP reaches the maximum loss error does not change significantly. Therefore, the present application can set the number of neurons in the hidden layer to 20.

[0451] Table 5

[0452] Action Name Expected output vector Raise your arms in front of you (1，0，0，0，0) Raise your arms (0，1，0，0，0) Cross your arms (0，0，1，0，0) Left leg raise (0，0，0，1，0) Right leg lift (0，0，0，0，1)

[0453] Exemplarily, there is a significant correlation between the learning efficiency of BP and the initialization parameters of the threshold and weight. For nonlinear systems, the weights and thresholds set in the initial state have a great influence on the local convergence effect in learning. When setting the initial weights and thresholds, it is necessary to control the values ​​of the initialization parameters, and at the same time, it is necessary to maintain a significant difference between the values ​​of different parameters; if the initial value is large, the neuron function will reach saturation in a shorter time during the BP learning process; otherwise, the BP learning ability will be lost. Therefore, in the embodiment of the present application, the setting of the weights and thresholds of BP mainly adopts random numbers between -1 and 1.

[0454] Exemplarily, the BP loss function is mainly used to evaluate the degree of match between the output data of BP and the action identifiers contained in the sample data. It is essentially a relationship function. When the value calculated by the loss function is small, it can be determined that BP has good robustness; among them, the loss function is one of many risk functions. In the embodiment of the present application, the loss function of BP can be a cross entropy function.

[0455] Exemplarily, the optimization algorithm used in the BP training process may be a stochastic gradient descent algorithm, and its learning rate may be set to 0.001; wherein, the stochastic gradient descent algorithm may update and optimize BP parameters through a targeted sample, and has extremely high efficiency in learning.

[0456] From the above, it can be seen that in the method for determining the degree of coma provided in the embodiment of the present application, the depth video data is segmented to obtain a depth image. In this way, the depth image can comprehensively and accurately reflect the spatial position change state of at least one part of the patient during the process of performing an action; and the depth image is identified to obtain the skeletal node positions of the limb parts, and the skeletal node positions are identified through the action recognition model to obtain the action recognition results. Since the skeletal node positions only contain the skeletal node information of the limb parts, the amount of calculation of the action recognition model to obtain the action recognition results can be reduced, thereby improving the efficiency of obtaining the action recognition results; at the same time, since the action recognition model includes a cascaded CNN and LSTM, the time-related skeletal node movement relationship contained in the skeletal node positions can be extracted through CNN and LSTM, thereby improving the accuracy of the action recognition results.

[0457] Based on the above embodiments, in the method for determining the degree of coma provided in the embodiments of the present application, the position of the skeletal nodes is identified by the action recognition model to obtain the action recognition result, which can also be achieved in the following ways:

[0458] Determine the number of bone nodes associated with limb parts, and determine the duration of at least one part performing an action; based on the duration, determine the target number of frames for at least one part to perform the action; integrate the number of bone nodes, the positions of the bone nodes associated with the number of bone nodes, and the target number of frames to obtain input data; process the input data through a bone recognition model to obtain an action recognition result.

[0459] In one embodiment, different limb parts are associated with different numbers of bone nodes. For example, in the aforementioned embodiment, the number of bone nodes corresponding to the skeleton sample data may be different from the number of bone nodes corresponding to the hand sample data.

[0460] In one embodiment, the duration may include the time period from the start of the patient's action to the end of the action; illustratively, the duration may be determined by analyzing the action features contained in the depth video data.

[0461] In one implementation, the target frame number may be greater than or equal to 1; illustratively, the depth video data may be segmented based on the duration to obtain an image set of depth images of the target frame number.

[0462] In one embodiment, the input data may be obtained by:

[0463] According to the action time associated with the depth image in the image set, the number of bone nodes corresponding to the depth image and the bone node positions associated with the number of bone nodes are integrated to obtain a high-dimensional data block, and the above data block is determined as input data; illustratively, the input data can be expressed by formula (5):

[0464]

[0465] in, It can be a data block including the positions of the bone nodes between time t and time t+T, where t and T are both integers greater than 0; T can be a duration.

[0466] Exemplarily, the above data blocks can be input into CNN and LSTM in a cascade state, and the output result of LSTM is sent to the softmax activation function through the fully connected layer for action category classification, thereby obtaining the action recognition result.

[0467] Figure 5L A flow chart of action recognition provided in an embodiment of the present application is shown in FIG. Figure 5L As shown, the process may include the following steps:

[0468] Step 517, start.

[0469] Step 518, initialization.

[0470] Exemplarily, the motion recognition model may be set to switch to a ready state during the initialization phase.

[0471] Step 519: Extract skeleton data.

[0472] Illustratively, the skeleton data may include input data.

[0473] Step 520: Determine whether to start recognition.

[0474] Exemplarily, if recognition is started, step 521 is executed; if recognition is not started, step 518 is executed.

[0475] Step 521: Identify the skeleton data to obtain the action recognition result.

[0476] Exemplarily, the skeleton data can be recognized by a skeleton recognition model to obtain an action recognition result.

[0477] Step 522: Display the action recognition result.

[0478] Step 523: Determine whether to continue recognition.

[0479] Exemplarily, whether to continue recognition can be determined based on the action recognition result. For example, if the action recognition result indicates that at least one limb part of the patient to be diagnosed can perform at least part of the specified action, recognition can be continued, and step 520 can be executed at this time; if the action recognition result indicates that the patient to be diagnosed did not perform the action, recognition can be discontinued, and step 524 can be executed at this time.

[0480] Step 524, end.

[0481] Through the above process, continuous, automatic and intelligent detection and recognition of at least one action performed by the patient to be diagnosed can be achieved, thereby improving the efficiency of action pre-diagnosis.

[0482] From the above, it can be seen that the method for determining the degree of coma provided in the embodiment of the present application provides skeletal node guarantee for accurate tracking and identification of actions by determining the number of skeletal nodes associated with limb parts; and after determining the duration of at least one part performing the action, the target number of frames for at least one part to perform the action is determined based on the duration, so that the range of the depth image based on which the action recognition is based can be locked, thereby reducing the amount of computational complexity of the action recognition; at the same time, the number of skeletal nodes, the positions of the skeletal nodes associated with the number of skeletal nodes, and the target number of frames are integrated to obtain input data, so that the input data can comprehensively include the action execution process of the patient to be diagnosed. On this basis, the accuracy of the action recognition results obtained by processing the input data by the action recognition model can be improved.

[0483] Fig. 6A A schematic diagram of the structure of a device for determining the degree of coma provided in an embodiment of the present application is shown in FIG. Fig. 6A As shown, the coma degree determination device 6 can be the auxiliary diagnosis robot in the aforementioned embodiment.

[0484] Exemplarily, the coma degree determination device 6 may include a pre-processing module 601 , an eye-opening reaction detection module 602 , a speech reaction detection module 603 , and a non-hemiplegic side movement reaction detection module 604 .

[0485] Among them, the preprocessing module 601 can be used to obtain the historical health data of the patient to be diagnosed, the intake status of the patient to be diagnosed for at least one specified substance and the age information of the patient to be diagnosed, and according to the historical health data, intake status and age information, determine the selection of the eye opening reaction detection module 602, the speech reaction detection module 603 and the non-hemiplegic side motor reaction detection module 604 to start the prediagnosis strategy of the corresponding category detection.

[0486] Exemplarily, the eye opening reaction detection module 602 can output a first list of questions and supraorbital nerve pressing operations to the patient to be diagnosed by asking questions and providing pain stimulation based on the eye pre-diagnosis strategy determined in the aforementioned embodiments, and simultaneously collect eye video data, and then use the eye recognition model to determine the eye opening feedback and mark and score it accordingly.

[0487] Exemplarily, the speech response detection module 603 can output a second question list by asking questions based on the speech pre-diagnosis strategy in the aforementioned embodiment, simultaneously collect the speech data output by the patient to be diagnosed, and process the speech data through the speech recognition model to determine the speech feedback and mark and score it accordingly.

[0488] Exemplarily, the non-hemiplegic side action reaction detection module 604 can output a second question list and the stimulation reaction movements of the patient to be diagnosed respectively through questioning and pain stimulation based on the limb movement pre-diagnosis strategy provided by the aforementioned embodiment, and then judge the action feedback through the action recognition model and mark and score accordingly.

[0489] Exemplarily, the eye opening reaction detection module 602, the speech reaction detection module 603 and the non-hemiplegic side movement reaction detection module 604 can be switched to the running state at the same time to simultaneously perform eye opening reaction detection, speech reaction detection and non-hemiplegic side movement reaction detection.

[0490] pass Fig. 6A It can be seen from the structure shown that the functions of each module in the coma degree determination device are independent of each other, and can respectively realize independent data collection and detection functions of corresponding items.

[0491] Table 6

[0492]

[0493] Figure 6B FIG. 1 is a schematic diagram of the structure of a monitoring device provided in the related art. Figure 6B As shown, the monitoring device 605 may include a data acquisition and preprocessing module 606 , a stimulation module 607 , a feature extraction and classification module 608 , and a consciousness assessment module 609 .

[0494] The data acquisition and preprocessing module 606 can acquire the patient's vital sign data and perform preliminary analysis on the UI vital sign data; for example, the data acquisition and preprocessing module 606 can also acquire EEG signals and perform bandpass filtering on them.

[0495] For example, the stimulation module 607 may apply stimulation to the patient during monitoring.

[0496] Exemplarily, the feature extraction and classification recognition module 608 may perform feature extraction and classification recognition on the vital sign data respectively, and perform a decision fusion operation on them.

[0497] Exemplarily, the consciousness assessment module 609 may perform a multi-parameter assessment of the state of consciousness based on the results of the behavioral scale and the resting brain state assessment, thereby determining the state of consciousness of the patient.

[0498] according to Figure 6B It can be seen that the data acquisition and preprocessing module 606, the stimulation module 607, the feature extraction and classification module 608 and the consciousness assessment module 609 cooperate with each other. Although they can assess the patient's state of consciousness, the device can only be used for monitoring after the patient is admitted to the hospital, and is not suitable for pre-hospital emergency diagnosis scenarios.

[0499] Table 6 is a description of the categories of auxiliary diagnosis robots provided in this application. It can be seen from Table 6 that various types of auxiliary diagnosis robots can automatically perform various auxiliary diagnosis and treatment actions and have obvious technical advantages. Therefore, auxiliary diagnosis robots have broad application prospects.

[0500] Figure 6C A schematic diagram of the development status of the auxiliary diagnosis robot provided in an embodiment of the present application. Figure 6C The horizontal axis of the coordinates shown is time, the unit is years, and the vertical axis is value, the unit is 100 million RMB.

[0501] from Figure 6C It can be seen that from 2014 to 2023, the scale of China's medical auxiliary diagnosis robots showed a clear trend of year-on-year growth, among which, from 2014 to 2018, the annual compound growth rate reached 27.6%, and from 2019 to 2023, the annual compound growth rate reached 22.3%. Therefore, the coma degree determination device or auxiliary diagnosis robot provided in the embodiment of the present application will have a wide range of application prospects; and the coma degree determination device provided in the embodiment of the present application can be applied to emergency scenes, which can automatically, intelligently and comprehensively collect various data of patients to be diagnosed, thereby not only improving the efficiency of emergency treatment, but also providing targeted emergency treatment.

[0502] Based on the foregoing embodiments, an embodiment of the present application further provides a computer-readable storage medium, in which a computer program is stored. When the computer program is executed by a processor of an electronic device, the method for determining the degree of coma provided in any of the previous embodiments can be implemented.

[0503] The above description of various embodiments tends to emphasize the differences between the various embodiments. The same or similar aspects can be referenced to each other, and for the sake of brevity, they will not be repeated herein.

[0504] The methods disclosed in the various method embodiments provided in this application can be arbitrarily combined without conflict to obtain new method embodiments.

[0505] The features disclosed in the various product embodiments provided in this application can be arbitrarily combined without conflict to obtain new product embodiments.

[0506] The features disclosed in the various method or device embodiments provided in this application can be arbitrarily combined without conflict to obtain new method embodiments or device embodiments.

[0507] It should be noted that the above-mentioned computer-readable storage medium can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a magnetic random access memory (FRAM), a flash memory (Flash Memory), a magnetic surface memory, an optical disc, or a compact disc read-only memory (CD-ROM) and other memories; it can also be various electronic devices including one or any combination of the above-mentioned memories, such as mobile phones, computers, tablet devices, personal digital assistants, etc.

[0508] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.

[0509] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0510] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus necessary general hardware nodes, and of course can also be implemented by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in each embodiment of the present application.

[0511] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0512] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0513] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0514] The above are only preferred embodiments of the present application, and are not intended to limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A method for determining the degree of coma, characterized in that: The method comprises: Acquiring historical health data of a patient to be diagnosed and the patient's intake status of at least one designated substance; Determining a pre-diagnosis strategy for the patient to be diagnosed based on the historical health data and the intake status; Processing the collected data for the patient to be diagnosed based on the pre-diagnosis strategy to obtain a processing result; wherein the collected data is at least associated with a body part included in the pre-diagnosis strategy; The coma degree of the patient to be diagnosed is determined based on the processing result.

2. The method according to claim 1, characterized in that The step of determining a pre-diagnosis strategy for the patient to be diagnosed based on the historical health data and the intake status includes: Determining the age information of the patient to be diagnosed; Determining a target weight associated with target data; wherein the target data includes at least one of the age information of the patient to be diagnosed, the historical health data, and the intake status; The prognostic strategy is determined based on the target weight and the target data.

3. The method according to claim 1, characterized in that The step of determining a pre-diagnosis strategy for the patient to be diagnosed based on the historical health data and the intake status includes: Analyze the historical health data and the intake status to determine the health status of the patient to be diagnosed; Based on the health level, the prognostic strategy is determined from a plurality of strategies included in a strategy set; wherein the strategies in the strategy set are associated with the health level.

4. The method according to claim 1, characterized in that: The collected data includes eye video data of the patient to be diagnosed; the processing result includes eye movement data; the pre-diagnosis strategy includes an eye pre-diagnosis strategy; the collected data for the patient to be diagnosed is processed based on the pre-diagnosis strategy to obtain the processing result, including: If the eye video data indicates that the eye of the patient to be diagnosed does not contain an open wound, processing the eye video data based on the eye pre-diagnosis strategy to obtain the eye motion data; If the eye video data indicates that the eye of the patient to be diagnosed contains the open wound, first information is output to prompt medical personnel to treat the open wound; wherein the first information at least includes the location information of the patient to be diagnosed and the information that the eye contains the open wound.

5. The method according to claim 4, characterized in that The eye pre-diagnosis strategy includes an eye recognition model; and processing the eye video data based on the eye pre-diagnosis strategy to obtain the eye movement data includes: Segmenting the eye video data to obtain at least one frame of eye image; Determining eye edge features based on the at least one frame of eye image; Determining a set of primitive attributes of the eye; The eye edge features and the primitive attribute set are processed by the eye recognition model to obtain the eye movement data.

6. The method according to claim 5, characterized in that The method further comprises: Acquire eye sample data; wherein the eye sample data is at least associated with a first aid scene; the eye sample data at least includes an eye image containing stains and an eye image that is covered; the stains at least include blood stains and dust; The eye recognition model in an initial state is trained based on the eye sample data to obtain the eye recognition model.

7. The method according to claim 4, characterized in that The eye pre-diagnosis strategy at least includes stimulation parameters for the patient to be diagnosed; the method further includes: In the process of outputting the audio data included in the stimulation parameters, collecting the eye video data, and / or, In the process of performing supraorbital nerve compression based on the eye compression parameters included in the stimulation parameters, the expression video data and the eye video data of the patient to be diagnosed are collected.

8. The method according to claim 4, characterized in that The step of processing the eye video data based on the eye pre-diagnosis strategy to obtain the eye motion data includes: If the eye video data indicates that the eye is covered by an occluding object, identifying the occluding object, removing the occluding object from the eye video data, and obtaining a first removed image; The first removal image is processed based on the eye pre-diagnosis strategy to obtain the eye motion data.

9. The method according to claim 1, characterized in that: The collected data includes voice data output by the patient to be diagnosed; the processing result includes a voice recognition result for the voice data; the pre-diagnosis strategy includes a voice pre-diagnosis strategy; The processing of the collected data for the patient to be diagnosed based on the pre-diagnosis strategy to obtain a processing result includes: Output the first question list; collecting the voice data output by the patient to be diagnosed when answering at least one question in the first question list; The voice data is processed based on the voice pre-diagnosis strategy to obtain the voice recognition result.

10. The method according to claim 9, characterized in that The outputting of the first question list includes: Determining the language expression ability of the patient to be diagnosed based on the historical health data; If the language expression ability indicates that the patient to be diagnosed is not in a state of language expression disorder, the first question list is output.

11. The method according to claim 9, characterized in that The outputting of the first question list includes: If it is detected that the auditory organ of the patient to be diagnosed is in a damaged state, the coma degree determination device is controlled to move to the side of the auditory organ where the damage degree is less than the damage threshold, and the first question list is output.

12. The method according to claim 9, characterized in that The outputting of the first question list includes: Identify whether a designated part of the patient to be diagnosed contains an open wound; wherein the designated part is at least associated with a vocal organ of the patient to be diagnosed; If the designated part does not include the open wound, outputting the first question list; The method further comprises: If the designated part includes the open wound, second information is output to prompt medical personnel to treat the open wound; wherein the second information includes information that the designated part includes the open wound and location information of the patient to be diagnosed.

13. The method according to claim 9, characterized in that The outputting of the first question list includes: Identify the facial video data containing the facial parts of the patient to be diagnosed, and obtain the facial coverage status of the patient to be diagnosed; If the facial features coverage status indicates that the facial features of the patient to be diagnosed are not covered, outputting the first question list; The method further comprises: If the facial features coverage state indicates that the facial features of the patient to be diagnosed are in a covered state, the facial features video data is processed by an image processing algorithm to remove covering objects covering the facial features, thereby obtaining a second removed image; If the covering object in the second removal image at least partially covers the five sense organs, third information is output to prompt medical personnel to deal with the covering object; wherein the third information includes information that the covering object covers the five sense organs, and location information of the patient to be diagnosed.

14. The method according to claim 9, characterized in that The speech pre-diagnosis strategy includes a speech recognition model; and processing the speech data based on the speech pre-diagnosis strategy to obtain the speech recognition result includes: Determining the clarity of the voice data; Processing the speech data through the speech recognition model to determine a degree of match between the speech data and questions in the first question list; The speech recognition result is determined based on the clarity and the matching degree.

15. The method according to claim 1, characterized in that The pre-diagnosis strategy includes a limb movement pre-diagnosis strategy; the collected data includes movement video data of the limb parts of the patient to be diagnosed; the processing result includes a movement recognition result; the collected data for the patient to be diagnosed is processed based on the pre-diagnosis strategy to obtain the processing result, including: If at least one of the limb parts is not in a state of injury of a specified degree, outputting a second list of questions; collecting the action performed by the patient to be diagnosed in controlling the at least one part in response to at least one question in the second question list to obtain the action video data; The action video data is processed based on the action pre-diagnosis strategy to obtain the action recognition result.

16. The method according to claim 15, characterized in that At least one question in the second question list includes a question about finger movements; collecting the movement of the patient to be diagnosed to control the at least one part in response to at least one question in the second question list to obtain the movement video data includes: collecting hand movements performed by the patient to be diagnosed for the finger movement problem to obtain hand movement data; If the hand motion data indicates that the at least one finger does not perform the specified finger motion, performing a pressure stimulation operation on at least one body part of the patient to be diagnosed; During the execution of the pressure stimulation operation, collecting the reaction action of the at least one body part to obtain stimulation action data; The hand motion data or the stimulation motion data is determined to be the motion video data.

17. The method according to claim 15, characterized in that The outputting of the second question list includes: Identify whether the limb part is in a specified state; wherein the specified state includes at least one of the limb part having an open wound, being heavily compressed, and being swollen; If the limb part is not in the specified state, outputting the second question list; The method further comprises: If the limb part is in the specified state, fourth information indicating that the limb part is in the specified state is output to prompt medical personnel to treat the limb part in the specified state; wherein the fourth information at least includes the position information of the patient to be diagnosed.

18. The method according to claim 15, characterized in that The action video data includes depth video data containing the action; the limb action pre-diagnosis strategy includes an action recognition model; the action video data is processed based on the action pre-diagnosis strategy to obtain the action recognition result; including: Segmenting the depth video data to obtain a depth image; Identify the depth image to obtain the bone node position of the limb part; The skeletal node positions are identified by the action recognition model to obtain the action recognition result; wherein the action recognition model at least includes a convolutional neural network and a long short-term memory network arranged in cascade.

19. The method according to claim 18, characterized in that The step of identifying the position of the skeleton node by using the action recognition model to obtain the action recognition result includes: Determining the number of bone nodes associated with the limb part; determining a duration of the at least one part performing the action; Based on the duration, determining a target number of frames for the at least one part to perform the action; Integrate the number of bone nodes, the positions of the bone nodes associated with the number of bone nodes, and the target frame number to obtain input data; The action recognition model is used to extract features from the input data to obtain the action recognition result.

20. A computer-readable storage medium, characterized in that: The storage medium stores a computer program; when the computer program is executed by a processor of an electronic device, the method for determining the degree of coma as described in any one of claims 1 to 19 can be implemented.

Images