An automatic evaluation method for sedation level and related products

Abstract

The present application discloses an automatic evaluation method for sedation level and related products. The method includes: acquiring image data and optical flow field data of a target object in the same time period; obtaining a first sedation level evaluation result for the target object by processing the image data; and obtaining a second sedation level evaluation result for the target object by processing the optical flow field data; obtaining a comprehensive sedation level evaluation result for the target object based on the first sedation level evaluation result and the second sedation level evaluation result. It realizes the intelligent evaluation and real-time monitoring of the sedation level of personnel, and can be completely intelligent or only use a small amount of medical staff in the judgment of the sedation level, thus making an important contribution to the realization of intelligent intensive care.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and particularly to a method for automatically evaluating sedation level and related products. Background Art

[0002] With the development of modern medicine, the Intensive Care Unit (ICU) has become a very important department in hospital clinical practice. Improving the informatization level of the ICU has become an indispensable part of hospital informatization construction. The patients monitored in the ICU department are generally critically ill patients, and it is required that the medical staff in the department be equipped with strong professional theoretical knowledge and practical clinical working ability.

[0003] Since the condition of ICU patients is critical and the rescue work is time-sensitive, it is necessary for ICU medical staff to be able to immediately understand when the patient's condition changes in order to handle it in a timely manner and not miss the best opportunity to rescue the patient. The pressure on medical staff to complete relevant nursing work is very high, and there may also be misjudgments or incorrect judgments due to fatigue, busyness, etc., which ultimately affect the care of the patient.

[0004] Sedation level assessment is a very important task in the intensive care process. This task generally requires medical staff to constantly monitor the patient, which increases the burden on medical staff. At the same time, it is inevitable that the sedation level of patients cannot be evaluated at all times in an artificial manner. Therefore, it may not be possible to detect the patient's abnormalities in a timely manner, and even delay the timing of the patient's diagnosis and treatment. Summary of the Invention

[0005] Based on the above problems, the present application provides a method for automatically evaluating sedation level and related products, aiming to solve the problems of excessive burden and untimely evaluation of medical staff in manually evaluating the sedation level of patients in the ICU.

[0006] The embodiments of the present application disclose the following technical solutions:

[0007] The first aspect of the present application provides a method for automatically evaluating sedation level, including:

[0008] Obtaining image data and optical flow field data of a target object in the same time period;

[0009] Obtaining a first sedation level evaluation result of the target object by processing the image data; and obtaining a second sedation level evaluation result of the target object by processing the optical flow field data;

[0010] Obtaining a comprehensive sedation level evaluation result of the target object based on the first sedation level evaluation result and the second sedation level evaluation result.

[0011] In an alternative implementation, the comprehensive evaluation result of the sedation level is a first type of evaluation result or a second type of evaluation result, where the sedation level indicated by the first type of evaluation result is higher than the sedation level indicated by the second type of evaluation result;

[0012] After obtaining the comprehensive evaluation result of the sedation level of the target object, the method further includes:

[0013] According to the cumulative number of times that the evaluation result of the sedation level is the second type of evaluation result from the initial counting moment to the current moment, determine whether to generate a prompt signal regarding the sedation level of the target object.

[0014] In an alternative implementation, the determining whether to generate a prompt signal regarding the sedation level of the target object according to the cumulative number of times that the evaluation result of the sedation level is the second type of evaluation result from the initial counting moment to the current moment includes:

[0015] When the ratio of the cumulative number of times to the number of times of comprehensively evaluating the sedation level of the target object from the initial counting moment to the current moment exceeds a preset threshold, generate a prompt signal regarding the sedation level of the target object.

[0016] In an alternative implementation, the method further includes:

[0017] Establish a data set; the data set includes multiple groups of training samples; the training samples include image training data, optical flow field training data, and sedation level labels;

[0018] Train a first evaluation model with the image training data and sedation level labels in the training samples, and train a second evaluation model with the optical flow field training data and sedation level labels in the training samples;

[0019] Based on the output result of the first evaluation model, the output result of the second evaluation model, and the sedation level label, train the fusion coefficient for fusing the first evaluation model and the second evaluation model;

[0020] Evaluate the training effects of the first evaluation model and the second evaluation model, and stop training when the training effect evaluation is qualified to obtain a trained first evaluation model, a trained second evaluation model, and a trained fusion coefficient;

[0021] The trained first evaluation model is used to process the image data to obtain a first sedation level evaluation result of the target object, the trained second evaluation model is used to process the optical flow field data to obtain a second sedation level evaluation result of the target object, and the trained fusion coefficient is used to fuse the first sedation level evaluation result and the second sedation level evaluation result to obtain the comprehensive sedation level evaluation result.

[0022] In an alternative implementation, training the fusion coefficient for fusing the first evaluation model and the second evaluation model based on the output result of the first evaluation model, the output result of the second evaluation model, and the sedation level label specifically includes:

[0023] Fusing the output result of the first evaluation model and the output result of the second evaluation model using the current fusion coefficient to obtain a fused sedation level prediction result corresponding to the training sample;

[0024] Comparing the fused sedation level prediction result with the sedation level label in the training sample to obtain cross-entropy as the sedation level evaluation error;

[0025] Backpropagating the sedation level evaluation error to optimize the current fusion coefficient and the parameters of the first evaluation model and the second evaluation model.

[0026] In an alternative implementation, obtaining the image data and the optical flow field data of the target object in the same time period specifically includes:

[0027] Obtaining video data of the target object during the time period;

[0028] Separating the image data and the optical flow field data from the video data.

[0029] A second aspect of the present application provides an automatic evaluation device for sedation level, including:

[0030] An acquisition module, configured to acquire image data and optical flow field data of a target object in the same time period;

[0031] An evaluation module, configured to obtain a first sedation level evaluation result of the target object by processing the image data; and, configured to obtain a second sedation level evaluation result of the target object by processing the optical flow field data;

[0032] A fusion module, configured to obtain a comprehensive sedation level evaluation result of the target object based on the first sedation level evaluation result and the second sedation level evaluation result.

[0033] In an alternative implementation, the comprehensive evaluation result of the sedation level is a first type of evaluation result or a second type of evaluation result, where the sedation level indicated by the first type of evaluation result is higher than the sedation level indicated by the second type of evaluation result;

[0034] The device further includes:

[0035] A prompt module, configured to determine whether to generate a prompt signal regarding the sedation level of the target object according to the cumulative number of times that the evaluation result of the sedation level is the second type of evaluation result from the initial counting moment to the current moment.

[0036] In an alternative implementation, the prompt module is specifically configured to generate a prompt signal regarding the sedation level of the target object when the ratio of the cumulative number of times to the number of times of cumulatively evaluating the sedation level of the target object from the initial counting moment to the current moment exceeds a preset threshold.

[0037] In an alternative implementation, the device further includes:

[0038] A dataset establishment module, configured to establish a dataset; the dataset includes multiple groups of training samples; the training samples include image training data, optical flow field training data, and sedation level labels;

[0039] A model training module, configured to train a first evaluation model through the image training data and sedation level labels in the training samples, and train a second evaluation model through the optical flow field training data and sedation level labels in the training samples; based on the output results of the first evaluation model, the output results of the second evaluation model, and the sedation level labels, train the fusion coefficient for fusing the first evaluation model and the second evaluation model; evaluate the training effects of the first evaluation model and the second evaluation model, and stop training when the training effect evaluation is qualified to obtain a trained first evaluation model, a trained second evaluation model, and a trained fusion coefficient;

[0040] Wherein the trained first evaluation model is configured to process the image data to obtain a first sedation level evaluation result of the target object, the trained second evaluation model is configured to process the optical flow field data to obtain a second sedation level evaluation result of the target object, and the trained fusion coefficient is configured to fuse the first sedation level evaluation result and the second sedation level evaluation result to obtain the comprehensive evaluation result of the sedation level.

[0041] In an alternative implementation, the model training module specifically includes:

[0042] A fusion prediction unit, configured to fuse the output results of the first evaluation model and the second evaluation model by using a current fusion coefficient, so as to obtain a fusion prediction result of the sedation level corresponding to the training sample;

[0043] An error evaluation unit, configured to compare the fusion prediction result of the sedation level with the sedation level label in the training sample, so as to obtain cross entropy as an evaluation error of the sedation level;

[0044] A parameter optimization unit, configured to back-propagate the evaluation error of the sedation level, so as to optimize the current fusion coefficient and the parameters of the first evaluation model and the second evaluation model.

[0045] In an optional implementation manner, the obtaining module specifically includes:

[0046] A video obtaining unit, configured to obtain video data of the target object during the time period;

[0047] A data separation unit, configured to separate the image data and the optical flow field data from the video data.

[0048] A third aspect of the present application provides a computer-readable storage medium, on which a computer program is stored, and when the program is executed by a processor, the steps of any of the automatic evaluation methods of the sedation level in the first aspect are implemented.

[0049] A fourth aspect of the present application provides an electronic device, including:

[0050] A memory, on which a computer program is stored;

[0051] A processor, configured to execute the computer program in the memory to implement the steps of any of the automatic evaluation methods of the sedation level in the first aspect.

[0052] Compared with the prior art, the present application has the following beneficial effects:

[0053] The automatic evaluation method and related products for the sedation level provided by this application first obtain the image data and optical flow field data of the target object at the same time period; then obtain the first sedation level evaluation result of the target object by processing the image data, and obtain the second sedation level evaluation result of the target object by processing the optical flow field data; finally, based on the first sedation level evaluation result and the second sedation level evaluation result, obtain the comprehensive sedation level evaluation result of the target object. Since the sedation level is evaluated separately through image data and optical flow field data, and finally a comprehensive evaluation result is obtained based on the sedation levels evaluated through two methods of image recognition and behavior recognition, the accuracy of the sedation level evaluation result obtained through one method can be corrected. Therefore, the embodiments of this application can be realized automatically, which can reduce the workload of medical staff and improve the timeliness of sedation level evaluation. While realizing automation, the accuracy of the sedation level evaluation result can also be improved, making the final evaluation result more reliable and timely. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] In order to more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the following will briefly introduce the drawings required for the description of the embodiments or the prior art. Obviously, the drawings in the following description are only some embodiments of this application. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings.

[0055] Figure 1 It is a flowchart of an automatic evaluation method for the sedation level provided by an embodiment of this application;

[0056] Figure 2 It is a flowchart of another automatic evaluation method for the sedation level provided by an embodiment of this application;

[0057] Figure 3 It is a schematic diagram of training, testing and applying the first evaluation model provided by an embodiment of this application;

[0058] Figure 4 It is a schematic diagram of fusing and training two evaluation models provided by an embodiment of this application;

[0059] Figure 5 It is a flowchart of another automatic evaluation method for the sedation level provided by an embodiment of this application;

[0060] Figure 6 It is a schematic structural diagram of an automatic evaluation device for the sedation level provided by an embodiment of this application;

[0061] Figure 7 It is a hardware structure diagram of an automatic evaluation device for the sedation level provided by an embodiment of this application. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0062] Sedation level assessment is a very important task during the intensive care process. This task generally requires medical staff to constantly monitor patients, which increases the burden on medical staff. At the same time, it is inevitable that sedation level assessment of patients in an artificial form cannot be carried out all the time. Therefore, it may not be possible to detect patients' abnormalities in a timely manner, and even delay the timing of diagnosis and treatment of patients. There may also be misjudgments or incorrect judgments due to problems such as fatigue and busyness, which ultimately have an adverse impact on the care and treatment of patients.

[0063] In view of this problem, an automatic assessment method for sedation level and related products are proposed in this application. The sedation level of the target object is evaluated by means of image recognition and behavior recognition respectively, and the results evaluated by the two methods are comprehensively considered to finally obtain a sedation level assessment result with higher accuracy. This solution can also reduce the work burden of medical staff and realize real-time sedation state assessment.

[0064] The technical solution of this application will be described below in conjunction with embodiments and drawings.

[0065] Figure 1 It is a flowchart of an automatic assessment method for sedation level provided by an embodiment of this application. As Figure 1 shown, the automatic assessment method for sedation level includes:

[0066] Step 101, obtain the image data and optical flow field data of the target object in the same time period.

[0067] In the embodiment of this application, the target object refers to the object whose sedation level needs to be evaluated. Specifically, in implementation, the target object can be any object whose sedation level needs to be evaluated in any scenario, such as critically ill patients monitored in the hospital ICU. It should be noted that in some scenarios, to evaluate the sedation level of the target object, permission or consent from the target object or the guardian of the target object is required. And without the permission or consent of the target object or the guardian of the target object, the data collected during the assessment process will not be used for purposes other than evaluating its sedation level. If the solution is used in a medical scenario, it can also be defaulted that in scenarios such as the ICU, the collection and acquisition of the above data are for the sake of the life and health of the target object and are necessary to be collected and acquired. However, the use of the data collected in the medical scenario needs to be restricted by relevant units or personnel to ensure the reasonableness and legality of data use. In an alternative implementation, video data of the target object in a certain time period can be collected first, and then the image data and optical flow field data are separated from it. In other implementations, the above process can also be carried out in real time.

[0068] Image data refers to the static data inherent in the image itself and is two-dimensional data. Optical flow field data refers to the displacement relationship of the target object between two frames with a very short time interval. The optical flow field refers to a two-dimensional instantaneous velocity field composed of all pixel points in the image, where the two-dimensional velocity vector is the projection of the three-dimensional velocity vector of the visible points in the scene onto the imaging surface. In space, motion can be described by a motion field, while on an image plane, the motion of an object is often reflected by the differences in the gray-scale distributions of different images in an image sequence. Thus, when the motion field in space is transferred to an image, it is represented as an optical flow field. The optical flow field is a two-dimensional vector field that reflects the change trend of the gray scale at each point in the image and can be regarded as an instantaneous velocity field generated by the movement of pixel points with gray scale on the image plane. The information it contains is the instantaneous motion velocity vector information of each image point.

[0069] The image data at least includes eye images, and the optical flow field data may include the optical flow field of the whole body. The eye images show the static eye effect of the target object at a certain moment during a certain period; the optical flow field of the whole body refers to the displacement relationship of the whole body of the target object between two frames with a very short time interval during this period, thereby reflecting the displacement of the whole body's actions and facilitating the recognition of actions. In other implementation manners, the image data may also involve parts such as eyebrows and lips in the face. This can improve the accuracy of the evaluated sedation level.

[0070] In specific implementation, video data can be collected through a video acquisition device, such as a surveillance camera installed in the ICU. The separation of the image data and the optical flow field data can be achieved by a device with video processing functions and information extraction functions. On the one hand, it needs to sample at regular intervals and sample the video into images. On the other hand, it needs to form time-series data from the collected images in chronological order and convert them into a column of photos arranged in chronological order to obtain the optical flow field data.

[0071] For example, obtain the video data within the last 3 minutes up to the current time and separate it. This time period refers to the time period from 3 minutes ago to the current moment. The time period and the length of the time period can both be set according to actual needs and are not limited here.

[0072] Step 102, obtain a first sedation level evaluation result for the target object by processing the image data; and, obtain a second sedation level evaluation result for the target object by processing the optical flow field data.

[0073] In practical applications, an image recognition algorithm (such as the yolov5 algorithm) can be used to analyze and evaluate multiple frames of images within a period of time to obtain the evaluation result of the sedation level of the target object. The open / closed state of the patient's eyes in the image is evaluated. Since critically ill patients are usually in a sleeping state under sedation, their eyes are mainly closed at this time; when critically ill patients are in a non-sedated state, their eyes are open. Therefore, the sedation level of the patient can be preliminarily judged based on the open / closed state of the eyes. For example, if there are more frames with open eyes in the image, it is determined that the sedation level is lower; conversely, if there are more frames with closed eyes in the image, it is determined that the sedation level is higher. The higher the sedation level means getting closer to the sedated state, and the lower the sedation level means getting closer to the non-sedated state.

[0074] In addition, the evaluation result of the sedation level of the target object can also be obtained through an action evaluation algorithm (such as the iDT algorithm) for optical flow field data. For example, if the optical flow field data can reflect that the target object frequently raises its hand, it is determined that the sedation level is lower; if the optical flow field data reflects that the target object raises its hand less frequently, it is determined that the sedation level is higher. Of course, raising the hand is only one example of an identified action, and other types of actions can also be identified, such as shaking the head, raising the leg, etc. To facilitate the distinction between the sedation level evaluation results obtained by the two methods (i.e., image recognition and behavior recognition), in the embodiments of the present application, the sedation level evaluation result obtained based on image data and the sedation level evaluation result obtained based on optical flow field data are respectively named the first sedation level evaluation result and the second sedation level evaluation result.

[0075] The above is only an example method of step 102. In practical applications, it can also be done in other ways. For example, evaluation can be carried out through a pre-trained evaluation model. This implementation method will be specifically described later. It will not be elaborated here for the time being.

[0076] Step 103: Based on the first sedation level evaluation result and the second sedation level evaluation result, obtain the comprehensive evaluation result of the sedation level of the target object.

[0077] In practical applications, due to reasons such as the camera angle and the size of the human eye, the first sedation level assessment result obtained by image recognition may have loopholes in accuracy and / or assessment precision. For example, the target object is in an open-eye state, but due to the angle of the camera or the small eyes of the target object, the open-eye state is detected as a closed-eye state, resulting in an error in the final assessment result. After investigation, it was found that patients with open eyes often have more small movements, while patients in deep sleep have fewer small movements. Therefore, the second sedation level assessment result obtained by motion assessment can be used to correct the first sedation level assessment result. In step 103, the first sedation level assessment result and the second sedation level assessment result can be fused to obtain a comprehensive assessment result of the sedation level of the target object. Compared with the first sedation level assessment result, since the comprehensive sedation level assessment result is obtained by combining the results of the two methods, the precision and / or accuracy are improved.

[0078] The above is an automatic evaluation method for sedation level provided in an embodiment of the present application. Since the sedation level is evaluated by image data and optical flow field data respectively, and finally a comprehensive evaluation result is obtained based on the sedation level evaluated by two approaches of image recognition and behavior recognition, the accuracy of the sedation level evaluation result obtained by one approach can be corrected. The embodiment of the present application can be implemented automatically, which can reduce the workload of medical staff and improve the timeliness of sedation level evaluation. While being implemented automatically, the accuracy of the sedation level evaluation results can also be improved, making the final evaluation results more reliable and timely.

[0079] The present application also provides a method for automatically evaluating sedation level through a model. Figure 2 A flowchart of another automatic evaluation method for sedation level provided in an embodiment of the present application. Figure 2 As shown, the training process of the model is first introduced through steps 201-204.

[0080] Step 201: Establish a data set, wherein the data set includes multiple groups of training samples, and the training samples include image training data, optical flow field training data, and sedation level labels.

[0081] The image training data and the optical flow field training data are respectively of the same nature as the image data and the optical flow field data introduced in the aforementioned embodiments. That is to say, the image training data and the optical flow field training data are also different types of data obtained in the same time period. In a specific implementation, the image training data and the optical flow field training data can be extracted from videos of patients collected historically. The sedation level label can be the sedation level manually assessed by experienced medical staff for the patients to whom the image training data and the optical flow field training data belong, for the time period to which these data belong. During the training process, the sedation level label is used as the true value to guide the model training.

[0082] Step 202: Train a first evaluation model using the image training data and sedation level labels in the training samples, and train a second evaluation model using the optical flow field training data and sedation level labels in the training samples.

[0083] In this step, the first evaluation model refers to a model that needs to output an evaluation result of the sedation level based on image data, and the second evaluation model refers to a model that needs to output an evaluation result of the sedation level based on optical flow field data. Both the first evaluation model and the second evaluation model need to be trained.

[0084] In an optional implementation, use the image training data and sedation level labels in the training samples as a set of data corresponding to training the first evaluation model; use the optical flow field training data and sedation level evaluation model in the training samples as a set of data corresponding to training the second evaluation model. The training of the first evaluation model and the second evaluation model can be executed sequentially or synchronously.

[0085] Figure 3 FIG. is a schematic diagram of training, testing, and applying a first evaluation model provided by an embodiment of the present application. In this example implementation, a dataset of the open / closed eye states of critically ill patients is constructed, and the dataset is divided into a training set, a validation set, and a test set in a ratio of 6:1:3. That is to say, in the dataset constructed here, 60% of the data is used to train the model, 10% of the data is used to validate the model, and 30% of the data is used to test the model. In this example implementation, the training of the model refers to using the training set and the validation set to train the evaluation model, and the testing of the model refers to using the test set to test the accuracy and robustness of the obtained model; the application of the model refers to using the model after testing to predict new data.

[0086] As Figure 3 shown, several terms are involved in the training stage of the model: Dropout, BN, cross-validation, learning rate self-decay, etc. Dropout refers to the ability to use the Dropout strategy during model training to prevent overfitting of the model. BN refers to using the BN strategy to prevent the problem of gradient disappearance during the process of backpropagation of the model gradient. Cross-validation refers to the validation strategy adopted during training; learning self-decay refers to the learning strategy adopted during the training process of the first evaluation model. As Figure 3 shown, several terms are involved in the testing stage of the model: recall rate, precision rate, multi-class average precision, IOU (Intersection over Union), etc. The recall rate, precision rate, multi-class average precision, and IOU value can all be used as evaluation indicators for the training effect of the first evaluation model. As Figure 3 shown, several terms such as accuracy and robustness are involved in the application stage of the model. The accuracy and robustness reflect the performance of the model during actual use.

[0087] Similarly to the training of the first evaluation model, the dataset for training the second evaluation model can be divided and operations in stages such as training and testing can be carried out with reference to the Figure 3 process shown. And a suitable strategy is set in the training stage of the second evaluation model, and suitable evaluation metrics are set in the testing stage.

[0088] It should be noted that in the first evaluation model and the second evaluation model during training, the algorithm used to finally judge the sedation level of the target object is the softmax algorithm. For easy understanding, it is represented by a vector below. The vector where a1 and a2 represent two categories respectively. For example, a1 and a2 are two different sedation levels. For example, a1 indicates sedation and a2 indicates non-sedation. It can be understood that the evaluation process of the entire model can also be understood as a classification task, and the number of elements in the vector is equal to the number of categories in the classification task. The vector represents the feature vector obtained after passing through the fully connected layer of the model. In the following formula, p(class i ) represents the probability that the evaluation result is the i-th classification among multiple classifications:

[0089]

[0090] In the above example, i takes values between 1 and 2. c represents the total number of categories. It can be seen from the formula that after passing through softmax, a new vector will be obtained, where b1 and b2 are real numbers greater than or equal to 0 and less than or equal to 1, and satisfy b1 + b2 = 1. In the classification process, the category corresponding to the largest value is selected as the classification result.

[0091] Step 203: Train the fusion coefficient for fusing the first evaluation model and the second evaluation model based on the output result of the first evaluation model, the output result of the second evaluation model, and the sedation level label.

[0092] However, whether it is action evaluation or eyes-open / closed state evaluation, there are certain errors, and even patients in a deep sleep may have action states. Therefore, it is unreasonable to directly deny the image evaluation result using the action evaluation result. For this reason, in this application, a method for fusing the first evaluation model and the second evaluation model is designed to train a model with better evaluation performance. The final output result solves the technical problems existing in the prior art. The evaluation results of the models are fused using the fusion coefficient. See the following formula:

[0093] f(x) = αf1(x) + (1 - α)f2(x)

[0094] Among them, f(x) represents the fused prediction result of the sedation level, f1(x) represents the output result of the first evaluation module, and f2(x) represents the output result of the second evaluation model. In this algorithm, if f1(x) = 0 and f2(x) = 0, then f(x) = 0; if f1(x) = 0 and f2(x) = 1, then f(x) = (1 - α)f2(x). If f1(x) = 1 and f2(x) = 0, then f(x) = αf1(x). If f1(x) = 1 and f2(x) = 1, then f(x) = 1. As can be seen from the above, this algorithm satisfies the endpoint values. At the same time, the value range of f(x) is [0, 1].

[0095] α represents the fusion coefficient (which can be regarded as a weight greater than 0 and less than 1), and the value of this coefficient can be determined during training. In step 203 of the embodiment of the present application, joint training needs to be performed on two models to determine the final value of α.

[0096] The following introduces an example implementation manner of this step: Using the current fusion coefficient to fuse the output results of the first evaluation model and the second evaluation model to obtain the fused prediction result of the sedation level corresponding to the training sample; comparing the fused prediction result of the sedation level with the sedation level label in the training sample to obtain the cross-entropy as the sedation level evaluation error; backpropagating the sedation level evaluation error to optimize the current fusion coefficient and the parameters of the first evaluation model and the second evaluation model. Figure 4 It is a schematic diagram for fusing and training two evaluation models. The evaluation results of the first evaluation model and the second evaluation model are propagated forward to the final fusion stage. The feedback of the fusion is used to optimize the parameters of the two models. Figure 4 In [the above], the cross-entropy refers to that the error calculation is the cross-entropy loss; BP refers to the error backpropagation algorithm, and parameter optimization refers to the optimizer's search for optimization.

[0097] Step 204: Evaluate the training effects of the first evaluation model and the second evaluation model. When the training effect evaluation is qualified, stop training to obtain the trained first evaluation model, the trained second evaluation model, and the trained fusion coefficient.

[0098] Among them, the trained first evaluation model is used to process the image data to obtain the first sedation level evaluation result of the target object, the trained second evaluation model is used to process the optical flow field data to obtain the second sedation level evaluation result of the target object, and the trained fusion coefficient is used to fuse the first sedation level evaluation result and the second sedation level evaluation result to obtain the comprehensive sedation level evaluation result.

[0099] Step 205: Obtain the image data and optical flow field data of the target object in the same time period.

[0100] Here, the image data and optical flow field data can be the data separated from the real-time collected patient video data. These data are used as the input for actually applying the above-trained model.

[0101] Step 206: Process the image data through the trained first evaluation model to obtain the first sedation level evaluation result of the target object; and, process the optical flow field data through the trained second evaluation model to obtain the second sedation level evaluation result of the target object.

[0102] Since the first evaluation model and the second evaluation model have been trained previously, that is to say, their training effects have been fully verified and tested. At this time, after obtaining the image data and optical flow field data, they can be respectively input into the trained first evaluation model and the second evaluation model and evaluated separately.

[0103] Step 207: Fuse the first sedation level evaluation result and the second sedation level evaluation result through the trained fusion coefficient to obtain the comprehensive sedation level evaluation result.

[0104] Whether it is action recognition or image recognition, there are certain errors, and even patients in a deep sleep may have action states. Therefore, it is unreasonable to directly deny the image evaluation result using the action evaluation result. By fusing the two evaluation results through the fusion coefficient, the problem that the sedation level evaluation is inaccurate due to the camera angle or the size of the human eye in one evaluation method can be effectively solved, as well as the problem that the evaluation accuracy is poor due to too single consideration factors. Considering the two evaluation results comprehensively can effectively improve the accuracy and precision of the sedation level evaluation.

[0105] In practical applications, the target object may enter the sleep state again after a short period of wakefulness. In the short wakeful state, at this moment, the evaluation results of the image data and the optical flow field data are both non-sedated states, but if a sedation level prompt is given at this time, it is obviously easy to frequently give incorrect guidance to medical staff. To solve this problem, the present application also provides another automatic evaluation method for the sedation level, see Figure 5 the flowchart shown.

[0106] Step 501: Obtain the image data and optical flow field data of the target object in the same time period.

[0107] Step 502: Obtain the first sedation level evaluation result of the target object by processing the image data; and, obtain the second sedation level evaluation result of the target object by processing the optical flow field data.

[0108] Step 503: Obtain a comprehensive sedation level assessment result for the target object based on the first sedation level assessment result and the second sedation level assessment result.

[0109] The implementation manners of steps 501 - 503 are basically the same as those of steps 101 - 103 in the foregoing embodiments. The relevant descriptions can refer to the embodiments introduced above and will not be elaborated here. It should be noted that the above assessment process is continuous, and a comprehensive sedation level assessment result can be obtained each time step 103 is executed. The time windows for each execution of the assessment can overlap in time or be independent of each other. The assessment is performed according to a preset frequency or period to improve the real-time performance of sedation level assessment. It can be understood that even if the above steps 501 - 503 are evaluated according to a preset frequency or period, if the quality of the collected raw data is poor, or no effective human eyes or body parts are evaluated therein, it may be impossible to evaluate the sedation level.

[0110] In an example implementation manner, the comprehensive sedation level assessment result is a first type of assessment result or a second type of assessment result, where the sedation level indicated by the first type of assessment result is higher than the sedation level indicated by the second type of assessment result.

[0111] After obtaining the comprehensive sedation level assessment result for the target object, the automatic sedation level assessment method provided by the embodiments of the present application further includes:

[0112] Step 504: Determine whether to generate a prompt signal regarding the sedation level of the target object according to the cumulative number of times that the sedation level assessment result is the second type of assessment result from the initial counting moment to the current moment.

[0113] In the embodiments of the present application, an initial counting moment can be set. For example, it is necessary to start paying attention to the sedation level of the target object at 15:00 on the same day and provide timely assistance when the sedation level is poor. Therefore, 15:00 can be set as the initial counting moment. That is, the initial counting moment represents the starting time for paying attention to the sedation level of the target object within a certain time interval.

[0114] Once the assessment result is the second type of assessment result, it indicates that the sedation level of the target object is poor during this execution of the assessment. However, the second type of assessment results obtained in individual cases may be due to the short-term wakefulness of the target object after sleep. It is impossible to determine whether attention is needed. Therefore, in this step, the number of times that the sedation level assessment result is the second type of assessment result from the initial counting moment to the current moment is accumulated. Then, according to the cumulative number of times, it is determined whether to generate a prompt signal for the sedation level of the target object.

[0115] The initial counting moment can be set according to actual monitoring and evaluation requirements, or the count value at the initial counting moment can be reset to zero.

[0116] In an alternative implementation, this step can be implemented as follows:

[0117] When the ratio of the cumulative number of times to the number of times the sedation level of the target object is evaluated cumulatively from the initial counting moment to the current moment exceeds a preset threshold, a prompt signal regarding the sedation level of the target object is generated.

[0118] In this implementation, the ratio of the cumulative number of times to the number of times the sedation level of the target object is evaluated cumulatively from the initial counting moment to the current moment is used as the main basis for generating the prompt signal for the sedation level. A preset threshold is set here. If the ratio exceeds this threshold, it means that the proportion of poor sedation levels evaluated is relatively large, and medical staff need to be prompted to pay attention to the situation of this target object. The following inequality represents this implementation:

[0119]

[0120] In the above inequality, t represents the current moment, t0 represents the initial counting moment, represents the cumulative number of times the evaluation result is the second type of evaluation result from t0 to t, D i represents the total number of times the sedation level is evaluated from t0 to t. k represents the preset threshold, or can be called a hyperparameter, and a suitable value of k can be determined through multiple experiments.

[0121] The above is only an example implementation of step 504. In other implementations, it can also be determined according to the value of the cumulative number of times the sedation level evaluation result is the second type of evaluation result from the initial counting moment to the current moment. For example, if the cumulative number of times is greater than a preset value, a prompt signal regarding the sedation level of the target object is generated. Therefore, the specific implementation of step 504 in the embodiments of the present application is not limited.

[0122] The automatic evaluation method of the sedation level introduced above realizes the intelligent evaluation and real-time monitoring of the sedation level of personnel, and can be fully intelligent or only use a small amount of medical staff in the judgment of the sedation level, thus making an important contribution to the realization of intelligent intensive care.

[0123] Based on the method introduced above, correspondingly, the present application also provides an automatic evaluation device for the sedation level. Specific descriptions will be made below in conjunction with embodiments.

[0124] Figure 6This is a schematic structural diagram of an automatic sedation level assessment device provided by an embodiment of the present application. As Figure 6 shown, the automatic sedation level assessment device 600 includes:

[0125] An acquisition module 601, configured to acquire image data and optical flow field data of a target object in the same time period;

[0126] An evaluation module 602, configured to obtain a first sedation level evaluation result of the target object by processing the image data; and, configured to obtain a second sedation level evaluation result of the target object by processing the optical flow field data;

[0127] A fusion module 603, configured to obtain a comprehensive sedation level evaluation result of the target object based on the first sedation level evaluation result and the second sedation level evaluation result.

[0128] Since the sedation level is evaluated separately through image data and optical flow field data, and finally a comprehensive evaluation result is obtained based on the sedation levels evaluated through two approaches of image recognition and behavior recognition, the accuracy of the sedation level evaluation result obtained through one approach can be corrected. Thereby, the accuracy of the sedation level evaluation result is improved, making the final evaluation result more reliable and timely. In addition, the embodiment of the present application can be implemented automatically, which can reduce the workload of medical staff and improve the timeliness of sedation level evaluation.

[0129] Optionally, the comprehensive sedation level evaluation result is a first type of evaluation result or a second type of evaluation result, where the sedation level indicated by the first type of evaluation result is higher than the sedation level indicated by the second type of evaluation result;

[0130] The automatic sedation level assessment device 600 may further include:

[0131] A prompt module, configured to determine whether to generate a prompt signal regarding the sedation level of the target object according to the cumulative number of times that the sedation level evaluation result is the second type of evaluation result from the initial counting moment to the current moment.

[0132] Optionally, the prompt module is specifically configured to generate a prompt signal regarding the sedation level of the target object when the ratio of the cumulative number of times to the total number of times of evaluating the sedation level of the target object from the initial counting moment to the current moment exceeds a preset threshold.

[0133] Optionally, the automatic sedation level assessment device 600 may further include:

[0134] A dataset building module for building a dataset; the dataset includes multiple groups of training samples; the training samples include image training data, optical flow field training data, and sedation level labels;

[0135] A model training module for training a first evaluation model through the image training data and sedation level labels in the training samples, and training a second evaluation model through the optical flow field training data and sedation level labels in the training samples; based on the output results of the first evaluation model, the output results of the second evaluation model, and the sedation level labels, training the fusion coefficients for fusing the first evaluation model and the second evaluation model; evaluating the training effects of the first evaluation model and the second evaluation model, and stopping training when the training effect evaluation is qualified to obtain the trained first evaluation model, the trained second evaluation model, and the trained fusion coefficients;

[0136] Wherein the trained first evaluation model is used to process the image data to obtain a first sedation level evaluation result of the target object, the trained second evaluation model is used to process the optical flow field data to obtain a second sedation level evaluation result of the target object, and the trained fusion coefficients are used to fuse the first sedation level evaluation result and the second sedation level evaluation result to obtain the comprehensive sedation level evaluation result.

[0137] Optionally, the model training module specifically includes:

[0138] A fusion prediction unit for fusing the output results of the first evaluation model and the second evaluation model using the current fusion coefficients to obtain a sedation level fusion prediction result corresponding to the training samples;

[0139] An error evaluation unit for comparing the sedation level fusion prediction result with the sedation level labels in the training samples to obtain the cross entropy as the sedation level evaluation error;

[0140] A parameter optimization unit for backpropagating the sedation level evaluation error to optimize the current fusion coefficients and the parameters of the first evaluation model and the second evaluation model.

[0141] Optionally, the acquisition module 601 specifically includes:

[0142] A video acquisition unit for acquiring the video data of the target object during the period;

[0143] A data separation unit for separating the image data and the optical flow field data from the video data.

[0144] Based on the automatic evaluation method and device for sedation level provided in the foregoing embodiments, an embodiment of the present application further provides a computer-readable storage medium. A program is stored on the storage medium, and when the program is executed by a processor, some or all of the steps in the automatic evaluation method for sedation level protected by the foregoing method embodiments of the present application are implemented.

[0145] The storage medium may be various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disc.

[0146] Based on the automatic evaluation method, device, and storage medium for sedation level provided in the foregoing embodiments, an embodiment of the present application provides a processor. The processor is used to run a program, and when the program runs, some or all of the steps in the automatic evaluation method for sedation level provided by the foregoing method embodiments are executed.

[0147] Based on the storage medium and processor provided in the foregoing embodiments, the present application further provides a device for automatic evaluation of sedation level. Refer to Figure 7 , which is a hardware structure diagram of the device for automatic evaluation of sedation level provided in this embodiment.

[0148] As Figure 7 shown, the device for automatic evaluation of sedation level includes: a memory 1401, a processor 1402, a communication bus 1403, and a communication interface 1404.

[0149] Among them, a program that can run on the processor is stored on the memory 1401, and when the program is executed, some or all of the steps in the automatic evaluation method for sedation level provided by the foregoing method embodiments of the present application are implemented. The memory 1401 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one magnetic disk storage device, a flash memory device, or other volatile solid-state storage devices.

[0150] In the device for automatic evaluation of sedation level, the processor 1402 and the memory 1401 transmit signaling, logical instructions, etc. through the communication bus 1403. The device can communicate and interact with other devices through the communication interface 1404.

[0151] In an embodiment of the present application, the device for automatic evaluation of sedation level may be implemented by a device that locally generates medical images, or may be implemented on other devices, such as a terminal (laptop, desktop computer, mobile phone, etc.) that communicates with the device that generates medical images. Additionally, it may also be implemented on a physical server. Furthermore, the automatic evaluation method and device for sedation level provided in an embodiment of the present application may also be implemented on a cloud server.

[0152] As described above, it is only a specific embodiment of the present application. However, the protection scope of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present application should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. An automatic evaluation method for sedation level, characterized in that, Including: Obtaining image data and optical flow field data of a target object in the same time period; the image data at least includes an eye image; the optical flow field data includes the optical flow field of the whole body; Processing the image data through a trained first evaluation model to obtain a first sedation level evaluation result of the target object; and, processing the optical flow field data through a trained second evaluation model to obtain a second sedation level evaluation result of the target object; wherein, the first sedation level evaluation result is evaluated based on an image recognition approach, and the second sedation level evaluation result is evaluated based on a behavior recognition approach; the first evaluation model is used to analyze the eye opening and closing state of the target object in the image data; the second evaluation model is used to identify the frequency of at least one of raising the hand, shaking the head, or raising the leg; Fusing the first sedation level evaluation result and the second sedation level evaluation result through a trained fusion coefficient to obtain a comprehensive sedation level evaluation result of the target object; the comprehensive sedation level evaluation result is a first type of evaluation result or a second type of evaluation result, where the sedation level indicated by the first type of evaluation result is higher than the sedation level indicated by the second type of evaluation result; When the ratio of the cumulative number of times to the number of times of comprehensively evaluating the sedation level of the target object from the initial counting moment to the current moment exceeds a preset threshold, generating a prompt signal regarding the sedation level of the target object; the cumulative number of times is the sum of the number of times the sedation level evaluation result is the second type of evaluation result from the initial counting moment to the current moment; Wherein, the first evaluation model is trained based on the image training data and sedation level labels in the training samples, and the second evaluation model is trained based on the optical flow field training data and sedation level labels in the training samples; The fusion coefficient is trained based on the output results of the first evaluation model, the output results of the second evaluation model, and the sedation level labels in the training samples during model training; the fusion coefficient is used to correct the accuracy of the sedation level evaluation result obtained by a single image recognition approach or a single behavior recognition graph through the fusion of the first sedation level evaluation result and the second sedation level evaluation result.

2. The automatic evaluation method for the sedation level according to claim 1, characterized in that Training the fusion coefficient for fusing the first evaluation model and the second evaluation model based on the output results of the first evaluation model, the output results of the second evaluation model, and the sedation level labels specifically includes: Fusing the output results of the first evaluation model and the second evaluation model using the current fusion coefficient to obtain a sedation level fusion prediction result corresponding to the training sample; Comparing the sedation level fusion prediction result with the sedation level labels in the training sample to obtain cross-entropy as the sedation level evaluation error; Backpropagating the sedation level evaluation error to optimize the current fusion coefficient and the parameters of the first evaluation model and the second evaluation model.

3. The automatic assessment method for the sedation level according to claim 1, characterized in that, The obtaining of the image data and the optical flow field data of the target object in the same time period specifically includes: Obtain the video data of the target object during the time period; Separate the image data and the optical flow field data from the video data.

4. An automatic evaluation device for sedation level, characterized in that, It includes: An acquisition module for acquiring image data and optical flow field data of a target object during the same time period; the image data at least includes eye images; the optical flow field data includes the optical flow field of the whole body; An evaluation module for processing the image data through a trained first evaluation model to obtain a first sedation level evaluation result for the target object; and for processing the optical flow field data through a trained second evaluation model to obtain a second sedation level evaluation result for the target object; wherein, the first sedation level evaluation result is evaluated based on an image recognition approach, and the second sedation level evaluation result is evaluated based on a behavior recognition approach; the first evaluation model is used to analyze the eye opening and closing state of the target object in the image data; the second evaluation model is used to identify the frequency of at least one of the actions of raising the hand, shaking the head, or raising the leg; A fusion module for fusing the first sedation level evaluation result and the second sedation level evaluation result through a trained fusion coefficient to obtain a comprehensive sedation level evaluation result for the target object; the comprehensive sedation level evaluation result is a first type of evaluation result or a second type of evaluation result, where the sedation level indicated by the first type of evaluation result is higher than the sedation level indicated by the second type of evaluation result; A prompt module, specifically for generating a prompt signal regarding the sedation level of the target object when the ratio of the cumulative number of times to the number of times of evaluating the sedation level of the target object from the initial counting moment to the current moment exceeds a preset threshold; the cumulative number of times is the sum of the number of times when the sedation level evaluation result is the second type of evaluation result from the initial counting moment to the current moment; Wherein, the first evaluation model is trained based on the image training data and sedation level labels in the training samples, and the second evaluation model is trained based on the optical flow field training data and sedation level labels in the training samples; The fusion coefficient is trained based on the output results of the first evaluation model, the output results of the second evaluation model, and the sedation level labels in the training samples during model training; the fusion coefficient is used to correct the accuracy of the sedation level evaluation result obtained by a single image recognition approach or a single behavior recognition pattern through the fusion of the first sedation level evaluation result and the second sedation level evaluation result.

5. The automatic evaluation device for the sedation level according to claim 4, wherein The device further includes a model training module for training a first evaluation model through the image training data and sedation level labels in the training samples, and training a second evaluation model through the optical flow field training data and sedation level labels in the training samples; training the fusion coefficient for fusing the first evaluation model and the second evaluation model based on the output results of the first evaluation model, the output results of the second evaluation model, and the sedation level labels; Evaluate the training effects of the first evaluation model and the second evaluation model. When the training effect evaluation is qualified, stop the training to obtain the trained first evaluation model, the trained second evaluation model, and the trained fusion coefficient; The model training module includes: A fusion prediction unit, configured to fuse the output results of the first evaluation model and the second evaluation model by using the current fusion coefficient to obtain a fusion prediction result of the sedation level corresponding to the training sample; An error evaluation unit, configured to compare the fusion prediction result of the sedation level with the sedation level label in the training sample to obtain the cross entropy as the sedation level evaluation error; A parameter optimization unit, configured to backpropagate the sedation level evaluation error to optimize the current fusion coefficient and the parameters of the first evaluation model and the second evaluation model.

6. The automatic evaluation device for sedation level according to claim 4, characterized in that, The obtaining module specifically includes: A video obtaining unit, configured to obtain video data of the target object during the time period; A data separation unit, configured to separate the image data and the optical flow field data from the video data.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by a processor, the steps of the method according to any one of claims 1-3 are implemented.

8. An electronic device, characterized in that, Including: A memory, on which a computer program is stored; A processor, configured to execute the computer program in the memory to implement the steps of the method according to any one of claims 1-3.

Images