Injury treatment skill training method and system based on deep learning

Abstract

The present application relates to the technical field of wounded person rescue skill training, and particularly relates to a wound condition rescue skill training method and system based on deep learning, which comprises the following steps: acquiring a user-selected wound condition type and tool type; generating a training list according to the wound condition type and the tool type, wherein the training list comprises a plurality of training subsets arranged in sequence; acquiring a training start signal and entering a training phase; training according to the sequence of each training subset in the training list and acquiring training data and training evaluation feedback information for each training subset, wherein the evaluation feedback information comprises improvement suggestions corresponding to each training subset; retrieving the corresponding training subset for secondary training according to the improvement suggestions; and acquiring and displaying the improvement suggestions for each scoring item. The present application can effectively improve the rescue skills of students.

Description

[0001] This application is a divisional application of the invention patent application filed on November 7, 2023, entitled "Interactive Training Method and System for Wounded Person Rescue Skills Based on Evaluation Feedback", with application number 202311476106.4. Technical Field

[0002] This invention relates to the field of training technology for wounded personnel treatment skills, and in particular to a method and system for training wounded personnel treatment skills based on deep learning. Background Technology

[0003] To improve the skills of medical personnel in treating the wounded in critical situations, the applicant proposed using a human-computer interaction method to organize training for medical personnel in critical situation treatment skills. Users select elements such as training scenarios, injuries, and tools through the training system's selection interface. The system then generates a training list consisting of tracks based on the user's selections, allowing trainees to practice according to the list's content. While this method can automatically generate training lists according to training needs, the training effectiveness of trainees cannot be fed back in a timely manner, hindering their progress and failing to meet the need for trainees to rapidly improve their treatment skills. Summary of the Invention

[0004] In view of this, embodiments of the present invention provide a method for obtaining the injury type and tool type selected by the user;

[0005] A training list is generated based on the injury type and the tool type, and the training list includes several training subsets arranged sequentially.

[0006] Receive the training start signal and enter the training phase;

[0007] Training is performed in the order of the training subsets in the training list, and training data and training evaluation feedback information for each training subset are obtained. The evaluation feedback information includes improvement suggestions corresponding to each training subset.

[0008] Based on the improvement suggestions, the corresponding training subset was retrieved for secondary training;

[0009] The step of training according to the order of the training subsets in the training list and obtaining training data and training evaluation feedback information for each training subset also includes the following steps:

[0010] The evaluator's display interface shows the name of the current training subset and the assessment subject;

[0011] Obtain and display the training time of the current training subset and the total training time of the training phase in the evaluator's display interface;

[0012] The display interface shows the various scoring items for the current training subset;

[0013] Retrieve and display the score results of each scoring item in the current training subset;

[0014] Retrieve and display suggestions for improvement for each current rating item;

[0015] Establish a correspondence between various non-standard rescue actions and various types of improvement suggestions;

[0016] Obtain the training video image data corresponding to each scoring item;

[0017] Continuous training motion video data and posture accuracy training motion video data are extracted from the training video image data corresponding to the scoring items.

[0018] The continuous training video data is input into the time series data deep learning model for processing to obtain the identification results of non-standard rescue actions in the continuous training.

[0019] Based on the identification results of non-standard rescue actions in continuous training and the correspondence between non-standard rescue actions and improvement suggestions, corresponding improvement suggestions are obtained.

[0020] The video data of the posture accuracy training action is input into the posture recognition deep learning model for processing, and then the recognition result of the non-standard rescue action of the posture accuracy training action is obtained.

[0021] Based on the identification results of non-standard rescue actions during posture accuracy training and the correspondence between non-standard rescue actions and improvement suggestions, corresponding improvement suggestions are obtained.

[0022] Preferably, the step of obtaining and displaying the scoring results of each scoring item in the current training subset further includes the following steps:

[0023] Several scoring options are displayed in the corresponding positions of each scoring item on the evaluator's display interface;

[0024] The rating results for each rating item are obtained based on the user's selection of the rating options in the corresponding positions of each rating item;

[0025] Training videos of trainees are collected during the training phase;

[0026] Extract the training video segments corresponding to the scoring items from the training videos;

[0027] Analyze the erroneous operations that occur in each training video segment, mark the location of the erroneous action in the corresponding video frame image, and save the marked video frame as a reference video image to the training database;

[0028] Obtain a video demonstrating the correct operating procedure;

[0029] Extract the standard operation video segments corresponding to the scoring items from the standard operation video;

[0030] When several scoring options are displayed in the corresponding position of each scoring item on the examiner's display interface, retrieve the reference video image and standard operation video clip corresponding to the scoring item;

[0031] The corresponding reference video images and standardized operation video clips for each scoring item are displayed in the corresponding positions on the evaluator's display interface.

[0032] Preferably, before displaying the name of the current training subset and the assessment subject on the evaluator's display interface, the method further includes:

[0033] Video images of trainees in various training subsets are acquired using image acquisition equipment;

[0034] Wearable devices are used to obtain trainees' blood oxygen, heart rate, blood pressure, and body temperature data in various training subsets.

[0035] Preferably, before obtaining the identification result of non-standard rescue actions of the continuous training actions by inputting the continuous training action video data into the time series data deep learning model for processing, the method further includes:

[0036] Video data of non-standard rescue actions in various continuous training movements were collected as the first training sample;

[0037] The first training sample is labeled with the type of non-standard rescue movements in continuous training actions;

[0038] Establish a deep learning model for the initial time series data;

[0039] The deep learning model for time series data is obtained by training the initial model of the deep learning model for time series data using the first labeled training sample.

[0040] Video data of non-standard rescue actions from various accuracy training movements were collected as a second training sample;

[0041] The types of non-standard rescue actions used in the accuracy training were labeled as the second training samples;

[0042] Establish an initial pose recognition deep learning model;

[0043] The pose recognition deep learning model is obtained by training the initial pose recognition deep learning model with the labeled second training samples.

[0044] Preferably, the step of extracting continuous training motion video data and posture accuracy training motion video data from the training video image data corresponding to the scoring items further includes the following steps:

[0045] The feature pose image data corresponding to the pose accuracy training actions are collected as the third training sample;

[0046] Establish an initial model for first pose recognition;

[0047] The first pose recognition model is obtained by training the initial model of the first pose recognition with the third training sample;

[0048] The trigger action image data and the end action image data corresponding to the continuous training actions are collected as the fourth training sample;

[0049] The trigger action image data and the end action image data in the fourth training sample are labeled respectively;

[0050] Establish an initial model for second pose recognition;

[0051] The second pose recognition model is obtained by training the initial model of the second pose recognition using the labeled fourth training sample;

[0052] The training video image data corresponding to the scoring items is input into the first posture recognition model to identify the image data corresponding to the feature posture as the posture accuracy training action video data;

[0053] The training video image data corresponding to the scoring item is input into the second pose recognition model to identify the image data corresponding to the trigger action and the image data corresponding to the end action;

[0054] The image data corresponding to the trigger action, the image data corresponding to the end action, and the image data in between are used as continuous training action video data.

[0055] Preferably, the step of retrieving the corresponding training subset for secondary training based on the improvement suggestions further includes the following steps:

[0056] Based on the improvement suggestions, extract training video clips corresponding to the improvement suggestions from the training videos of the trainees;

[0057] The erroneous actions in the video clips are filtered out, and the positions of the erroneous actions are marked in the video clips to obtain the marked video clips;

[0058] Extract the standardized action video segments that correspond to the improvement suggestions from the standardized operation videos;

[0059] The marked video clips and the standard action video clips are displayed side-by-side in the display interface;

[0060] Extract the human posture and first ground baseline of the standardized movements from video clips of standardized movements;

[0061] Extracting erroneous human poses and a second ground baseline from labeled video clips;

[0062] Using the first and second ground baselines as coincident references, the extracted human postures of standard movements and incorrect movements are merged and displayed in the same image;

[0063] The process of obtaining and displaying improvement suggestions for each current rating item also includes the following steps:

[0064] An input box for improvement suggestions is displayed in the corresponding position of each scoring item on the evaluator's display interface;

[0065] Based on the user's input information in the corresponding improvement suggestion input box for each rating item, obtain the student's improvement suggestions for the corresponding rating item.

[0066] Preferably, the types of injuries include lower limb fractures due to falls from heights, right lower limb fractures caused by crushing buildings, facial injuries, and head injuries resulting in shock. The types of tools include towing bags, folding stretchers, first-aid kits, unmanned stretcher carts, and roll-up stretchers.

[0067] Preferably, generating a training list based on the injury type and the tool type, wherein the training list includes several sequentially arranged training subsets, further includes the following steps:

[0068] Based on the type of injury, obtain the steps for treating the corresponding injury and the order in which the steps are performed;

[0069] Based on each step of the treatment of the corresponding injury, a training subset corresponding to each step is determined;

[0070] The training list is formed by arranging the training subsets in the order of each step and the correspondence between each step and each training subset.

[0071] The step of determining the training subset corresponding to each step of the treatment of the corresponding injury also includes the following steps:

[0072] Select one step from each step whose corresponding training subset is not yet determined as the step to be matched;

[0073] Retrieve the training subjects corresponding to the steps to be matched from the syllabus subject database as matching subjects;

[0074] Generate a corresponding training subset using the matched subjects as the training content;

[0075] The generated training subset is determined as the training subset that matches the step to be matched;

[0076] Repeat the aforementioned process of generating and determining training subsets until all matching steps have completed the determination of the corresponding training subsets.

[0077] Preferably, the method further includes the following steps:

[0078] Access the historical score query interface based on the historical score query command sent by the user.

[0079] Obtain the query information entered by the user on the historical grades query interface;

[0080] The corresponding score information will be displayed on the historical score query interface based on the query information.

[0081] In a second aspect, the present invention provides a deep learning-based injury treatment skills training system, the system comprising at least one processor, at least one memory, and computer program instructions stored in the memory, wherein when the computer program instructions are executed by the processor, the method described in the first aspect is implemented.

[0082] Beneficial Effects: The deep learning-based injury treatment skills training method and system of this invention generates a training list based on the user-selected injury type and tool type. This allows for convenient and targeted training tailored to various injuries, enabling trainees to master treatment skills for different conditions. Furthermore, this invention not only acquires training data for each training subset during the training phase but also obtains training evaluation information and improvement suggestions for each subset. This provides trainees with timely feedback on training evaluation results and improvement suggestions, facilitating further improvement of their treatment skills. Attached Figure Description

[0083] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the embodiments of the present invention will be briefly introduced below. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort, and these are all within the protection scope of the present invention.

[0084] Figure 1 This is a flowchart illustrating the interactive wound care skills training method based on evaluation feedback of the present invention.

[0085] Figure 2 This is a flowchart illustrating the method for generating a training list based on user parameters in the training mode of this invention.

[0086] Figure 3 This is a flowchart illustrating the scoring method under the training mode of the present invention;

[0087] Figure 4 This is a flowchart illustrating the method for training in treating the wounded in a flood scenario according to the present invention.

[0088] Figure 5 This is a flowchart illustrating the method for generating an assessment list based on extracted content under the assessment model of the present invention.

[0089] Figure 6 This is a flowchart illustrating the method of inserting emergency situation assessment into the assessment process according to the present invention.

[0090] Figure 7 This is a flowchart illustrating the method of adjusting the assessment list based on the inserted risks according to the present invention.

[0091] Figure 8 This is a schematic diagram of the process of the assessment method in team mode according to the present invention. Detailed Implementation

[0092] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Unless otherwise specified, the embodiments of the present invention and the various features therein can be combined with each other, all of which are within the protection scope of the present invention.

[0093] Example 1

[0094] like Figure 1 As shown in the figure, this embodiment of the invention provides an interactive training method for casualty care skills based on evaluation feedback. The method includes the following steps:

[0095] S1: Obtain the injury type and tool type selected by the user;

[0096] In this step, a menu for selecting injury type and tool type can be displayed on the training mode interface. This menu, when expanded, displays various types of injuries and tools. Users select the injury type and tool type by clicking on one of these categories. After the user completes the selection, the system obtains the selected injury type and tool type based on the signal generated by the user's action and extracts the corresponding injury and tool data from the injury database. Injury types include, but are not limited to, lower limb fractures caused by falls from heights, fractures of the right lower limb from being crushed by a large building, maxillofacial injuries, and head injuries resulting in shock. Tool types include, but are not limited to, drag bags, folding stretchers, first-aid kits, unmanned stretcher carts, and roll-up stretchers.

[0097] S2: Generate a training list based on the injury type and the tool type, wherein the training list includes several training subsets arranged sequentially;

[0098] Each training subset includes corresponding training subjects. Trainees follow the order of the training subsets in the training list for training, and the content of each training session is the content of the training subject of that training subset.

[0099] S3: Enter the training phase after detecting the training start signal;

[0100] Once the training list is generated, a start button is displayed on the screen. When the user clicks the start button, the system generates a start signal. Upon detecting the start signal, the training phase begins, and relevant information about the current training subset is displayed on the screen.

[0101] S4: After entering the training phase, train according to the order of each training subset in the training list and obtain training data and training evaluation information and improvement suggestions for each training subset.

[0102] This step guides trainees to train according to the order of the training subsets on the training checklist. The training content of each subset is the content of that subset's training subject. This step can obtain evaluation information on the trainee's training performance for each training subset during training or after the current training subset is completed. Evaluation information can be in the form of a score. For example, the maximum score is 100 points, and the score is based on the trainee's training performance, with higher scores for better performance. Evaluation information can also be in the form of deductions, with lower deductions for worse performance. Evaluation information can also be in the form of brief comments, such as "average," "poor," or "not done," allowing trainees to more clearly understand their performance in training. In practice, any combination of the above formats can be used. Improvement suggestions can be in the form of text descriptions, image prompts, or video action instructions, and in practice, any combination of the above formats can be used.

[0103] In this embodiment, S4: After entering the training phase, training is performed according to the order of each training subset in the training list, and training data, training evaluation information and improvement suggestions for each training subset are obtained. This also includes the following steps:

[0104] S41: Acquire video images of trainees in various training subsets using image acquisition equipment;

[0105] During the training of each training subset, high-definition cameras can be used to record the trainees' training process, acquiring relevant video image data. This video data allows for the recording of operational details during training, facilitating precise evaluation of trainees' performance later on.

[0106] S42: Acquire trainees' blood oxygen, heart rate, blood pressure, and body temperature data in various training subsets through wearable devices.

[0107] Wearable devices include, but are not limited to, smart bracelets, smartwatches, and smart glasses. Physiological information such as blood oxygen, heart rate, blood pressure, and body temperature data collected during training can provide a more comprehensive picture of a trainee's performance.

[0108] In this embodiment, step S4: after entering the training phase, training is performed according to the order of each training subset in the training list, and training data and training evaluation information and improvement suggestions for each training subset are obtained, which further includes the following steps:

[0109] S43: Display the name of the current training subset and the assessment subject on the evaluator display interface;

[0110] Once the training phase begins, this step displays the name of the training subset and the assessment subject for the current trainee on the evaluator's screen to facilitate the evaluator's understanding of the relevant training information.

[0111] S44: Obtain and display the training time of the current training subset and the overall training time of the training phase in the evaluator's display interface;

[0112] When the first training subset begins, timing begins simultaneously for both the first training subset and the entire training phase. When the first training subset ends, the timing for that subset is reset to zero, and timing for the entire training phase pauses. When the next training subset begins, the timing for that subset restarts, while timing for the entire training phase continues.

[0113] S45: Display the scoring items of the current training subset in the display interface;

[0114] For example, neat attire, complete equipment, standardized movements, and timely completion can all be used as scoring items for the current training subset.

[0115] S46: Get and display the scoring results of each scoring item in the current training subset;

[0116] The scoring result is the score of each scoring item. In this embodiment, S46: obtaining and displaying the scoring results of each scoring item in the current training subset further includes the following steps:

[0117] S461: Display several scoring options in the corresponding positions of each scoring item on the evaluator's display interface;

[0118] For example, after the "neat attire" rating option, three rating options can be set: "Average", "Poor", and "Not done".

[0119] S462: Obtain the rating results for each rating item based on the user's selection of the rating options for each rating item.

[0120] Users can input their ratings into the system by clicking the corresponding rating option.

[0121] As an optional but advantageous implementation, in this embodiment, training videos of trainees are collected during the training phase;

[0122] This step involves using an image acquisition device to capture video footage of the trainee's operational process during training.

[0123] Extract the training video segments corresponding to the scoring items from the training videos;

[0124] Training videos often include all the actions of the trainees in all the scoring items during the training phase. Each scoring item is only related to certain segments in the training video. Therefore, this step extracts the video segments in the training video that are related to each scoring item.

[0125] Analyze the erroneous operations that occur in each training video segment, mark the location of the erroneous action in the corresponding video frame image, and save the marked video frame as a reference video image to the training database;

[0126] This step analyzes the training video segments, identifies erroneous operations, extracts the video frames corresponding to the time the erroneous operation occurred, and marks the location of the erroneous action in the image of the video frame.

[0127] For example, in the assessment of artificial respiration in emergency treatment, the standard procedure is that the operator should place the heel of their left hand at the midpoint of the line connecting the patient's two nipples, with the right hand overlapping the back of the left hand. The operator's shoulder, elbow, and wrist should be on the same axis, with the entire shoulder and elbow perpendicular to the patient's body, using the heel of the palm as the point of force, and the elbow joint should be straight and not bent.

[0128] This application can determine, through analysis and comparison, whether the position of the trainee's left palm heel is correct; if incorrect, mark the position of the left palm heel. It can also determine whether the trainee's shoulder, elbow, and wrist are on the same axis; if incorrect, mark the angle between the lines connecting the shoulder, elbow, and wrist. Furthermore, it can determine whether the trainee's entire shoulder and elbow are perpendicular to the patient's body; if incorrect, mark the angle between the trainee's entire shoulder and elbow and the patient's body. Finally, it can determine whether the trainee's elbow joint is fully extended; if not, mark the flexion angle of the trainee's elbow joint.

[0129] Obtain a video demonstrating the correct operating procedure;

[0130] A standard operation video is a video showing the correct operation actions recorded.

[0131] Extract the standard operation video segments corresponding to the scoring items from the standard operation video;

[0132] Since the standardized operation video records the correct actions for all scoring items, this step extracts the standardized action videos corresponding to the scoring items.

[0133] When several scoring options are displayed in the corresponding position of each scoring item on the examiner's display interface, retrieve the reference video image and standard operation video clip corresponding to the scoring item;

[0134] The corresponding reference video images and standardized operation video clips for each scoring item are displayed in the corresponding positions on the evaluator's display interface.

[0135] This application retrieves and displays reference video images and standardized operation video clips simultaneously during the evaluation process. Since the reference video images mark the locations of incorrect actions, the evaluators can quickly identify the errors made by the trainees during training by using the reference video images and standardized operation video clips. They can also easily and accurately determine the magnitude of the difference between the incorrect actions and the standardized actions, thus providing the evaluators with a reliable basis for scoring.

[0136] As an optional but advantageous implementation, S5: retrieving the corresponding training subset for secondary training based on the improvement suggestions further includes the following steps:

[0137] Based on the improvement suggestions, extract training video clips corresponding to the improvement suggestions from the training videos of the trainees;

[0138] This step involves using an image acquisition device to capture video footage of the trainee's operational process during training.

[0139] Since the improvement suggestions correspond to the scoring items, this step can first find the scoring items that correspond to the improvement suggestions, and then find the corresponding training video segments through the scoring items.

[0140] The erroneous actions in the video clips are filtered out, and the positions of the erroneous actions are marked in the video clips to obtain the marked video clips;

[0141] This step analyzes the extracted video clips, identifies erroneous actions, and marks the erroneous actions in the video clips.

[0142] Extract the standardized action video segments that correspond to the improvement suggestions from the standardized operation videos;

[0143] Since the improvement suggestions correspond to the scoring items, this step can first find the scoring items that correspond to the improvement suggestions, and then find the corresponding training video segments from the pre-prepared standard movement videos through the scoring items.

[0144] The marked video clips and the standard action video clips are displayed side-by-side in the display interface.

[0145] This step displays both video clips with error messages and video clips showing correct movements while trainees review improvement suggestions. This helps trainees quickly and accurately identify their mistakes by combining the suggestions with the two types of video clips, and then learn the correct movements in a targeted manner based on the suggestions, thereby further improving training efficiency.

[0146] Extract the human posture and first ground baseline of the standardized movements from video clips of standardized movements;

[0147] The first ground baseline is the horizontal line on the ground where the people are located in the video clip.

[0148] Extracting erroneous human poses and a second ground baseline from labeled video clips;

[0149] The second ground baseline is the horizontal line on the ground where the people are located in the video clip.

[0150] Using the first and second ground baselines as coincident references, the extracted human postures of standard movements and incorrect movements are merged and displayed in the same image.

[0151] In this embodiment, the two human postures mentioned above are superimposed to be displayed in one image. In specific operation, the two ground baselines can be aligned, which allows the trainee to carefully compare the differences between their own movements and the standard movements.

[0152] S47: Retrieve and display improvement suggestions for each current rating item.

[0153] To help trainees understand the shortcomings exposed in their training and make targeted improvements in future training, this embodiment not only obtains and displays the scoring results of each scoring item, but also obtains improvement suggestions for each scoring item.

[0154] As an optional but advantageous implementation, in this embodiment, S47: obtaining and displaying improvement suggestions for each current rating item further includes the following steps:

[0155] S471: Display an input box for improvement suggestions in the corresponding position of each scoring item on the evaluator's display interface;

[0156] Evaluators can enter their evaluation suggestions into the input box, and can also insert relevant images into the input box.

[0157] S472: Obtain student improvement suggestions for each rating item based on the input information from the user's input box for each rating item.

[0158] To help trainees understand which scoring items the improvement suggestions target, this example will record the improvement suggestions corresponding to each scoring item.

[0159] To improve the efficiency and accuracy of feedback evaluation, this embodiment also provides a method for the system to automatically provide improvement suggestions based on training performance. As an optional but advantageous implementation method, S47: obtaining and displaying improvement suggestions for each current scoring item in this embodiment further includes the following steps:

[0160] S471A: Establish the correspondence between various non-standard rescue actions and various types of improvement suggestions;

[0161] Inexperienced or poorly trained rescue personnel often employ non-standard rescue procedures. These non-standard procedures vary, and the corrective measures for each differ. Therefore, this embodiment first identifies the most effective improvement suggestions for various non-standard rescue procedures and establishes a corresponding relationship between these suggestions and the various non-standard rescue procedures. To facilitate real-time access to the most effective improvement suggestions, a database of non-standard rescue procedures and improvement suggestions can be established. This database stores various types of non-standard rescue procedures and improvement suggestions, along with the corresponding relationships between them.

[0162] S472A: Obtain the training video image data corresponding to each scoring item;

[0163] When a user trains according to each scoring item, the camera can be used to record the user's training process, thereby obtaining training video image data corresponding to each scoring item.

[0164] S473A: Extract continuous training motion video data and posture accuracy training motion video data from the training video image data corresponding to the scoring items, respectively.

[0165] For emergency medical treatment of the injured, certain actions require a high degree of continuity, while others require a high degree of accuracy. For example, in maintaining an open airway, the rescuer must accurately place one hand on the injured person's forehead to tilt their head back, and use the other hand to lift the back of the neck or chin to keep the airway open. Similarly, in mouth-to-mouth resuscitation, the rescuer must pinch the nostrils (or lips) closed with one hand, take a deep breath, and quickly and forcefully blow air into the injured person's mouth (or nose), then release the nostrils (or lips), repeating this sequence every 5 seconds. This step will extract and process the video data requiring both high accuracy and high continuity separately.

[0166] S474A: The continuous training action video data is input into the time series data deep learning model for processing to obtain the identification result of non-standard rescue actions of continuous training actions;

[0167] For continuous training video data, this embodiment employs a time-series deep learning model, which is more suitable for capturing temporal information in datasets with temporal sequences, for processing. The deep learning model can be an RNN (Recurrent Neural Network), an LSTM (Long Short-Term Memory), or a GRU (Gated Recurrent Units).

[0168] This step uses a deep learning model based on time series data to identify the types of non-standard rescue actions in the user's continuous training movements.

[0169] S475A: Based on the identification results of non-standard rescue actions in continuous training movements and the correspondence between non-standard rescue actions and improvement suggestions, obtain corresponding improvement suggestions;

[0170] Once the types of non-standard rescue actions obtained from the user's continuous training actions are identified, the corresponding improvement suggestions can be found by matching the non-standard rescue actions with the improvement suggestions, and then provided to the user.

[0171] S476A: Input the video data of the posture accuracy training action into the posture recognition deep learning model for processing to obtain the recognition result of the non-standard rescue action of the posture accuracy training action;

[0172] For training motion video data on posture accuracy, this embodiment uses a posture recognition deep learning model that is more suitable for accurate posture identification. The OpenPose model can be used as the posture recognition deep learning model.

[0173] This step utilizes a pose recognition deep learning model to identify the types of non-standard rescue actions that the user's pose accuracy training movements are not in accordance with.

[0174] S477A: Based on the identification results of non-standard rescue actions during posture accuracy training and the correspondence between non-standard rescue actions and improvement suggestions, corresponding improvement suggestions are obtained.

[0175] Once the types of non-standard rescue actions in the user's posture accuracy training are obtained, the corresponding improvement suggestions can be found by matching the non-standard rescue actions with the improvement suggestions, and then the corresponding improvement suggestions can be provided to the user.

[0176] This embodiment extracts continuous training video data and posture accuracy training video data separately, and then uses different models to identify the types of non-standard movements for each. Therefore, it can more accurately analyze the error characteristics of the user's movements and thus provide the best suggestions in a targeted manner.

[0177] To more accurately identify the types of non-standard rescue actions, this embodiment includes the following step before S474A: inputting the continuous training action video data into the time series data deep learning model for processing to obtain the identification result of the non-standard rescue actions of the continuous training actions:

[0178] S4741A: Collect video data of non-standard rescue actions in various continuous training movements as the first training sample;

[0179] In practice, relevant personnel can first demonstrate various non-standard rescue actions in continuous training movements, and then film the demonstrations to obtain video data as the first training sample.

[0180] S4742A: The type of non-standard rescue action in continuous training is used as the first training sample for labeling;

[0181] This step marks the types of non-standard rescue movements in continuous training actions, and provides feedback to the model through the model output results and corresponding marks.

[0182] S4743A: Establish a deep learning model for initial time series data;

[0183] This step first establishes an initial model for subsequent training.

[0184] S4744A: A deep learning model for time series data is obtained by training the initial time series data deep learning model with the first labeled training sample.

[0185] S4741A: Collect video data of non-standard rescue actions of various continuous training movements as a second training sample;

[0186] In practice, relevant personnel can first demonstrate various non-standard rescue actions for accuracy training, and then film the demonstrations to obtain video data as a second training sample.

[0187] S4742A: Use the type of non-standard rescue movements in continuous training actions to label the second training sample;

[0188] This step marks the types of non-standard rescue movements in the accuracy training, and uses the model output results and corresponding marks to provide feedback to the model and automatically adjust the parameters.

[0189] S4743A: Establish an initial pose recognition deep learning model;

[0190] This step first establishes an initial pose recognition deep learning model for subsequent training.

[0191] S4744A: A time-series data deep learning model is obtained by training the initial pose recognition deep learning model with the labeled second training samples.

[0192] The trained pose recognition deep learning model and time series data deep learning model can be used to identify various non-standard movements of the trainee.

[0193] In practice, continuous training motion video data and posture accuracy training motion video data can be manually extracted from the training video image data corresponding to the scoring items. Furthermore, this embodiment also provides a method for automatically extracting continuous training motion video data and posture accuracy training motion video data by the system. In this embodiment, S473A: extracting continuous training motion video data and posture accuracy training motion video data from the training video image data corresponding to the scoring items further includes the following steps:

[0194] S4731A: Collect feature pose image data corresponding to the pose accuracy training action as the third training sample;

[0195] The characteristic postures corresponding to the posture accuracy training movements refer to the postures exhibited by the trainee when attempting the movements required for posture accuracy training. Since the trainee's movements may not be standardized, the sample images in this step should include both the postures when the movements are standardized and, as far as possible, the postures when the movements are not standardized but still belong to the trainee's attempts at the relevant movements.

[0196] To address this, relevant personnel can demonstrate various movements that require posture accuracy training beforehand, and the relevant video and image data can be captured using a camera.

[0197] S4732A: Establish the initial model for first pose recognition;

[0198] The initial model for the first pose recognition can be the PoseNet pose estimation model.

[0199] S4733A: The first pose recognition model is obtained by training the initial model of the first pose recognition with the third training sample;

[0200] S4734A: Collect the trigger action image data and the end action image data corresponding to the continuous training actions as the fourth training sample;

[0201] In continuous training exercises, the trigger action refers to the movement that indicates the start of the continuous training exercise, while the ending action refers to the movement that indicates the end of the continuous training exercise. Relevant personnel can demonstrate the trigger action image data and the ending action beforehand, and the relevant video image data can be captured using a camera.

[0202] S4735: Label the trigger action image data and the end action image data in the fourth training sample respectively;

[0203] S4736A: Establish the initial model for second pose recognition;

[0204] The initial model for the second pose recognition can be the PoseNet pose estimation model.

[0205] S4737A: The second pose recognition model is obtained by training the initial model of the second pose recognition with the labeled fourth training sample;

[0206] S4738A: Input the training video image data corresponding to the scoring item into the first posture recognition model to identify the image data corresponding to the feature posture as the posture accuracy training action video data;

[0207] S4738A: Input the training video image data corresponding to the scoring item into the second pose recognition model to identify the image data corresponding to the trigger action and the image data corresponding to the end action;

[0208] S4739A: Use the image data corresponding to the trigger action, the image data corresponding to the end action, and the image data in between as continuous training action video data.

[0209] This step can find a series of image data whose timing is between the image data corresponding to the trigger action and the image data corresponding to the end action, and then use this series of image data together with the image data corresponding to the trigger action and the image data corresponding to the end action as continuous training action video data.

[0210] This embodiment uses a first posture recognition model trained with samples to identify posture accuracy training video data, and uses a second posture recognition model trained with samples to extract continuous training video data, eliminating the need for manual data extraction and making the product more intelligent.

[0211] As an optional but advantageous implementation, in this embodiment, S2: generating a training list based on the injury type and the tool type, the training list including several sequentially arranged training subsets, further includes the following steps:

[0212] S21: Obtain the steps for treating the corresponding injury and the order of these steps according to the type of injury.

[0213] Since the treatment of injuries is carried out in steps, this embodiment adopts the training according to the steps of injury treatment. For example, the treatment of injuries caused by dangerous explosives with partial intestinal protrusion includes six steps in sequence: "search," "rescue," "transfer," "rescue," "transport," and "treatment."

[0214] S22: Determine the training subset corresponding to each step according to the steps of the treatment of the corresponding injury;

[0215] In practice, each step corresponds to a training subset, and the name of each training subset is the same as the name of the step corresponding to that training subset.

[0216] S23: Arrange the training subsets in the order of each step and the correspondence between each step and each training subset to form the training list.

[0217] That is, the training subsets corresponding to the earlier steps are listed first, and the training subsets corresponding to the later steps are listed last.

[0218] S22, which involves determining the training subset corresponding to each step of the treatment of the corresponding injury, specifically includes the following steps:

[0219] S221: Select one step from each step whose corresponding training subset is not yet determined as the step to be matched;

[0220] S222: Obtain the training subjects corresponding to the steps to be matched from the syllabus subject database as matching subjects;

[0221] For example, the assessment subject corresponding to the step "rescue" is chest cavity sealing dressing, the subject corresponding to the step "rescue" is abdominal cavity contents protection, and the subject corresponding to the step "treatment" is injury assessment, pain judgment, and analgesia. The correspondence between each step and training subject can be stored in advance in the syllabus subject database.

[0222] S223: Generate a corresponding training subset using the matched subjects as training content;

[0223] That is, the training content of the generated training subset is the matching subject.

[0224] S224: Determine the generated training subset as the training subset that matches the step to be matched;

[0225] The training subset generated in this step is the training subset matched with any target step of the matching subject of the training content of the training subset. If the step to be matched is "treatment", and the corresponding training subject of this step is injury assessment, pain judgment and analgesia, then a training subset with injury assessment, pain judgment and analgesia as the training content is generated, and this training subset is determined to be the training subset that matches the step to be matched "treatment".

[0226] S225: Repeat the aforementioned process of generating and determining training subsets until all matching steps have completed the determination of the corresponding training subsets.

[0227] Steps S221 to S224 can be repeated continuously to determine a training subset for each step. Once all training subsets have been determined, they are arranged into a training list in sequence.

[0228] As an optional but advantageous implementation, the interactive casualty care skills training method based on evaluation feedback further includes the following steps:

[0229] S5: Enter the historical score query interface based on the historical score query command sent by the user;

[0230] The system can display buttons for retrieving historical scores on the same screen. When a user clicks a button related to querying historical scores, the system will generate a historical score query command. When a historical score query command is detected, the system will display the historical score query interface.

[0231] S6: Obtain the query information entered by the user on the historical grades query interface;

[0232] Users can enter information such as training session, unit, student name, time, and training subject in the historical results query interface.

[0233] S7: Display the corresponding score information on the historical score query interface based on the query information.

[0234] The system retrieves the corresponding score from the imaging database based on the query information and displays it. The historical score query interface displays the corresponding score information, including each operation step, the corresponding syllabus subject, time taken, score, grade, and a button to view the reason for deduction.

[0235] Example 2

[0236] like Figure 5 As shown, this embodiment also provides an interactive wound care skills training system based on evaluation feedback. The system includes at least one processor, at least one memory, and computer program instructions stored in the memory. When the computer program instructions are executed by the processor, the system implements the method described in Embodiment 1.

[0237] Specifically, the processor may be a central processing unit (CPU), an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement embodiments of the present invention.

[0238] The memory may include a large-capacity storage device for data or instructions. For example, and not limitingly, the memory may include a hard disk drive (HDD), a floppy disk drive, flash memory, an optical disk drive, a magneto-optical disk drive, magnetic tape, or a Universal Serial Bus (USB) drive, or a combination of two or more of these. Where appropriate, the memory may include removable or non-removable (or fixed) media. Where appropriate, the memory may be internal or external to the data processing device. In a particular embodiment, the memory is a non-volatile solid-state memory. In a particular embodiment, the memory includes a read-only memory (ROM). Where appropriate, the ROM may be a mask-programmed ROM, a programmable ROM (PROM), an erasable PROM (EPROM), an electrically erasable PROM (EEPROM), an electrically rewritable ROM (EAROM), or flash memory, or a combination of two or more of these.

[0239] The processor implements any of the interaction methods described in the above embodiments by reading and executing computer program instructions stored in memory.

[0240] In one example, the display screen of this embodiment may further include a communication interface and a bus. The control circuit, memory, and communication interface are connected via the bus and communicate with each other.

[0241] The communication interface is mainly used to enable communication between various modules, devices, units and / or equipment in the embodiments of the present invention.

[0242] A bus, including hardware, software, or both, couples various components used for a display screen together. For example, and not limitingly, a bus may include an Accelerated Graphics Port (AGP) or other graphics bus, an Enhanced Industry Standard Architecture (EISA) bus, a Front Side Bus (FSB), HyperTransport (HT) interconnect, an Industry Standard Architecture (ISA) bus, an Infinite Bandwidth Interconnect, a Low Pin Count (LPC) bus, a memory bus, a Microchannel Architecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express (PCI-X) bus, a Serial Advanced Technology Attachment (SATA) bus, a Video Electronics Standards Association Local (VLB) bus, or other suitable buses, or combinations of two or more of these. Where appropriate, bus 410 may include one or more buses. While specific buses are described and illustrated in embodiments of the invention, the invention contemplates any suitable bus or interconnect.

[0243] The above is a detailed introduction to the interactive wound care skills training method and system based on evaluation feedback of the present invention.

[0244] It should be clarified that the present invention is not limited to the specific configurations and processes described above and shown in the figures. For the sake of brevity, detailed descriptions of known methods are omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of the present invention is not limited to the specific steps described and shown. Those skilled in the art can make various changes, modifications, and additions, or change the order of steps, after understanding the spirit of the present invention.

[0245] The functional blocks shown in the above-described structural diagram can be implemented as hardware, software, firmware, or a combination thereof. When implemented in hardware, they can be, for example, electronic circuits, application-specific integrated circuits (ASICs), appropriate firmware, plug-ins, function cards, etc. When implemented in software, the elements of this invention are programs or code segments used to perform the required tasks. The programs or code segments can be stored on a machine-readable medium or transmitted over a transmission medium or communication link via data signals carried in a carrier wave. "Machine-readable medium" can include any medium capable of storing or transmitting information. Examples of machine-readable media include electronic circuits, semiconductor memory devices, ROM, flash memory, erasable ROM (EROM), floppy disks, CD-ROMs, optical disks, hard disks, fiber optic media, radio frequency (RF) links, etc. Code segments can be downloaded via computer networks such as the Internet, intranets, etc.

[0246] It should also be noted that the exemplary embodiments mentioned in this invention describe methods or systems based on a series of steps or apparatus. However, this invention is not limited to the order of the steps described above; that is, the steps can be performed in the order mentioned in the embodiments, or in a different order, or several steps can be performed simultaneously.

[0247] The above description is merely a specific embodiment of the present invention. Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, modules, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here. It should be understood that the protection scope of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present invention, and these modifications or substitutions should all be covered within the protection scope of the present invention.

Claims

1. A deep learning-based method for training injury treatment skills, characterized in that, Includes the following steps: Obtain the injury type and tool type selected by the user; A training list is generated based on the injury type and the tool type, and the training list includes several training subsets arranged sequentially. Receive the training start signal and enter the training phase; Training is performed in the order of the training subsets in the training list, and training data and training evaluation feedback information for each training subset are obtained. The evaluation feedback information includes improvement suggestions corresponding to each training subset. Based on the improvement suggestions, the corresponding training subset was retrieved for secondary training; The step of training according to the order of the training subsets in the training list and obtaining training data and training evaluation feedback information for each training subset also includes the following steps: The evaluator's display interface shows the name of the current training subset and the assessment subject; Obtain and display the training time of the current training subset and the total training time of the training phase in the evaluator's display interface; The display interface shows the various scoring items for the current training subset; Retrieve and display the score results of each scoring item in the current training subset; Retrieve and display suggestions for improvement for each current rating item; Establish a correspondence between various non-standard rescue actions and various types of improvement suggestions; Obtain the training video image data corresponding to each scoring item; Continuous training motion video data and posture accuracy training motion video data are extracted from the training video image data corresponding to the scoring items. The continuous training video data is input into the time series data deep learning model for processing to obtain the identification results of non-standard rescue actions in the continuous training. Based on the identification results of non-standard rescue actions in continuous training and the correspondence between non-standard rescue actions and improvement suggestions, corresponding improvement suggestions are obtained. The video data of the posture accuracy training action is input into the posture recognition deep learning model for processing, and then the recognition result of the non-standard rescue action of the posture accuracy training action is obtained. Based on the identification results of non-standard rescue actions during posture accuracy training and the correspondence between non-standard rescue actions and improvement suggestions, corresponding improvement suggestions are obtained.

2. The method for training injury treatment skills based on deep learning according to claim 1, characterized in that, The process of obtaining and displaying the scoring results of each scoring item in the current training subset also includes the following steps: Several scoring options are displayed in the corresponding positions of each scoring item on the evaluator's display interface; The rating results for each rating item are obtained based on the user's selection of the rating options in the corresponding positions of each rating item; Training videos of trainees are collected during the training phase; Extract the training video segments corresponding to the scoring items from the training videos; Analyze the erroneous operations that occur in each training video segment, mark the location of the erroneous action in the corresponding video frame image, and save the marked video frame as a reference video image to the training database; Obtain a video demonstrating the correct operating procedure; Extract the standard operation video segments corresponding to the scoring items from the standard operation video; When several scoring options are displayed in the corresponding position of each scoring item on the examiner's display interface, retrieve the reference video image and standard operation video clip corresponding to the scoring item; The corresponding reference video images and standardized operation video clips for each scoring item are displayed in the corresponding positions on the evaluator's display interface.

3. The method for training injury treatment skills based on deep learning according to claim 1, characterized in that, Before displaying the name of the current training subset and the assessment subject on the evaluator's display interface, the following is also included: Video images of trainees in various training subsets are acquired using image acquisition equipment; Wearable devices are used to obtain trainees' blood oxygen, heart rate, blood pressure, and body temperature data in various training subsets.

4. The method for training injury treatment skills based on deep learning according to claim 1, characterized in that, Before obtaining the identification results of non-standard rescue actions from continuous training video data after processing it into a time-series deep learning model, the process also includes: Video data of non-standard rescue actions in various continuous training movements were collected as the first training sample; The first training sample is labeled with the type of non-standard rescue movements in continuous training actions; Establish a deep learning model for the initial time series data; The deep learning model for time series data is obtained by training the initial model of the deep learning model for time series data using the first labeled training sample. Video data of non-standard rescue actions from various accuracy training movements were collected as a second training sample; The types of non-standard rescue actions used in the accuracy training were labeled as the second training samples; Establish an initial pose recognition deep learning model; The pose recognition deep learning model is obtained by training the initial pose recognition deep learning model with the labeled second training samples.

5. The method for training injury treatment skills based on deep learning according to claim 4, characterized in that, The step of extracting continuous training motion video data and posture accuracy training motion video data from the training video image data corresponding to the scoring items also includes the following steps: The feature pose image data corresponding to the pose accuracy training actions are collected as the third training sample; Establish an initial model for first pose recognition; The first pose recognition model is obtained by training the initial model of the first pose recognition with the third training sample; The trigger action image data and the end action image data corresponding to the continuous training actions are collected as the fourth training sample; The trigger action image data and the end action image data in the fourth training sample are labeled respectively; Establish an initial model for second pose recognition; The second pose recognition model is obtained by training the initial model of the second pose recognition using the labeled fourth training sample; The training video image data corresponding to the scoring items is input into the first posture recognition model to identify the image data corresponding to the feature posture as the posture accuracy training action video data; The training video image data corresponding to the scoring item is input into the second pose recognition model to identify the image data corresponding to the trigger action and the image data corresponding to the end action; The image data corresponding to the trigger action, the image data corresponding to the end action, and the image data in between are used as continuous training action video data.

6. The method for training injury treatment skills based on deep learning according to claim 4, characterized in that, The step of retrieving the corresponding training subset for secondary training based on the improvement suggestions also includes the following steps: Based on the improvement suggestions, extract training video clips corresponding to the improvement suggestions from the training videos of the trainees; The erroneous actions in the video clips are filtered out, and the positions of the erroneous actions are marked in the video clips to obtain the marked video clips; Extract the standardized action video segments that correspond to the improvement suggestions from the standardized operation videos; The marked video clips and the standard action video clips are displayed side-by-side in the display interface; Extract the human posture and first ground baseline of the standardized movements from video clips of standardized movements; Extracting erroneous human poses and a second ground baseline from labeled video clips; Using the first and second ground baselines as coincident references, the extracted human postures of standard movements and incorrect movements are merged and displayed in the same image; The process of obtaining and displaying improvement suggestions for each current rating item also includes the following steps: An input box for improvement suggestions is displayed in the corresponding position of each scoring item on the evaluator's display interface; Based on the user's input information in the corresponding improvement suggestion input box for each rating item, obtain the student's improvement suggestions for the corresponding rating item.

7. The method for training injury treatment skills based on deep learning according to claim 1, characterized in that, The types of injuries include lower limb fractures caused by falls from heights, right lower limb fractures caused by crushing buildings, facial injuries, and head injuries resulting in shock. The types of tools include towing bags, folding stretchers, first aid kits, unmanned stretcher carts, and roll-up stretchers.

8. The method for training injury treatment skills based on deep learning according to any one of claims 1 to 7, characterized in that, A training list is generated based on the injury type and the tool type. The training list includes several sequentially arranged training subsets and further includes the following steps: Based on the type of injury, obtain the steps for treating the corresponding injury and the order in which the steps are performed; Based on each step of the treatment of the corresponding injury, a training subset corresponding to each step is determined; The training list is formed by arranging the training subsets in the order of each step and the correspondence between each step and each training subset. The step of determining the training subset corresponding to each step of the treatment of the corresponding injury also includes the following steps: Select one step from each step whose corresponding training subset is not yet determined as the step to be matched; Retrieve the training subjects corresponding to the steps to be matched from the syllabus subject database as matching subjects; Generate a corresponding training subset using the matched subjects as the training content; The generated training subset is determined as the training subset that matches the step to be matched; Repeat the aforementioned process of generating and determining training subsets until all matching steps have completed the determination of the corresponding training subsets.

9. The method for training injury treatment skills based on deep learning according to any one of claims 1 to 7, characterized in that, The method further includes: Access the historical score query interface based on the historical score query command sent by the user. Obtain the query information entered by the user on the historical grades query interface; The corresponding score information will be displayed on the historical score query interface based on the query information.

10. A deep learning-based injury treatment skills training system, characterized in that, The system includes at least one processor, at least one memory, and computer program instructions stored in the memory, which, when executed by the processor, implement the method as described in any one of claims 1-9.

Images