Auxiliary training method, device and electronic device for post-extubation dysphagia

Abstract

The present invention belongs to the field of computer technology, and discloses an auxiliary training method, device and electronic device for post-extubation dysphagia. The method includes: performing facial calibration on the target patient; in response to the completion of facial calibration, collecting the target facial image; performing K-point positioning according to the target facial image; controlling a mechanical arm with an icylic acid cotton swab to perform K-point stimulation on the first K-point position and the second K-point position respectively; in response to the completion of K-point stimulation, guiding the target patient to perform preset oral actions and collecting the target facial video through a head-mounted VR device; performing action recognition on the target facial video to generate an action information set; and according to the action information set, guiding the target patient to perform actions through a head-mounted VR device. This implementation method can achieve effective auxiliary training for the target patient, thereby effectively alleviating the patient's dysphagia problem.

Description

Technical Field

[0001] Embodiments of the present disclosure relate to the field of computer technology, and in particular to auxiliary training methods, devices, and electronic equipment for post-extubation dysphagia. Background Art

[0002] Tracheal intubation is one of the most important measures to maintain the airway. However, due to the long-term compression of the tracheal tube and the air bag on the patient's mouth, tongue, throat, tracheal mucosa, and vocal cords, when the tracheal tube is removed, patients often experience post-extubation dysphagia (PED), which is manifested by coughing when drinking water, swallowing weakness, and wheezing after swallowing. If post-extubation dysphagia is not effectively intervened in time, it may cause patients to suffer from aspiration, aspiration pneumonia, and other problems.

[0003] The above information disclosed in this Background section is only for enhancement of understanding of the background of the inventive concept and therefore it may contain information that does not form the prior art that is already known to a person of ordinary skill in the art. Summary of the invention

[0004] The content of this disclosure is used to introduce concepts in a brief form, which will be described in detail in the detailed implementation section below. The content of this disclosure is not intended to identify the key features or essential features of the technical solution claimed for protection, nor is it intended to limit the scope of the technical solution claimed for protection.

[0005] Some embodiments of the present disclosure propose auxiliary training methods, devices and electronic devices for post-extubation dysphagia to solve one or more of the technical problems mentioned in the above background technology section.

[0006] In a first aspect, some embodiments of the present disclosure provide an auxiliary training method for post-extubation dysphagia, the method comprising: performing facial calibration on a target patient, wherein the target patient is a patient to be treated with assisted dysphagia therapy; in response to completion of the facial calibration, acquiring a target facial image, wherein the target facial image is a facial image acquired after the target patient is guided to open his mouth by a head-mounted VR device, and the head-mounted VR device is worn on the head of the target patient; performing K-point positioning according to the target facial image to obtain K-point position information, wherein the K-point position information includes: a first K-point position and a second K-point position; controlling a robotic arm holding an icy acid cotton swab to respectively locate the first K-point position and the second K-point position; The first K point and the second K point position are positioned to perform K-point stimulation; in response to completion of the K-point stimulation, the target patient is guided to perform a preset oral action through the head-mounted VR device, and a target facial video is collected, wherein the target facial video is a facial video of the target patient during the period of performing the preset oral action, and the action types of the preset oral action include: a first action type, a second action type, a third action type and a fourth action type; action recognition is performed on the target facial video to generate an action information set, wherein the action information in the action information set includes: an action type and an action matching degree; according to the action information set, the target patient is guided to perform actions through the head-mounted VR device.

[0007] In a second aspect, some embodiments of the present disclosure provide an auxiliary training device for post-extubation dysphagia, the device comprising: a facial calibration unit, configured to perform facial calibration on a target patient, wherein the target patient is a patient to be treated with assisted dysphagia treatment; a collection unit, configured to collect a target facial image in response to completion of facial calibration, wherein the target facial image is a facial image collected after the target patient is guided to open his mouth by a head-mounted VR device, and the head-mounted VR device is worn on the head of the target patient; a K-point positioning unit, configured to perform K-point positioning according to the target facial image to obtain K-point position information, wherein the K-point position information includes: a first K-point position and a second K-point position; a control unit, configured to control a robotic arm holding an icy acid cotton swab to respectively locate the first K-point and the second K-point positions of the target patient; K-point stimulation is performed at the K-point position and the above-mentioned second K-point position; an execution and acquisition unit is configured to, in response to the completion of the K-point stimulation, guide the above-mentioned target patient to perform a preset oral action through the above-mentioned head-mounted VR device, and to acquire a target facial video, wherein the above-mentioned target facial video is a facial video of the above-mentioned target patient during the period of performing the above-mentioned preset oral action, and the action types of the above-mentioned preset oral action include: a first action type, a second action type, a third action type and a fourth action type; an action recognition unit is configured to perform action recognition on the above-mentioned target facial video to generate an action information set, wherein the action information in the above-mentioned action information set includes: an action type and an action matching degree; an action guidance unit is configured to provide action guidance to the above-mentioned target patient through the above-mentioned head-mounted VR device according to the above-mentioned action information set.

[0008] In a third aspect, some embodiments of the present disclosure provide an electronic device comprising: one or more processors; a storage device on which one or more programs are stored, and when the one or more programs are executed by the one or more processors, the one or more processors implement the method described in any implementation manner of the above-mentioned first aspect.

[0009] In a fourth aspect, some embodiments of the present disclosure provide a computer-readable medium having a computer program stored thereon, wherein when the program is executed by a processor, the method described in any implementation manner of the above-mentioned first aspect is implemented.

[0010] The above-mentioned embodiments of the present disclosure have the following beneficial effects: through the auxiliary training method for post-extubation dysphagia applied to some embodiments of the present disclosure, effective auxiliary training for patients with dysphagia is achieved, and the dysphagia problem of the patient can be effectively alleviated. Specifically, first, the face of the target patient is calibrated, wherein the target patient is a patient to be treated with dysphagia. In practice, since the robot arm needs to be controlled later so that the ice acid cotton swab stimulates the K point, precise control is required, and thus the face of the target patient needs to be calibrated. Secondly, in response to the completion of the face calibration, the target facial image is collected, wherein the target facial image is a facial image collected after the target patient is guided to open his mouth by a head-mounted VR device, and the head-mounted VR device is worn on the head of the target patient. The head-mounted VR device can effectively guide the target patient to perform actions and display the screen in real time. Then, according to the target facial image, K point positioning is performed to obtain K point position information, wherein the K point position information includes: a first K point position and a second K point position. Further, the robotic arm with the glacial acid cotton swab is controlled to perform K-point stimulation on the first K-point position and the second K-point position, respectively. In this way, automated K-point stimulation is achieved. In addition, in response to the completion of the K-point stimulation, the target patient is guided to perform a preset oral action through the head-mounted VR device, and a target facial video is collected, wherein the target facial video is a facial video of the target patient during the period of performing the preset oral action, and the action type of the preset oral action includes: a first action type, a second action type, a third action type, and a fourth action type. Then, the target facial video is subjected to action recognition to generate an action information set, wherein the action information in the action information set includes: an action type and an action matching degree. Through action recognition, the type and standard degree of the oral action performed by the target patient can be effectively identified. Finally, according to the action information set, the target patient is guided to perform actions through the head-mounted VR device. In this way, it is ensured that the target patient can perform oral actions in a relatively standard manner, thereby achieving effective auxiliary training for the target patient, and thus effectively alleviating the swallowing disorder of the patient. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the accompanying drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and that components and elements are not necessarily drawn to scale.

[0012] Figure 1 is a flow chart of some embodiments of the auxiliary training method for post-extubation dysphagia according to the present disclosure;

[0013] Figure 2 It is a schematic diagram of the location of point K;

[0014] Figure 3 is a schematic structural diagram of some embodiments of the auxiliary training device for post-extubation dysphagia according to the present disclosure;

[0015] Figure 4 It is a schematic diagram of the structure of an electronic device suitable for implementing some embodiments of the present disclosure. DETAILED DESCRIPTION

[0016] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments set forth herein. On the contrary, these embodiments are provided to provide a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.

[0017] It should also be noted that, for ease of description, only the parts related to the invention are shown in the drawings. In the absence of conflict, the embodiments and features in the embodiments of the present disclosure can be combined with each other.

[0018] It should be noted that the concepts such as "first" and "second" mentioned in the present disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0019] It should be noted that the modifications of "one" and "plurality" mentioned in the present disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, it should be understood as "one or more".

[0020] The names of the messages or information exchanged between multiple devices in the embodiments of the present disclosure are only used for illustrative purposes and are not used to limit the scope of these messages or information.

[0021] The present disclosure will be described in detail below with reference to the accompanying drawings and in conjunction with embodiments.

[0022] refer to Figure 1 , shows a process 100 of some embodiments of the auxiliary training method for post-extubation dysphagia according to the present disclosure. The auxiliary training method for post-extubation dysphagia includes the following steps:

[0023] Step 101: perform facial calibration on a target patient.

[0024] In some embodiments, the execution subject (e.g., a computing device) of the auxiliary training method for post-extubation dysphagia can perform facial calibration on the target patient. The target patient is a patient who needs auxiliary treatment for dysphagia. In practice, the target patient can wear a head-mounted VR device on his head. Specifically, the head-mounted VR device can display the internal oral cavity image in real time, as well as the image used to guide the target patient's actions.

[0025] It should be noted that the above-mentioned computing device can be hardware or software. When the computing device is hardware, it can be implemented as a single terminal device. When the computing device is embodied as software, it can be installed in the hardware devices listed above. It can be implemented as a single software or software module. No specific limitation is made here.

[0026] In some optional implementations of some embodiments, the execution subject performs facial calibration on the target patient, which may include the following steps:

[0027] In response to the target patient confirming to start the facial calibration step through the head mounted VR device, the facial calibration steps described above are performed as follows:

[0028] The first sub-step is to acquire a first calibration image.

[0029] Among them, the above-mentioned first calibration image is a facial image when the above-mentioned target patient is guided to perform the first calibration action by the above-mentioned head-mounted VR device. In practice, the above-mentioned execution subject can collect the above-mentioned first calibration image through a camera set on a robotic arm. Specifically, the above-mentioned robotic arm is a multi-axis robotic arm, and the above-mentioned robotic arm also includes: a mechanical gripper. Among them, the above-mentioned mechanical gripper is used to fix the glacial acid cotton swab. The above-mentioned robotic arm is facing the face of the above-mentioned target patient so that the above-mentioned camera can collect the above-mentioned first calibration image. The glacial acid cotton swab can be a medical cotton swab that is fully soaked in citric acid and then frozen in a refrigerator. The first calibration action can be an eye-closing action. In practice, since the target patient often has swallowing difficulties, oral movements often cannot be performed effectively, and thus the eye-closing action is used for facial calibration.

[0030] The second sub-step is to perform multi-scale image feature extraction on the first calibration image to generate a first image feature set.

[0031] In practice, the execution state may use the first convolution layer, the second convolution layer, and the third convolution layer to extract image features from the first calibration image, respectively, to obtain the first image feature set. The convolution kernel of the first convolution layer is 2×2. The convolution kernel of the second convolution layer is 3×3. The convolution kernel of the third convolution layer is 1×1.

[0032] The third sub-step is to generate a first fused image feature according to the first image feature set.

[0033] In practice, the execution subject may perform feature fusion on each first image feature in the first image feature set through a Filter Concatenation layer to obtain the first fused feature map.

[0034] The fourth sub-step is to generate a first calibration action type for the first calibration image according to the first fused image feature.

[0035] In practice, the execution subject may input the first fused image feature into a classifier to obtain the first calibration action type. In practice, the classifier is composed of at least one fully connected layer connected in series.

[0036] A fifth sub-step, in response to the calibration action corresponding to the first calibration action type being the first calibration action, acquiring a second calibration image.

[0037] The second calibration image is a facial image of the target patient when the head-mounted VR device guides the target patient to perform the second calibration action. In practice, the execution subject may further collect the second calibration image through a camera set on the robotic arm. In practice, the second calibration action may be a head shaking action.

[0038] The sixth sub-step is to perform multi-scale image feature extraction on the second calibration image to generate a second image feature set.

[0039] In practice, the execution entity may perform multi-scale image feature extraction on the second calibration image through the first convolution layer, the second convolution layer, and the third convolution layer, respectively, to generate a second image feature set.

[0040] The seventh sub-step is to generate a second fused image feature according to the second image feature set.

[0041] In practice, the execution entity may perform feature fusion on each second image feature in the second image feature set through a Filter Concatenation layer to generate a second fused image feature.

[0042] An eighth sub-step is to generate a second calibration action type for the second calibration image according to the second fused image feature.

[0043] In practice, the execution subject may input the second fused image feature into a classifier to obtain the second calibration action type. In practice, the classifier is composed of at least one fully connected layer connected in series.

[0044] A ninth sub-step, in response to the calibration action corresponding to the second calibration action type being the second calibration action, generates calibration status information indicating that the facial calibration is completed.

[0045] Optionally, in response to the calibration action corresponding to the first calibration action type being other than the first calibration action, the first sub-step to the fourth sub-step are re-executed.

[0046] Optionally, in response to the calibration action corresponding to the second calibration action type being other than the second calibration action, the fifth to eighth sub-steps are re-executed.

[0047] Step 102: In response to the facial calibration being completed, a target facial image is acquired.

[0048] In some embodiments, the execution subject may capture the target facial image in response to the completion of facial calibration. In practice, the execution subject may capture the target facial image through a camera disposed on a robotic arm. The target facial image is a facial image captured after the target patient is guided to open his mouth by a head-mounted VR device. The head-mounted VR device is worn on the head of the target patient. In practice, the head-mounted VR device may play prompt information to guide the target patient to open his mouth, so as to guide the target patient to open his mouth.

[0049] Step 103, performing K point positioning according to the target facial image to obtain K point position information.

[0050] In some embodiments, the above-mentioned execution subject can perform K-point positioning according to the target facial image to obtain K-point position information. Wherein, the K-point position information includes: a first K-point position and a second K-point position. In practice, the K-point can be a K-point in the oral cavity. Specifically, the K-point is located at the height of the retromolar triangle, the central position of the palatoglossal arch and the pterygoid mandibular velum, and is located behind the intersection of the two dental lines. The first K-point position can be the position of the K-point on the left side of the oral cavity. The second K-point position can be the position of the K-point on the right side of the oral cavity. In practice, the above-mentioned execution subject can perform K-point positioning according to the target facial image through the YOLO (You Only Look Once) model to obtain the K-point position information.

[0051] Among them, K-point stimulation is a swallowing function stimulation method discovered by Japanese speech therapist Chieko Kojima in 2002. Specifically, the K-point is located at the height of the retromolar triangle, in the depression of the palatoglossal arch and the pterygoid mandibular velum. Touching the K-point can cause reflexive mouth opening. For patients with severe difficulty in opening the mouth, the K-point can be directly stimulated with a cotton swab, and the patient is more likely to produce a mouth opening movement. Repeated stimulation can make patients with difficulty in opening the mouth open their own mouths, and then swallowing movements will occur. The K-point stimulation operation method is simple and easy to learn, easy to operate, and highly feasible. In recent years, domestic scholars have applied K-point stimulation to patients with swallowing dysfunction caused by stroke and craniocerebral injury, which can improve patients' swallowing function and clinical efficacy, improve quality of life, shorten the retention time of nasogastric feeding tubes, and help improve patients' nutritional status. Therefore, this application intends to apply K-point stimulation to patients with endotracheal intubation PED to improve patients' swallowing function.

[0052] For the specific location of point K, please refer to Figure 2 Schematic diagram of the position of point K is shown.

[0053] In some optional implementations of some embodiments, the execution subject performs K point positioning according to the target facial image to obtain K point position information, which may include the following steps:

[0054] In the first step, multi-scale feature extraction is performed on the target facial image through a multi-scale feature extraction network to generate facial image features.

[0055] In practice, the above multi-scale feature extraction network adopts Feature Pyramid Network (FPN).

[0056] The second step is to determine the mouth position information based on the facial image features and the first positioning model.

[0057] The above-mentioned mouth position information includes: mouth confidence, mouth features and mouth corner key point position group. In practice, the above-mentioned first positioning model uses MobileNet-SSD as the basic network architecture to determine the mouth confidence and mouth features. Based on the mouth features, the above-mentioned execution body combines the SIFT (Scale Invariant Feature Transform) algorithm to determine the mouth corner key point position group.

[0058] In the third step, in response to the mouth confidence being greater than a preset confidence, candidate K point position information is determined based on the mouth features and the second positioning model.

[0059] The above-mentioned candidate K point position information includes: a first candidate K point position and a second candidate K point position. The above-mentioned multi-scale feature extraction network, the above-mentioned first positioning model and the above-mentioned second positioning model are included in the K point positioning model. In practice, the second positioning model and the above-mentioned first positioning model can adopt the same network structure. Specifically, the first positioning model is obtained by training in a supervised manner using image samples marked with mouth positions. The second positioning model is obtained by training in a supervised manner using image samples marked with K point positions.

[0060] The fourth step is to determine the first slope based on the above mouth corner key point position group.

[0061] In practice, the execution subject may determine the first slope according to the position group of the key points of the corners of the mouth through a two-point equation.

[0062] In the fifth step, a second slope is determined according to the first candidate K point position and the second candidate K point position.

[0063] In practice, the execution entity may determine the second slope according to the first candidate K point position and the second candidate K point position through a two-point equation.

[0064] Step 6, determining a target angle according to the first slope and the second slope;

[0065] Wherein, tan(β)=abs((k1-k2) / (1+k1×k2)). Wherein, β is the target angle, k1 is the first slope, and k2 is the second slope.

[0066] In the seventh step, in response to the target angle being smaller than the preset angle, the candidate K-point position information is determined as the K-point position information.

[0067] The above-mentioned content of "in some optional implementation methods of some embodiments" is an inventive point of the present disclosure. When locating the K points one by one, the positioning may not be accurate enough due to factors such as angles, thereby affecting the accuracy of subsequent K-point stimulation. Based on this, the present disclosure takes into account that the K points in the oral cavity often present axial symmetry. At the same time, the straight line formed by the two K points and the straight line formed by the corners of the mouth are often parallel to each other. Therefore, the present disclosure first determines the key point group of the corners of the mouth to determine the first slope. Then, the second slope is determined based on the first candidate K point position and the second candidate K point position. Finally, the angle between the two straight lines is determined in combination with the first slope and the second slope to determine whether the first candidate K point position and the second candidate K point position obtained by positioning are accurately positioned.

[0068] Step 104 , controlling the robotic arm holding the glacial acid cotton swab to perform K-point stimulation on the first K-point position and the second K-point position respectively.

[0069] In some implementations, the execution subject may control a mechanical arm with an ice acid cotton swab to perform K-point stimulation on the first K-point position and the second K-point position, respectively. In practice, the execution subject may control the movement of the mechanical arm so that the ice acid cotton swab performs K-point stimulation on the first K-point position and the second K-point position, respectively.

[0070] In some optional implementations of some embodiments, the execution subject controls the mechanical arm with the icy acid cotton swab to perform K-point stimulation on the first K-point position and the second K-point position respectively, which may include the following steps:

[0071] The first step is to generate a first moving route according to the initial position and the first K point position.

[0072] The initial position is the starting position of the robotic arm holding the glacial acid cotton swab. In practice, the execution subject can use the initial position as the starting position and the first K point position as the ending position to plan the first moving route through the host computer.

[0073] The second step is to control the robotic arm holding the glacial acid cotton swab to move along the first moving route to the first K point position to perform K point stimulation at a fixed frequency on the first stimulation position.

[0074] In practice, the above-mentioned execution entity can control the ice acid cotton swab on the robotic arm to stimulate the first stimulation position 8 to 10 times.

[0075] The third step is, in response to completion of the stimulation of the first K point, controlling the robotic arm holding the glacial acid cotton swab to move to the initial position to replace the glacial acid cotton swab.

[0076] In practice, the icy acid cotton swab clamped by the robotic arm can be replaced by manual replacement.

[0077] In the fourth step, in response to the replacement being completed, a second moving route is generated according to the initial position and the second K point position.

[0078] In practice, the execution subject may plan the second moving route through a host computer with the initial position as the starting position and the second K-point position as the ending position.

[0079] The fifth step is to control the robotic arm holding the glacial acid cotton swab to move along the second moving route to the second K point position to perform K point stimulation at a fixed frequency on the first stimulation position.

[0080] In practice, the above-mentioned execution subject can control the ice acid cotton swab on the robotic arm to stimulate the second stimulation position 8 to 10 times.

[0081] Step 105, in response to the completion of the K point stimulation, the target patient is guided to perform preset oral actions through the head-mounted VR device, and a target facial video is collected.

[0082] In some embodiments, in response to the completion of K-point stimulation, the execution subject may guide the target patient to perform preset oral actions and collect target facial videos through a head-mounted VR device. The target facial video is a facial video of the target patient during the period of performing the preset oral actions, and the action types of the preset oral actions include: a first action type, a second action type, a third action type, and a fourth action type. The first action type represents a tongue muscle training action type. The second action type represents a masticatory muscle training action type. The third action type represents a swallowing training action type. The fourth action type represents a vocal cord closure and laryngeal lifting training action type. The execution subject may collect the target facial video through a camera provided on the robotic arm.

[0083] Step 106: Perform action recognition on the target facial video to generate an action information set.

[0084] In some embodiments, the execution subject may perform action recognition on the target facial video to generate an action information set. The action information in the action information set includes: action type and action matching degree. The action matching degree represents the action matching degree between the action corresponding to the action type and the standard action. In practice, the execution state may perform action recognition on the target facial video through the YOLO model to generate an action information set.

[0085] In some optional implementations of some embodiments, the execution subject performs action recognition on the target facial video to generate an action information set, which may include the following steps:

[0086] In the first step, temporal features are extracted from the target facial video to generate a facial video feature sequence.

[0087] In practice, the execution entity may extract temporal features from the target facial video through an RNN (Recurrent Neural Network) model to generate a facial video feature sequence.

[0088] In the second step, in response to the action type of the above-mentioned preset oral action being the above-mentioned first action type, the above-mentioned action information set is generated according to the above-mentioned facial video feature sequence and the tongue muscle action recognition model.

[0089] The third step is to generate the action information set according to the facial video feature sequence and the masticatory muscle action recognition model in response to the action type of the preset oral action being the second action type.

[0090] The fourth step is to generate the action information set according to the facial video feature sequence and the swallowing action recognition model in response to the action type of the preset oral action being the third action type.

[0091] The fifth step is to generate the action information set according to the facial video feature sequence and the throat action recognition model in response to the action type of the preset oral action being the fourth action type.

[0092] Among them, the tongue muscle action recognition model, the masticatory muscle action recognition model, the swallowing action recognition model and the throat action recognition model share a facial video feature depth extraction model and a key point extraction model. The facial video feature depth extraction model adopts a residual neural network model. The key point extraction model adopts a MobileNet model. Among them, the facial video feature depth extraction model and the key point extraction model are connected in series. The tongue muscle action recognition model also includes: a first classifier and a first action matching degree determination model. The masticatory muscle action recognition model also includes: a second classifier and a second action matching degree determination model. The swallowing action recognition model also includes: a third classifier and a third action matching degree determination model. The throat action recognition model also includes: a fourth classifier and a fourth action matching degree determination model. The first classifier is used to classify the specific action type corresponding to the tongue muscle action, such as the tongue against the upper palate action, the tongue against the lower jaw action, the tongue stretching forward and backward action, etc. The second classifier is used to classify the specific action type corresponding to the masticatory muscle action, such as the cheek puffing action, the tooth biting action, the tooth showing action, etc. The third classifier is used to classify the specific action type corresponding to the swallowing action, such as the throat movement action. The fourth classifier corresponds to a specific action type of the throat action, for example, an upper throat movement action, a lower throat movement action, etc. The first action matching degree determination model, the second action matching degree determination model, the third action matching degree determination model, and the fourth action matching degree determination model respectively pre-store standard actions corresponding to different action types, and the specific standard actions are stored in the form of a feature point set. After the classifier determines the specific action type, the action matching degree determination model can take out the feature points corresponding to the standard action matching the specific action type, combine them, and calculate the similarity as the corresponding action matching degree.

[0093] The above-mentioned content of "in some optional implementations of some embodiments", as an inventive point of the present disclosure, realizes the effective recognition of actions. In practice, in order to improve the effectiveness of auxiliary training, patients are often required to perform auxiliary training as much as possible according to standard actions. However, due to problems such as swallowing disorders, patients often have problems such as deformation of movements when performing movements, which in turn affects the quality of auxiliary training. At the same time, different action types are often composed of multiple fine movements. For example, the tongue muscle movement is composed of the tongue against the upper palate movement, the tongue against the lower jaw movement, and the tongue stretching forward and backward movement. Based on this, the present disclosure first sets the tongue muscle movement recognition model, the masticatory muscle movement recognition model, the swallowing movement recognition model and the throat movement recognition model according to the action type (the first action type, the second action type, the third action type and the fourth action type), so as to realize more accurate recognition of detailed actions under the first action type, the second action type, the third action type and the fourth action type. In addition, by setting the action matching degree determination model (the first action matching degree determination model, the second action matching degree determination model, the third action matching degree determination model and the fourth action matching degree determination model), the target patient's execution action and the standard action are matched to determine the target patient's action execution standard. In this way, the target patient's action can be accurately identified, so as to ensure that the target patient performs the action as standardized as possible, thereby improving the effect of auxiliary training.

[0094] Step 107, providing action guidance to the target patient through a head-mounted VR device according to the action information set.

[0095] In some embodiments, the above-mentioned execution entity can provide action guidance to the target patient through a head-mounted VR device based on the action information set.

[0096] In some optional implementations of some embodiments, the execution subject guides the target patient to perform actions through a head-mounted VR device according to the action information set, which may include the following steps:

[0097] In the first step, action information including an action matching degree less than a preset action matching degree is screened out from the above action information set as candidate action information to obtain a candidate action information set.

[0098] The second step is to display, for each candidate action information in the above-mentioned candidate action information set, the standard action corresponding to the above-mentioned candidate action information on the secondary content display interface of the above-mentioned head-mounted VR device to provide action guidance for the above-mentioned target patient.

[0099] Among them, the above-mentioned head-mounted VR device also includes: a main content display interface, and the above-mentioned main content display interface is used to display the above-mentioned target facial video in real time.

[0100] The above-mentioned embodiments of the present disclosure have the following beneficial effects: through the auxiliary training method for post-extubation dysphagia applied to some embodiments of the present disclosure, effective auxiliary training for patients with dysphagia is achieved, and the dysphagia problem of the patient can be effectively alleviated. Specifically, first, the face of the target patient is calibrated, wherein the target patient is a patient to be treated with dysphagia. In practice, since the robot arm needs to be controlled later so that the ice acid cotton swab stimulates the K point, precise control is required, and thus the face of the target patient needs to be calibrated. Secondly, in response to the completion of the face calibration, the target facial image is collected, wherein the target facial image is a facial image collected after the target patient is guided to open his mouth by a head-mounted VR device, and the head-mounted VR device is worn on the head of the target patient. The head-mounted VR device can effectively guide the target patient to perform actions and display the screen in real time. Then, according to the target facial image, K point positioning is performed to obtain K point position information, wherein the K point position information includes: a first K point position and a second K point position. Further, the robotic arm with the glacial acid cotton swab is controlled to perform K-point stimulation on the first K-point position and the second K-point position, respectively. In this way, automated K-point stimulation is achieved. In addition, in response to the completion of the K-point stimulation, the target patient is guided to perform a preset oral action through the head-mounted VR device, and a target facial video is collected, wherein the target facial video is a facial video of the target patient during the period of performing the preset oral action, and the action type of the preset oral action includes: a first action type, a second action type, a third action type, and a fourth action type. Then, the target facial video is subjected to action recognition to generate an action information set, wherein the action information in the action information set includes: an action type and an action matching degree. Through action recognition, the type and standard degree of the oral action performed by the target patient can be effectively identified. Finally, according to the action information set, the target patient is guided to perform actions through the head-mounted VR device. In this way, it is ensured that the target patient can perform oral actions in a relatively standard manner, thereby achieving effective auxiliary training for the target patient, and thus effectively alleviating the swallowing disorder of the patient.

[0101] Further references Figure 3 As an implementation of the methods shown in the above figures, the present disclosure provides some embodiments of an auxiliary training device for post-extubation dysphagia, and these device embodiments are Figure 1 Corresponding to the method embodiments shown, the auxiliary training device for post-extubation dysphagia can be specifically applied to various electronic devices.

[0102] like Figure 3As shown, in some embodiments, the auxiliary training device 300 for post-extubation dysphagia includes: a facial calibration unit 301, a collection unit 302, a K-point positioning unit 303, a control unit 304, an execution and collection unit 305, an action recognition unit 306, and an action guidance unit 307. The facial calibration unit 301 is configured to perform facial calibration on the target patient, wherein the target patient is a patient to be treated with dysphagia auxiliary therapy; the collection unit 302 is configured to collect the target facial image in response to the completion of facial calibration, wherein the target facial image is a facial image collected after the target patient is guided to open his mouth by a head-mounted VR device, and the head-mounted VR device is worn on the head of the target patient; the K-point positioning unit 303 is configured to perform K-point positioning according to the target facial image to obtain K-point position information, wherein the K-point position information includes: a first K-point position and a second K-point position; the control unit 304 is configured to control a mechanical arm with an icylic acid cotton swab to perform K-point puncture on the first K-point position and the second K-point position, respectively. Stimulation; the execution and acquisition unit 305 is configured to, in response to the completion of the K point stimulation, guide the target patient to perform the preset oral action and collect the target facial video through the head-mounted VR device, wherein the target facial video is the facial video of the target patient during the period of performing the preset oral action, and the action types of the preset oral action include: a first action type, a second action type, a third action type and a fourth action type; the action recognition unit 306 is configured to perform action recognition on the target facial video to generate an action information set, wherein the action information in the action information set includes: an action type and an action matching degree; the action guidance unit 307 is configured to provide action guidance to the target patient through the head-mounted VR device according to the action information set.

[0103] It is understandable that the units described in the auxiliary training device 300 for post-extubation dysphagia are similar to those described in the reference Figure 1 Therefore, the operations, features and beneficial effects described above for the method are also applicable to the auxiliary training device 300 for post-extubation dysphagia and the units contained therein, and will not be described in detail here.

[0104] Reference below Figure 4 , which shows a structural schematic diagram of an electronic device (eg, a computing device) 400 suitable for implementing some embodiments of the present disclosure. Figure 4 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0105] like Figure 4As shown, the electronic device 400 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 401, which can perform various appropriate actions and processes according to a program stored in a read-only memory 402 or a program loaded from a storage device 408 to a random access memory 404. Various programs and data required for the operation of the electronic device 400 are also stored in the random access memory 404. The processing device 401, the read-only memory 402, and the random access memory 403 are connected to each other via a bus 404. The input / output interface 402 is also connected to the bus 404.

[0106] Typically, the following devices may be connected to the I / O interface 402: input devices 406 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; output devices 407 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; storage devices 408 including, for example, a magnetic tape, a hard disk, etc.; and communication devices 409. The communication devices 409 may allow the electronic device 400 to communicate with other devices wirelessly or by wire to exchange data. Although Figure 4 The electronic device 400 is shown with various devices, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead. Figure 4 Each block shown in the figure may represent one device, or may represent multiple devices as required.

[0107] In particular, according to some embodiments of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, some embodiments of the present disclosure include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In some such embodiments, the computer program can be downloaded and installed from the network through the communication device 409, or installed from the storage device 408, or installed from the read-only memory 402. When the computer program is executed by the processing device 401, the above-mentioned functions defined in the method of some embodiments of the present disclosure are executed.

[0108] It should be noted that the computer-readable medium recorded in some embodiments of the present disclosure may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In some embodiments of the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In some embodiments of the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. The computer readable signal medium may also be any computer readable medium other than a computer readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0109] In some embodiments, the client and the server may communicate using any currently known or future developed network protocol such as HTTP (Hyper Text Transfer Protocol), and may be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), an internet (e.g., the Internet), and a peer-to-peer network (e.g., an ad hoc peer-to-peer network), as well as any currently known or future developed network.

[0110] The computer-readable medium may be included in the electronic device; or it may exist independently without being assembled into the electronic device. The computer-readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device: performs facial calibration on the target patient, wherein the target patient is a patient to undergo assisted treatment for dysphagia; in response to the completion of facial calibration, collects a target facial image, wherein the target facial image is a facial image collected after the target patient is guided to open his mouth by a head-mounted VR device, and the head-mounted VR device is worn on the head of the target patient; performs K-point positioning according to the target facial image to obtain K-point position information, wherein the K-point position information includes: a first K-point position and a second K-point position; controls a robotic arm holding an icy acid cotton swab to respectively locate the first K-point position and the second K-point position. K-point stimulation is performed at a K-point position and the second K-point position; in response to completion of the K-point stimulation, the target patient is guided to perform a preset oral action through the head-mounted VR device, and a target facial video is collected, wherein the target facial video is a facial video of the target patient during the period of performing the preset oral action, and the action types of the preset oral action include: a first action type, a second action type, a third action type and a fourth action type; action recognition is performed on the target facial video to generate an action information set, wherein the action information in the action information set includes: an action type and an action matching degree; according to the action information set, the target patient is guided to perform actions through the head-mounted VR device.

[0111] Computer program code for performing the operations of some embodiments of the present disclosure may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0112] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0113] The units described in some embodiments of the present disclosure may be implemented by software or by hardware. The described units may also be provided in a processor, for example, may be described as: a processor including a facial calibration unit, an acquisition unit, a K-point positioning unit, a control unit, an execution and acquisition unit, an action recognition unit, and an action guidance unit. The names of these units do not constitute limitations on the units themselves in certain cases, for example, the acquisition unit may also be described as a unit that "acquires a target facial image in response to completion of facial calibration, wherein the target facial image is a facial image acquired after the target patient is guided to open his mouth by a head-mounted VR device, and the head-mounted VR device is worn on the head of the target patient".

[0114] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0115] The above descriptions are only some preferred embodiments of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of the invention involved in the embodiments of the present disclosure is not limited to the technical solutions formed by a specific combination of the above-mentioned technical features, but should also cover other technical solutions formed by any combination of the above-mentioned technical features or their equivalent features without departing from the above-mentioned inventive concept. For example, the above-mentioned features are replaced with the technical features with similar functions disclosed in the embodiments of the present disclosure (but not limited to) and the technical solutions formed.

Claims

1. An auxiliary training device for post-extubation dysphagia, comprising: A facial calibration unit, configured to perform facial calibration on a target patient, wherein the target patient is a patient who is to undergo dysphagia auxiliary therapy; The acquisition unit is configured to acquire a target facial image in response to completion of facial calibration, wherein the target facial image is a facial image acquired after the target patient is guided to open his mouth by a head-mounted VR device, and the head-mounted VR device is worn on the head of the target patient; A K point positioning unit is configured to perform K point positioning according to the target facial image to obtain K point position information, wherein the K point position information includes: a first K point position and a second K point position; A control unit is configured to control a mechanical arm having an icylic acid cotton swab to perform K-point stimulation on the first K-point position and the second K-point position respectively; an execution and acquisition unit, configured to guide the target patient to perform a preset oral action through the head-mounted VR device in response to completion of the K-point stimulation, and to acquire a target facial video, wherein the target facial video is a facial video of the target patient during the period of performing the preset oral action, and the action types of the preset oral action include: a first action type, a second action type, a third action type, and a fourth action type; An action recognition unit is configured to perform action recognition on the target facial video to generate an action information set, wherein the action information in the action information set includes: an action type and an action matching degree; The action guidance unit is configured to provide action guidance to the target patient through the head-mounted VR device according to the action information set.

2. The device according to claim 1, wherein: The facial calibration of the target patient comprises: In response to the target patient confirming to start the facial calibration step through the head-mounted VR device, the facial calibration step described below is performed: Acquire a first calibration image, wherein the first calibration image is a facial image of the target patient when the head-mounted VR device guides the target patient to perform a first calibration action; Performing multi-scale image feature extraction on the first calibration image to generate a first image feature set; generating a first fused image feature according to the first image feature set; generating a first calibration action type for the first calibration image according to the first fused image feature; In response to the calibration action corresponding to the first calibration action type being the first calibration action, acquiring a second calibration image, wherein the second calibration image is a facial image of the target patient when the head-mounted VR device guides the target patient to perform the second calibration action; performing multi-scale image feature extraction on the second calibration image to generate a second image feature set; generating a second fused image feature according to the second image feature set; generating a second calibration action type for the second calibration image according to the second fused image feature; In response to the calibration action corresponding to the second calibration action type being the second calibration action, calibration status information indicating that facial calibration is completed is generated.

3. The device according to claim 2, wherein: The step of providing action guidance to the target patient through the head mounted VR device according to the action information set includes: Filtering out action information including an action matching degree less than a preset action matching degree from the action information set as candidate action information, and obtaining a candidate action information set; For each candidate action information in the candidate action information set, the standard action corresponding to the candidate action information is displayed on the secondary content display interface of the head-mounted VR device to provide action guidance for the target patient, wherein the head-mounted VR device also includes: a main content display interface, and the main content display interface is used to display the target facial video in real time.

4. The device according to claim 3, wherein: The step of performing K point positioning according to the target facial image to obtain K point position information includes: Performing multi-scale feature extraction on the target facial image through a multi-scale feature extraction network to generate facial image features; Determine the mouth position information according to the facial image features and the first positioning model, wherein the mouth position information includes: mouth confidence, mouth features and a mouth corner key point position group; In response to the mouth confidence being greater than a preset confidence, determining candidate K-point position information according to the mouth feature and the second positioning model, wherein the candidate K-point position information includes: a first candidate K-point position and a second candidate K-point position, and the multi-scale feature extraction network, the first positioning model, and the second positioning model are included in the K-point positioning model; Determining a first slope according to the mouth corner key point position group; Determining a second slope according to the first candidate K point position and the second candidate K point position; determining a target angle according to the first slope and the second slope; In response to the target angle being smaller than a preset angle, the candidate K-point position information is determined as the K-point position information.

Images