Round window membrane puncture state detection method based on SlowFast algorithm

Abstract

The invention discloses a round window membrane puncture state detection method based on a SlowFast algorithm, and relates to the field of medical instruments. The problems that due to the fact that the size of a micro needle of an existing otology surgical robot is extremely small, sensing is difficult in the injection puncture process, and surgical precision is affected are solved. The method comprises the following steps: step 1, realizing position detection of a round window and a microneedle through a target detection module for detecting whether the microneedle is in a round window area or not; if the microneedle is in the round window area, the next step is carried out; if the micro-needle is not in the round window area, position detection of the round window and the micro-needle is continued; 2, recognizing the puncture state of the microneedle through a state recognition module, and determining whether the microneedle is successfully punctured or not through a judgment algorithm; wherein the state recognition module adopts a trained and optimized SlowFast algorithm model to recognize a microneedle puncture state; and the identification of the SlowFast model is filtered through a judgment algorithm. The device is used for detecting the puncture state of the round window membrane.

Description

Technical Field

[0001] The present invention relates to a puncture state detection method, in particular to a round window membrane puncture state detection method based on a SlowFast algorithm, which is used in the field of medical devices. Background Art

[0002] Inner ear drug administration and sampling surgery usually needs to be performed inside the tympanic cavity. For ear diseases that require surgery, inner ear injection and sampling operations are rarely used as an independent operation in the clinical practice of functional ear surgery due to factors such as operation accuracy and path trauma, which affects the early intervention treatment of functional ear diseases such as deafness, tinnitus, and vertigo. In addition, indirect drug administration treatment methods such as systemic drug administration and intratympanic injection have disadvantages such as poor efficacy and large side effects.

[0003] Traditional ear surgery robotic systems drill holes in the surface of the temporal bone to create channels for surgical instruments to enter the middle ear or inner ear to complete the operation. In contrast, the trans-auricular inner ear injection-sampling surgical robotic system uses a flexible end effector to reach the surgical target through the ear canal and achieve precise injection and tissue sampling of the inner ear through a micro needle.

[0004] This new surgical method is less invasive, has a shorter operation time, and has a lower risk of postoperative complications. However, since this type of surgery uses a micro-needle, the micro-needle is extremely small, with a diameter of ≤100μm, which makes it difficult to integrate traditional force sensors, resulting in difficulty in sensing during the injection and puncture process, which in turn affects the accuracy of the surgery.

[0005] In summary, the existing ear surgical robots have extremely small micro-needles, which makes it difficult to perceive during the injection and puncture process, thus affecting the accuracy of the surgery. Summary of the invention

[0006] The purpose of the present invention is to solve the problem that the existing ear surgical robot has difficulty in sensing during the injection and puncture process due to the extremely small size of the micro needle, which affects the surgical accuracy, and further provide a round window membrane puncture state detection method based on the SlowFast algorithm.

[0007] The technical solution of the present invention is:

[0008] A round window membrane puncture state detection method based on SlowFast algorithm, comprising the following steps:

[0009] Step 1: The target detection module is used to detect the position of the round window and the microneedle. The target detection module uses the trained and optimized YOLOv8 model to detect whether the microneedle is in the round window area. If the microneedle is in the round window area, the next step is to detect the puncture status. If the microneedle is no longer in the round window area, the position detection of the round window and the microneedle continues.

[0010] Step 2: Identify the microneedle puncture state through the state recognition module, and determine whether the microneedle puncture is successful through the judgment algorithm;

[0011] Among them, the state recognition module uses the trained and optimized SlowFast algorithm model to identify the microneedle puncture state, which is divided into three types: no contact, contact and penetration; and filters the recognition of the SlowFast network through the judgment algorithm.

[0012] Furthermore, before step 1, the following steps are included:

[0013] Step 01: First, read the real-time video frames collected by the endoscope camera through the endoscope camera;

[0014] Step 02: Then input the video frame into this detection method;

[0015] Step 03: Perform a Flag flag judgment again. The Flag flag is used to select the algorithm module to be processed. The Flag flag is initially set to False.

[0016] Step 04: If the Flag identifier is set to False, the algorithm enters the target detection module and executes step 1. Otherwise, it enters the state recognition module and executes step 2.

[0017] Furthermore, step one includes the following steps:

[0018] First: When the YOLOv8 model detects the round window position, the target detection module starts to determine whether the microneedle is within the round window area;

[0019] Secondly: If the microneedle is not located in the round window area, the video frame will prompt: "Not contacted" and display this video frame, and record the coordinates of the four vertices of the round window detection frame at this moment, so as to facilitate the subsequent cropping of the round window part; if the microneedle is located in the round window area, the flag Flag will be set to Ture, indicating that the algorithm will enter the state recognition module.

[0020] Furthermore, step 2 includes the following steps:

[0021] Step 21: Input a video clip consisting of 8 frames of sequential images into the SlowFast model;

[0022] Step 22: The output of the SlowFast model is the microneedle puncture status and its probability;

[0023] Step 2 and 3: Input the output of the model into the judgment algorithm for judgment:

[0024] Step 231: Determination of penetration status:

[0025] The judgment algorithm checks whether the classification output by the SlowFast model is puncture and whether the prediction probability is greater than or equal to 0.95;

[0026] If both conditions are met, the process enters the penetration judgment part. At this time, the last three frames of images are read continuously and made into three 8-frame videos. The SlowFast model is input for judgment: if in these three videos, the SlowFast model judges penetration λ times and the probability is greater than or equal to 0.95, the current state is considered to be a penetration state, and the penetration information is prompted on the frame image and the frame image is displayed, where λ is a configurable parameter.

[0027] If the SlowFast model does not judge the video as penetration for λ times in these three videos and the probability is greater than or equal to 0.95, the frame image is displayed directly;

[0028] Step 232: Determination of non-penetration status:

[0029] If the judgment algorithm does not enter the penetration judgment part, the judgment algorithm will check whether the classification output by the SlowFast model is contact and whether the predicted probability is greater than or equal to 0.9. If both conditions are met, it will enter the contact judgment part.

[0030] Furthermore, the 8-frame sequence of images in step 21 includes the current frame and the 7 frames stored sequentially before, each frame of the image is a round window area in its original video frame, and each frame of the image is adjusted to a uniform size through image interpolation.

[0031] Furthermore, when entering the contact judgment part in step 232, step 233 is entered. The judgment process of step 233 is as follows: at this time, the image frame after the current frame is read and made into an 8-frame video, and then input into the SlowFast model for judgment. If the SlowFast model judges that it is contact and the probability is greater than or equal to 0.9, it is considered that the current state has been entered, and the n_c_flag flag is set to False, the contact information is prompted on the frame image, and the frame image is displayed.

[0032] Furthermore, if the judgment condition does not meet step 233, then proceed to step 234:

[0033] Step 2, 3, 4: Untouched judgment part:

[0034] If the judgment algorithm does not enter the contact judgment part, the judgment algorithm will check whether the classification output by the SlowFast model is no contact and whether the predicted probability is greater than or equal to 0.9 and whether the n_c_flag flag is set to True. If these three conditions are met, the no contact information will be prompted on the frame image and the frame image will be displayed.

[0035] Further, the initial state of the n_c_flag flag in step 233 is set to True, and there is a condition for judging whether the n_c_flag flag is set to True in the no-contact judgment;

[0036] In the initial stage, the judgment algorithm enters the untouched judgment part through the untouched judgment condition;

[0037] When the judgment algorithm sets the n_c_flag flag to False, the judgment algorithm will no longer enter the non-contact judgment part, that is, the image will no longer prompt the non-contact information.

[0038] Compared with the prior art, the present invention has the following effects:

[0039] Since the image changes before and after microneedle puncture are small, it is difficult to accurately judge the process of microneedle puncture using image classification algorithms. Due to the complex environment around the round window membrane, the use of a single video recognition network will be disturbed by the surrounding environment, resulting in a series of misjudgments. This algorithm uses the target detection module to avoid misjudgments caused by external environmental interference when the microneedle has not entered the round window area; cropping the round window area before sending it to the video recognition network helps reduce the interference of a large amount of background noise, allowing it to focus on the changes in the round window membrane area, while reducing the amount of calculation of the video recognition network; filtering the results output by the video recognition network through the judgment algorithm further improves the overall judgment accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 It is the overall block diagram of the detection method of the present invention;

[0041] Figure 2 It is a block diagram of the target detection module of the present invention;

[0042] Figure 3 It is a block diagram of the state identification module of the present invention;

[0043] Figure 4 It is a block diagram of the judgment algorithm of the present invention. DETAILED DESCRIPTION

[0044] Specific implementation method 1: Combination Figures 1 to 4 To illustrate this embodiment, the embodiment includes the following steps:

[0045] Step 1: Detect the position of the round window and the microneedle through the target detection module;

[0046] The target detection module uses the trained and optimized YOLOv8 model to detect whether the microneedle is in the round window area. If the microneedle is in the round window area, the next step is to detect the puncture status. If the microneedle is no longer in the round window area, the position detection of the round window and the microneedle continues.

[0047] Step 2: Identify the microneedle puncture state through the state recognition module, and determine whether the microneedle puncture is successful through the judgment algorithm; the state recognition module uses the trained and optimized SlowFast algorithm model to identify the microneedle puncture state, which is divided into three types: no contact, contact and penetration; and filter the recognition of the SlowFast network through the judgment algorithm.

[0048] The round window membrane puncture state detection method based on the SlowFast model of this embodiment effectively makes up for the lack of perception of the robot system during the injection puncture process, thereby improving the safety and accuracy of the operation.

[0049] Specific implementation method 2: Combination Figure 1 To illustrate this implementation mode, before step 1, this implementation mode includes the following steps:

[0050] Step 01: First, read the real-time video frames collected by the endoscope camera through the endoscope camera;

[0051] Step 02: Then input the video frame into this detection method;

[0052] Step 03: Perform a Flag flag judgment again. The Flag flag is used to select the algorithm module to be processed. The Flag flag is initially set to False.

[0053] Step 04: If the Flag identifier is set to False, the algorithm enters the target detection module and executes step 1. Otherwise, it enters the state recognition module and executes step 2.

[0054] This arrangement can avoid misjudgment when the needle tip does not enter the round window area, thereby improving the robustness of the system. Other components and connection relationships are the same as those of the first or second specific implementation.

[0055] Specific implementation method three: Combination Figures 1 to 4 To illustrate this implementation, step 1 of this implementation includes the following steps:

[0056] First: When the YOLOv8 model detects the round window position, the target detection module starts to determine whether the microneedle is within the round window area;

[0057] Secondly: if the microneedle is not in the round window area, the video frame will prompt: "Not in contact" and display this video frame, and the coordinates of the four vertices of the round window detection frame at this moment will be recorded to facilitate subsequent cropping of the round window part;

[0058] If the microneedle is located in the round window area, the flag Flag is set to Ture, indicating that the algorithm will enter the state recognition module.

[0059] In this way, preprocessing by the target detection module can effectively avoid misjudgment caused by the movement of the round window model or the camera when the microneedle does not enter the round window area. At the same time, cropping the round window area before sending it to the state recognition module helps reduce the interference of a large amount of background and noise in the picture, focusing on the changes in the round window membrane area, and reducing the calculation amount of the state module. Other components and connection relationships are the same as those of the first or second specific implementation.

[0060] Specific implementation method four: Combination Figures 1 to 4 To illustrate this implementation, step 2 of this implementation includes the following steps:

[0061] Step 21: Input a video clip consisting of 8 frames of sequential images into the SlowFast model;

[0062] Step 22: The output of the SlowFast model is the microneedle puncture status and its probability;

[0063] Step 2 and 3: Input the output of the model into the judgment algorithm for judgment:

[0064] Step 231: Determination of penetration status:

[0065] The judgment algorithm checks whether the classification output by the SlowFast model is puncture and whether the prediction probability is greater than or equal to 0.95;

[0066] If both conditions are met, the process enters the penetration judgment part. At this time, the last three frames of images are read continuously and made into three 8-frame videos. The SlowFast model is input for judgment: if in these three videos, the SlowFast model judges penetration λ times and the probability is greater than or equal to 0.95, the current state is considered to be a penetration state, and the penetration information is prompted on the frame image and the frame image is displayed, where λ is a configurable parameter.

[0067] If the SlowFast model does not judge the video as penetration for λ times in these three videos and the probability is greater than or equal to 0.95, the frame image is displayed directly;

[0068] Step 232: Determination of non-penetration status:

[0069] If the judgment algorithm does not enter the penetration judgment part, the judgment algorithm will check whether the classification output by the SlowFast model is contact and whether the predicted probability is greater than or equal to 0.9. If both conditions are met, it will enter the contact judgment part.

[0070] This configuration can filter out the erroneous judgments output by the SlowFast model and improve the recognition accuracy. The other components and connection relationships are the same as any one of the specific implementation modes 1 to 3.

[0071] Specific implementation method five: Combination Figures 1 to 4 To illustrate this embodiment, the 8-frame sequence image in step 21 of this embodiment includes the current frame and the 7 frames stored sequentially before, each frame image is a circular window area in its original video frame, and each frame image is adjusted to a uniform size through image interpolation.

[0072] Such a configuration helps the state recognition module to focus on the round window membrane and reduce interference from background and noise. Other components and connection relationships are the same as any one of the specific embodiments 1 to 4.

[0073] Specific implementation method six: Combination Figures 1 to 4 To explain this embodiment, when the contact determination part is entered in step 232 of this embodiment, step 233 is entered. The determination process of step 233 is as follows:

[0074] At this time, the image frame after the current frame will be read and made into an 8-frame video, and then input into the SlowFast model for judgment. If the SlowFast model judges that it is contact and the probability is greater than or equal to 0.9, it is considered that the current state has entered the contact state, and the n_c_flag flag is set to False, and the contact information is prompted on the frame image and the frame image is displayed. Among them, the n_c_flag flag is different from the previous Flag flag.

[0075] Such an arrangement can improve the accuracy of the model in identifying the contact state. Other components and connection relationships are the same as any one of the specific implementation modes 1 to 5.

[0076] Specific implementation method seven: Combination Figures 1 to 4 To explain this embodiment, in step 232 of this embodiment, when the contact determination part is not entered, step 234 is entered. The determination process of step 234 is as follows:

[0077] Step 2, 3, 4: Untouched judgment part:

[0078] If the judgment algorithm does not enter the contact judgment part, the judgment algorithm will check whether the classification output by the SlowFast model is no contact and whether the predicted probability is greater than or equal to 0.9 and whether the n_c_flag flag is set to True. If these three conditions are met, the no contact information will be prompted on the frame image and the frame image will be displayed.

[0079] In this way, when the n_c_flag identifier is set to False, the judgment algorithm no longer enters the non-contact judgment part, which can reduce the misjudgment of non-contact and contact. The other components and connection relationships are the same as any one of the specific implementation methods one to six.

[0080] Specific implementation method eight: Combination Figures 1 to 4 To explain this embodiment, the initial state of the n_c_flag flag in step 233 of this embodiment is set to True, and there is a condition for judging whether the n_c_flag flag is set to True in the no-contact judgment;

[0081] In the initial stage, the judgment algorithm enters the untouched judgment part through the untouched judgment condition;

[0082] When the judgment algorithm sets the n_c_flag flag to False, the judgment algorithm will no longer enter the non-contact judgment part, that is, the image will no longer prompt the non-contact information.

[0083] With this arrangement, when the microneedle is judged to have contacted the round window membrane, the system only needs to determine whether the microneedle has penetrated the round window membrane. In general, the microneedle penetration process is a continuous process, that is, the microneedle penetration process is generally a sequential process from non-contact to contact and then to penetration, and no operation such as needle withdrawal will be performed in the middle. The other components and connection relationships are the same as any one of the specific embodiments one to seven.

[0084] Combination Figures 1 to 4 Describe the embodiments of the present invention:

[0085] First, a microscope camera was used to collect multiple videos of microneedles piercing the round window membrane model. Video segments of the three stages of non-contact, contact, and insertion were collected from each video, and each video segment consisted of 8 frames. The obtained video segments were sent to the SlowFast model for training, where the Slow branch input the last 4 frames of the video segment and the Fast branch input the entire video. In addition, images of the non-contact video segment were randomly collected and made into a target detection dataset and sent to the YOLOv8 model for training. The weights and parameters of the trained SlowFast model and YOLOv8 model were configured.

[0086] The needle tip gradually approaches the round window, and the microscope camera reads the video frame in real time. In the initial state, the Flag identifier is set to False. Therefore, the target detection module is entered first. The YOLOv8 model detects the position of the round window and begins to judge the position of the needle tip and the round window. Since the round window is basically inscribed in the annotation box of the round window in the dataset annotation. Therefore, the coordinates of the center point of the round window detection box are approximately used as the center of the round window, and half of the maximum value of the length and width of the round window detection box is approximately used as the long axis r of the round window. Calculate the distance between the four points of the needle tip detection box and the center point of the round window, and compare the distance value with the size of the long axis r of the round window. If its distance value is greater than the long axis r of the round window, it means that the needle tip has not entered the round window area. Therefore, the video frame prompts that the character has not been touched and displays this video frame, and records the coordinates of the four vertices of the round window detection box at this moment. When the needle tip approaches the round window further, if its distance value is less than the long axis r of the round window, it means that the needle tip has entered the round window area, and the system sets the identifier Flag to Ture, indicating that the algorithm enters the state recognition module.

[0087] After entering the state recognition module, the round window area in the current frame read by the microscope camera and the first 7 frames of the stored image is cropped using the average coordinates of the four vertices of the round window detection frame. The image size is uniformly adjusted to (224, 224) through bilinear interpolation and sequentially combined into an 8-frame video segment. The video segment is input into the SlowFast model, with the last 4 frames input into the Slow branch and the Fast branch input into the entire video segment. Finally, the output result of the SlowFast model is input into the judgment algorithm (λ in the algorithm is set to 1) to complete the real-time detection of the puncture state and generate a classification label.

[0088] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A round window membrane puncture status detection method based on SlowFast algorithm, characterized in that: It includes the following steps: Step 1: Detect the position of the round window and the microneedle through the target detection module; Among them, the target detection module uses the trained and optimized YOLOv8 model to detect whether the microneedle is in the round window area; If the microneedle is in the round window area, the next step is to detect the puncture status; If the microneedle is no longer in the round window area, continue to detect the position of the round window and the microneedle; Step 2: Identify the microneedle puncture state through the state recognition module, and determine whether the microneedle puncture is successful through the judgment algorithm; Among them, the state recognition module uses the trained and optimized SlowFast algorithm model to identify the microneedle puncture state, which is divided into three types: no contact, contact and penetration; and filters the recognition of the SlowFast network through the judgment algorithm.

2. A round window membrane puncture state detection method based on SlowFast algorithm according to claim 1, characterized in that: Before step 1, the following steps are included: Step 01: First, read the real-time video frames collected by the endoscope camera through the endoscope camera; Step 02: Then input the video frame into this detection method; Step 03: Perform a Flag flag judgment again. The Flag flag is used to select the algorithm module to be processed. The Flag flag is initially set to False. Step 04: If the Flag identifier is set to False, the algorithm enters the target detection module and executes step 1. Otherwise, it enters the state recognition module and executes step 2.

3. A round window membrane puncture state detection method based on SlowFast algorithm according to claim 2, characterized in that: Step one includes the following steps: first: When the YOLOv8 model detects the round window position, the object detection module starts to determine whether the microneedle is located in the round window area; Secondly: If the microneedle is not in the round window area, the video frame will prompt: "Not contacted" and display this video frame, and record the coordinates of the four vertices of the round window detection frame at this moment, so as to crop the round window part later; If the microneedle is located in the round window area, the flag Flag is set to Ture, indicating that the algorithm will enter the state recognition module.

4. The round window membrane puncture state detection method based on the SlowFast algorithm according to claim 3, characterized in that: Step 2 includes the following steps: Step 21: Input a video clip consisting of 8 frames of sequential images into the SlowFast model; Step 22: The output of the SlowFast model is the microneedle puncture status and its probability; Step 2 and 3: Input the output of the model into the judgment algorithm for judgment: Step 231: Determination of penetration status: The judgment algorithm checks whether the classification output by the SlowFast model is puncture and whether the prediction probability is greater than or equal to 0.95; If both conditions are met, the process enters the penetration judgment part. At this time, the last three frames of images are read continuously and made into three 8-frame videos. The SlowFast model is input for judgment: if in these three videos, the SlowFast model judges penetration λ times and the probability is greater than or equal to 0.95, the current state is considered to be a penetration state, and the penetration information is prompted on the frame image and the frame image is displayed, where λ is a configurable parameter. If the SlowFast model does not judge the video as penetration for λ times in these three videos and the probability is greater than or equal to 0.95, the frame image is displayed directly; Step 232: Determination of non-penetration status: If the judgment algorithm does not enter the penetration judgment part, the judgment algorithm will check whether the classification output by the SlowFast model is contact and whether the predicted probability is greater than or equal to 0.

9. If both conditions are met, it will enter the contact judgment part.

5. The round window membrane puncture state detection method based on the SlowFast algorithm according to claim 4, characterized in that: The 8-frame sequence of images in step 21 includes the current frame and the 7 frames stored sequentially before, each frame of the image is a round window area in its original video frame, and each frame of the image is adjusted to a uniform size through image interpolation.

6. A round window membrane puncture state detection method based on SlowFast algorithm according to claim 5, characterized in that: When the contact judgment part is entered in step 232, step 233 is entered. The judgment process of step 233 is as follows: At this time, the image frame after the current frame will be read and made into an 8-frame video, and then input into the SlowFast model for judgment. If the SlowFast model judges that it is contact and the probability is greater than or equal to 0.9, it is considered that the current state has been contacted, and the n_c_flag flag is set to False. The contact information is prompted on the frame image and the frame image is displayed.

7. A round window membrane puncture state detection method based on SlowFast algorithm according to claim 6, characterized in that: If the judgment condition does not meet step 233, go to step 234: Step 2, 3, 4: Untouched judgment part: If the judgment algorithm does not enter the contact judgment part, the judgment algorithm will check whether the classification output by the SlowFast model is no contact and whether the predicted probability is greater than or equal to 0.9 and whether the n_c_flag flag is set to True. If these three conditions are met, the no contact information will be prompted on the frame image and the frame image will be displayed.

8. The round window membrane puncture state detection method based on the SlowFast algorithm according to claim 7, characterized in that: The initial state of the n_c_flag flag in step 233 is set to True, and there is a condition to determine whether the n_c_flag flag is set to True in the no-contact judgment; In the initial stage, the judgment algorithm enters the untouched judgment part through the untouched judgment condition; When the judgment algorithm sets the n_c_flag flag to False, the judgment algorithm will no longer enter the non-contact judgment part, that is, the image will no longer prompt the non-contact information.

Images