A round window membrane puncture state detection method based on a slowfast algorithm

Abstract

The application discloses a round window membrane puncture state detection method based on a SlowFast algorithm and relates to the field of medical devices.The application solves the problem that the existing otological surgery robot is difficult to perceive in the injection puncture process due to the extremely small size of the micro-needle, and the problem of affecting the operation precision.The step one of the application is to realize the position detection of the round window and the micro-needle through a target detection module, so as to detect whether the micro-needle is in the round window area; if the micro-needle is in the round window area, the next step is performed; if the micro-needle is not in the round window area, the position detection of the round window and the micro-needle is continuously performed; the step two is to recognize the puncture state of the micro-needle through a state recognition module, and to determine whether the micro-needle is successfully punctured through a judgment algorithm; wherein the state recognition module adopts a SlowFast algorithm model which is trained and optimized to recognize the puncture state of the micro-needle; and the recognition of the SlowFast model is filtered through the judgment algorithm.The application is used for the round window membrane puncture state detection.

Description

TECHNICAL FIELD

[0001] The application relates to a puncture state detection method, in particular to a round window membrane puncture state detection method based on a SlowFast algorithm, and is used in the field of medical instruments. BACKGROUND

[0002] In-ear administration-sampling surgery usually needs to be performed inside the tympanic cavity, and for otological diseases requiring surgery, in-ear injection and sampling operations are rarely used as an independent operation in functional ear surgery due to factors such as operation precision and path trauma, which affects the early intervention treatment of functional ear diseases such as deafness, tinnitus and vertigo, and the indirect drug treatment methods such as systemic administration and intratympanic injection have the disadvantages of poor curative effect and large side effects.

[0003] In contrast, the trans-ear in-ear injection-sampling surgical robot system uses a flexible end effector to directly reach the surgical target position through the ear canal and achieve precise injection and tissue sampling in the inner ear through a micro needle.

[0004] This new surgical method has less trauma, shorter operation time and lower risk of postoperative complications. However, since this surgery uses a micro needle, the micro needle is extremely small in size, with a diameter of less than or equal to 100 microns, which makes it difficult to integrate traditional force sensors, leading to difficulty in perception during the injection puncture process, which in turn affects the surgical precision.

[0005] In summary, the existing otological surgical robot has the problem of affecting the surgical precision due to the difficulty in perception during the injection puncture process caused by the extremely small size of the micro needle. SUMMARY

[0006] The purpose of the present application is to solve the problem of affecting the surgical precision due to the difficulty in perception during the injection puncture process caused by the extremely small size of the micro needle of the existing otological surgical robot, and to provide a round window membrane puncture state detection method based on a SlowFast algorithm.

[0007] The technical solution of the present application is:

[0008] A round window membrane puncture state detection method based on a SlowFast algorithm, comprising the following steps:

[0009] Step one: the position detection of round window and microneedle is realized by a target detection module; wherein the target detection module adopts a YOLOv8 model optimized by training to detect whether the microneedle is in the round window area; if the microneedle is in the round window area, the next step of puncture state detection is performed; if the microneedle is not in the round window area, the position detection of round window and microneedle is continued;

[0010] Step two: the microneedle puncture state is recognized by a state recognition module, and a judgment algorithm is used to determine whether the microneedle is punctured successfully;

[0011] Wherein the state recognition module adopts a SlowFast algorithm model optimized by training to recognize the microneedle puncture state, which is divided into three states: no contact, contact and penetration; and the SlowFast network recognition is filtered by the judgment algorithm.

[0012] Further, before step one, the following steps are included:

[0013] Step 01: first read the real-time video frame collected by the endoscope camera through the endoscope camera;

[0014] Step 02: then input the video frame into the detection method;

[0015] Step 03: then judge a Flag symbol, which is used to select the algorithm module to be processed, and the Flag symbol is initially set to False;

[0016] Step 04: if the Flag symbol is set to False, the algorithm enters the target detection module and executes step one, otherwise it enters the state recognition module and executes step two.

[0017] Further, step one includes the following steps:

[0018] First: when the YOLOv8 model detects the round window position, the target detection module starts to judge whether the microneedle is in the round window area;

[0019] Second: if the microneedle is not in the round window area, the video frame prompts the "no contact" character and displays the video frame, and records the four vertex coordinates of the round window detection frame at this moment, so as to crop the round window part later; if the microneedle is in the round window area, the symbol Flag is set to True, which means that the algorithm will enter the state recognition module.

[0020] Further, step two includes the following steps:

[0021] Step two one: input a video segment composed of 8 frame sequence images into the SlowFast model;

[0022] Step two two: the output of the SlowFast model is the state of the microneedle puncture and its probability;

[0023] Step two three: input the output of the model into the judgment algorithm for judgment:

[0024] Step two three one: judgment of the penetration state:

[0025] The judgment algorithm checks whether the classification output by the SlowFast model is penetration and whether the predicted probability is greater than or equal to 0.95;

[0026] If both conditions are met, enter the penetration judgment part; at this time, read the image of the next 3 frames continuously and make 3 8-frame videos; input the SlowFast model for judgment: if the SlowFast model judges penetration for λ times and the probability is greater than or equal to 0.95 in the 3 videos, it is considered that the current state is a penetration state, and the frame picture is prompted with penetration information and displayed, wherein λ is a settable parameter;

[0027] If the SlowFast model does not judge penetration for λ times and the probability is greater than or equal to 0.95 in the 3 videos, the frame picture is directly displayed;

[0028] Step two three two: judgment of the non-penetration state:

[0029] If the judgment algorithm does not enter the penetration judgment part, the judgment algorithm will check whether the classification output by the SlowFast model is contact and whether the predicted probability is greater than or equal to 0.9, and if both conditions are met, enter the contact judgment part.

[0030] Further, the 8-frame sequence images in step two one include the current frame and the sequentially stored 7 frames, each frame image is the circular window region in its original video frame, and each frame image is adjusted to a uniform size by image interpolation.

[0031] Further, when entering the contact judgment part in step two three two, step two three three is entered, and the judgment process of step two three three is as follows: at this time, read the image of the next 1 frame and make 1 8-frame video, then input the SlowFast model for judgment, if the SlowFast model judges contact and the probability is greater than or equal to 0.9, it is considered that the current state has entered the contact state, the n_c_flag flag is set to False, and the frame picture is prompted with contact information and displayed.

[0032] Further, if the judgment condition does not meet step two three three, step two three four is entered:

[0033] Step two three four: non-contact judgment part:

[0034] If the judgment algorithm does not enter the contact judgment part, the judgment algorithm will check whether the classification output by the SlowFast model is non-contact, whether the prediction probability is greater than or equal to 0.9, and whether the n_c_flag flag is set to True. If all three conditions are met, non-contact information is prompted on the frame picture and the frame picture is displayed.

[0035] Further, the n_c_flag flag in step 2333 is initially set to True, and there is a condition in the non-contact judgment to determine whether the n_c_flag flag is set to True.

[0036] In the initial stage, the judgment algorithm enters the non-contact judgment part through the non-contact judgment condition;

[0037] When the judgment algorithm sets the n_c_flag flag to False, the judgment algorithm will no longer enter the non-contact judgment part, that is, the non-contact information will not be prompted on the image.

[0038] Compared with the prior art, the present application has the following effects:

[0039] Because the image changes little before and after microneedle puncture, it is difficult to accurately judge the microneedle puncture process by using an image classification algorithm. Because the environment around the round window membrane is complex, a single video recognition network will be disturbed by the surrounding environment and cause a series of misjudgments. The algorithm uses a target detection module to avoid misjudgments caused by external environmental disturbances when the microneedle does not enter the round window area; cutting the round window area and then sending it into the video recognition network helps to reduce the interference of a large amount of background noise, focuses on the changes in the round window membrane area, and reduces the computational load of the video recognition network; the judgment algorithm filters the results output by the video recognition network, further improving the overall judgment accuracy. BRIEF DESCRIPTION OF DRAWINGS

[0040] Figure 1 is the overall block diagram of the detection method of the present application;

[0041] Figure 2 is the block diagram of the target detection module of the present application;

[0042] Figure 3 is the block diagram of the state recognition module of the present application;

[0043] Figure 4 is the block diagram of the judgment algorithm of the present application. DETAILED DESCRIPTION

[0044] Specific implementation one: combined with Figures 1 to 4 To illustrate this embodiment, the embodiment includes the following steps:

[0045] Step one: the position detection of the round window and the microneedle is realized by a target detection module;

[0046] The target detection module adopts a YOLOv8 model optimized by training to detect whether the microneedle is in the round window area; if the microneedle is in the round window area, the next step of the puncture state detection is performed; if the microneedle is not in the round window area, the position detection of the round window and the microneedle is continued;

[0047] Step two: the puncture state of the microneedle is recognized by a state recognition module, and a judgment algorithm is used to determine whether the microneedle is punctured successfully; the state recognition module adopts a SlowFast algorithm model optimized by training to recognize the puncture state of the microneedle, which is divided into three states: no contact, contact and penetration; and the judgment algorithm is used to filter the recognition of the SlowFast network.

[0048] The round window film puncture state detection method based on the SlowFast model of the embodiment effectively makes up for the insufficient perception of the robot system in the injection puncture process, thereby improving the safety and accuracy of the operation.

[0049] Specific implementation method two: combined Figure 1 In this embodiment, before step one, the following steps are included:

[0050] Step 01: first read the real-time video frame collected by the endoscope camera through the endoscope camera;

[0051] Step 02: then input the video frame into the detection method;

[0052] Step 03: then judge a Flag marker, which is used to select the algorithm module to be processed, and the Flag marker is initially set to False;

[0053] Step 04: if the Flag marker is set to False, the algorithm enters the target detection module to execute step one, otherwise it enters the state recognition module to execute step two.

[0054] In this way, the misjudgment when the needle tip does not enter the round window area can be avoided, thereby improving the robustness of the system. The other components and connection relationships are the same as those in specific implementation method one or two.

[0055] Specific implementation method three: combined Figures 1 to 4 In this embodiment, step one includes the following steps:

[0056] First: when the YOLOv8 model detects the round window position, the target detection module starts to judge whether the microneedle is located in the round window area;

[0057] Secondly, if the microneedle is not located in the round window area, the video frame is prompted with the "no contact" character and the four vertex coordinates of the round window detection frame at this moment are recorded for subsequent cutting of the round window part.

[0058] If the microneedle is located in the round window area, the flag symbol Flag is set to True, representing that the algorithm will enter the state recognition module.

[0059] In this way, the pre-processing by the target detection module can effectively avoid misjudgment caused by the movement of the round window model or the camera when the microneedle has not entered the round window area. Meanwhile, cutting the round window area and then sending it to the state recognition module can help reduce the interference of a large amount of background and noise in the picture, focus on the changes in the round window membrane area, and reduce the calculation amount of the state module. The other components and connection relationships are the same as those in the first or second embodiment.

[0060] Fourth embodiment: combination Figures 1 to 4 In this embodiment, step two includes the following steps:

[0061] Step two one: input a video segment composed of 8 frame sequence images into the SlowFast model;

[0062] Step two two: the output result of the SlowFast model is the microneedle puncture state and its probability;

[0063] Step two three: input the output result of the model into the judgment algorithm for judgment:

[0064] Step two three one: judgment of the penetration state:

[0065] The judgment algorithm checks whether the classification output by the SlowFast model is penetration and whether the prediction probability is greater than or equal to 0.95;

[0066] If both conditions are met, the penetration judgment part is entered; at this time, the last 3 frames of images are continuously read and made into 3 videos of 8 frames; the SlowFast model is input for judgment: if the SlowFast model judges penetration and the probability is greater than or equal to 0.95 for λ times in the 3 videos, it is considered that the current state is the penetration state, the penetration information is prompted on the frame picture, and the frame picture is displayed, wherein λ is a settable parameter;

[0067] If the SlowFast model does not judge penetration and the probability is greater than or equal to 0.95 for λ times in the 3 videos, the frame picture is directly displayed;

[0068] Step two three two: judgment of the non-penetration state:

[0069] If the judgment algorithm does not enter the penetration judgment part, the judgment algorithm will check whether the classification output by the SlowFast model is contact and whether the prediction probability is greater than or equal to 0.9. If both conditions are met, the contact judgment part is entered.

[0070] In this way, the wrong judgment of the SlowFast model output can be filtered, and the recognition accuracy can be improved. The other components and connection relationships are the same as any one of embodiments one to three.

[0071] Embodiment five: in combination Figures 1 to 4 In this embodiment, the 8-frame sequence images in step 2-1 include the current frame and the 7 frames stored in sequence before the current frame. Each frame image is a circular window region in the original video frame, and each frame image is adjusted to a uniform size by image interpolation.

[0072] In this way, it is helpful for the state recognition module to focus on the circular window film and reduce the interference of background and noise. The other components and connection relationships are the same as any one of embodiments one to four.

[0073] Embodiment six: in combination Figures 1 to 4 In this embodiment, when the contact judgment part is entered in step 2-32, step 2-33 is entered, and the judgment process of step 2-33 is as follows:

[0074] At this time, 1 frame image after the current frame is read and made into an 8-frame video, and then input into the SlowFast model for judgment. If the SlowFast model judges that it is in contact and the probability is greater than or equal to 0.9, it is considered that the current has entered the contact state, the n_c_flag marker is set to False, the contact information is prompted on the frame image, and the frame image is displayed. The n_c_flag marker is different from the previous flag marker.

[0075] In this way, the accuracy of the model in identifying the contact state can be improved. The other components and connection relationships are the same as any one of embodiments one to five.

[0076] Embodiment seven: in combination Figures 1 to 4 In this embodiment, when the contact judgment part is not entered in step 2-32, step 2-34 is entered, and the judgment process of step 2-34 is as follows:

[0077] Step 2-34: non-contact judgment part:

[0078] If the judgment algorithm does not enter the contact judgment part, the judgment algorithm will check whether the classification output by the SlowFast model is non-contact and whether the prediction probability is greater than or equal to 0.9 and whether the n_c_flag flag is set to True. If all three conditions are met, non-contact information is prompted on the frame picture and the frame picture is displayed.

[0079] In this way, after the n_c_flag flag is set to False, the judgment algorithm no longer enters the non-contact judgment part, which can reduce the misjudgment of non-contact and contact. The other components and connection relationships are the same as any one of embodiments one to six.

[0080] Embodiment eight: in combination Figures 1 to 4 In this embodiment, the n_c_flag flag in step 233 of the embodiment is initially set to True, and there is a condition in the non-contact judgment for judging whether the n_c_flag flag is set to True.

[0081] In the initial stage, the judgment algorithm enters the non-contact judgment part through the non-contact judgment condition;

[0082] After the judgment algorithm sets the n_c_flag flag to False, the judgment algorithm will no longer enter the non-contact judgment part, i.e. non-contact information will not be prompted on the image.

[0083] In this way, after the microneedle is judged to contact the round window membrane, the system only needs to judge whether the microneedle has penetrated the round window membrane. In general, the penetration process of the microneedle is a continuous process, i.e. the penetration process of the microneedle is generally in the order of non-contact, contact, and finally penetration, and no needle withdrawal operation is performed in between. The other components and connection relationships are the same as any one of embodiments one to seven.

[0084] in combination Figures 1 to 4 Explain the embodiments of the present application:

[0085] First, a microscopic camera is used to collect a video of the microneedle penetrating the round window membrane model. From each video, a non-contact, contact, and penetration video segment is collected, and each video segment is 8 frames of images. The obtained video segment is sent to the SlowFast model for training, wherein the Slow branch inputs the last 4 frames of the video segment, and the Fast branch inputs the entire video. In addition, random non-contact video segments are collected, and images are made into target detection data sets and sent to the YOLOv8 model for training. The weights and parameters of the trained SlowFast model and YOLOv8 model are configured.

[0086] The needle tip gradually approaches the round window, and the microscopic camera reads the video frames in real time. In the initial state, the Flag flag is set to False. Therefore, the target detection module is entered first. The YOLOv8 model detects the round window position and starts to judge the position of the needle tip and the round window. Since in the annotation of the data set, the round window is basically inscribed in the annotation box of the round window. Therefore, the center point coordinates of the round window detection box are used as the center of the round window, and the half of the maximum of the length and width of the round window detection box is used as the long axis r of the round window. The distance of the four points of the needle tip detection box from the center point of the round window is calculated, and the distance value is compared with the size of the long axis r of the round window. If the distance value is greater than the long axis r of the round window, it means that the needle tip has not entered the round window area, so the non-contact character is prompted on the video frame and this video frame is displayed, and the four vertex coordinates of the round window detection box at this moment are recorded. When the needle tip further approaches the round window, if the distance value is less than the long axis r of the round window, it means that the needle tip has entered the round window area, so the system sets the flag Flag to True, which means that the algorithm enters the state recognition module.

[0087] After entering the state recognition module, the average coordinates of the four vertices of the round window detection box are used to crop the round window area in the current frame read by the microscopic camera and the stored previous 7 frames of images, the picture size is uniformly adjusted to (224, 224) through bilinear interpolation, and an 8-frame video segment is sequentially combined. The video segment is input into the SlowFast model, wherein the last 4 frames are input into the Slow branch, and the Fast branch inputs the entire video segment. Finally, the output result of the SlowFast model is input into the judgment algorithm (the λ in the algorithm is set to 1) to complete the real-time detection of the puncture state and generate a classification label.

[0088] The above examples are only used to illustrate the technical solutions of the present application, but not to limit them; although the present application has been described in detail with reference to the foregoing examples, those skilled in the art should understand that they can still modify the technical solutions recorded in the foregoing examples, or make equivalent replacements for part of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A round window membrane puncture status detection method based on the SlowFast algorithm, characterized by: It includes the following steps: Step 1: Detect the position of the round window and the microneedle using the target detection module; The target detection module uses the trained and optimized YOLOv8 model to detect whether the microneedle is within the round window area. If the microneedle is in the round window area, the next step is to detect the puncture status; If the microneedle is no longer within the round window area, continue to detect the position of the round window and the microneedle; Step 2: Identify the microneedle puncture status through the state recognition module and determine whether the microneedle puncture is successful through the judgment algorithm; The state recognition module uses the trained and optimized SlowFast algorithm model to identify the microneedle puncture state, which is divided into three types: no contact, contact, and penetration; and filters the recognition of the SlowFast network through the judgment algorithm; Step 2 includes the following steps: Step 21: Input a video clip consisting of 8 frames of sequential images into the SlowFast model; Step 22: The output of the SlowFast model is the microneedle puncture status and its probability; Step 2 and 3: Input the output of the model into the judgment algorithm for judgment: Step 231: Determination of penetration status: The judgment algorithm checks whether the classification output by the SlowFast model is penetration and whether the predicted probability is greater than or equal to 0.95; If both conditions are met, the process proceeds to the insertion judgment section. The next three frames are read continuously and compiled into three 8-frame videos. The SlowFast model is then input for judgment. If the SlowFast model determines an insertion with a probability greater than or equal to 0.95 in any of the three videos, the current state is considered an insertion state, and the insertion information is displayed on the frame image. Here, is a configurable parameter. If the SlowFast model does not identify any penetration in these three videos and the probability is greater than or equal to 0.95, the frame image is displayed directly; Step 232: Determination of non-penetration status: If the judgment algorithm does not enter the penetration judgment part, the judgment algorithm will check whether the classification output by the SlowFast model is contact and whether the predicted probability is greater than or equal to 0.

9. If both conditions are met, it will enter the contact judgment part; When the contact judgment part is entered in step 232, step 233 is entered. The judgment process of step 233 is as follows: At this time, the image frame after the current frame will be read and made into an 8-frame video. It will then be input into the SlowFast model for judgment. If the SlowFast model determines that there is contact and the probability is greater than or equal to 0.9, it is considered that the current state has entered the contact state, the n_c_flag flag is set to False, the contact information is displayed on the frame image, and the frame image is displayed; If the judgment condition does not meet step 233, then go to step 234: Steps 2, 3, and 4: Untouched judgment part: If the judgment algorithm does not enter the contact judgment part, the judgment algorithm will check whether the classification output by the SlowFast model is no contact and whether the predicted probability is greater than or equal to 0.9 and whether the n_c_flag identifier is set to True. If these three conditions are met, the no contact information will be prompted on the frame image and the frame image will be displayed.

2. The round window membrane puncture status detection method based on the SlowFast algorithm according to claim 1, characterized in that: Before step 1, the following steps are included: Step 01: First, read the real-time video frames captured by the endoscope camera through the endoscope camera; Step 02: Then input the video frame into this detection method; Step 03: A Flag flag is then checked to select the algorithm module to be processed. The Flag flag is initially set to False. Step 04: If the Flag indicator is set to False, the algorithm enters the target detection module and executes step 1. Otherwise, it enters the state recognition module and executes step 2.

3. The round window membrane puncture status detection method based on the SlowFast algorithm according to claim 2, characterized in that: Step 1 includes the following steps: first: When the YOLOv8 model detects the round window position, the object detection module begins to determine whether the microneedle is within the round window area; Secondly: If the microneedle is not within the round window area, the video frame will display the word "Not Contacted" and the video frame will be displayed. At the same time, the coordinates of the four vertices of the round window detection frame at this moment will be recorded for subsequent cropping of the round window part. If the microneedle is located in the round window area, the flag is set to True, indicating that the algorithm will enter the state recognition module.

4. The round window membrane puncture status detection method based on the SlowFast algorithm according to claim 3, characterized in that: The 8-frame sequence of images in step 21 includes the current frame and the 7 frames stored sequentially before. Each frame of the image is the circular window area in its original video frame, and each frame of the image is adjusted to a uniform size through image interpolation.

5. The round window membrane puncture status detection method based on the SlowFast algorithm according to claim 4, characterized in that: The initial state of the n_c_flag flag in step 233 is set to True, and there is a condition to determine whether the n_c_flag flag is set to True in the no-contact judgment; In the initial stage, the judgment algorithm enters the untouched judgment part through the untouched judgment condition; When the judgment algorithm sets the n_c_flag flag to False, the judgment algorithm will no longer enter the non-contact judgment part, that is, the image will no longer prompt the non-contact information.

Images