Acupuncture mirror surgery control system

Abstract

The present application discloses an acupuncture mirror surgery control system. It belongs to the field of medical device technology. An acupuncture mirror surgery control system includes: an intervention device, which is used to construct a surgical channel and perform various surgical operations; a collection device, which is used to collect video information of the operating end of the intervention device in real time; a marking device, which emits laser to the surgical area to mark the pre-marked line; an information processing device, which obtains the video information when the marking device emits a linear laser, and extracts an image information set representing the surgical path; in the technical solution provided by the present application, the surgeon can intuitively see the marked path previously marked with laser in the video information collected by the collection device, thereby improving the efficiency of the operation.

Description

Technical Field

[0001] The present application relates to the field of medical devices, and more specifically, to a needle knife mirror surgery control system. Background Art

[0002] The knee joint is one of the most important joints in the human body. During surgery, it is crucial to ensure that the knee joint's movement function is preserved to the greatest extent possible. Whether it is the precise removal of a ruptured meniscus or the removal of uric acid crystals that are tightly embedded in the knee joint tissue, the surgeon must be extremely cautious when performing the cutting operation to avoid any surgical errors.

[0003] Currently, acupotomy surgery relies on advanced AR technology operating systems. Medical staff use high-definition video transmitted in real time by micro-lenses to remotely control micro-probes to perform fine movements such as flushing, cutting, and grasping. To reduce trauma to the patient, the surgeon will first assess the patient's condition after implanting the acupotomy into the joint cavity, and then plan the approximate cutting area and path.

[0004] However, in actual operations, some doctors will try to use dyes to mark the expected cutting path on the tissue in order to improve surgical accuracy. However, this method has limitations, because the dye is easily affected by tissue fluid and flows and smudges, especially when high precision needs to be maintained for a long time. The marking effect will be greatly reduced and it is difficult to play an effective guiding role. In the absence of accurate reference, this causes the surgeon to frequently adjust the zoom ratio of the control interface during the operation: perform precise cutting in a highly magnified state, and then zoom out to observe the overall situation and confirm whether the cutting path deviates from expectations.

[0005] Therefore, the existing acupotomy operating system lacks effective cutting path marking means, which requires the surgeon to constantly verify the correctness of the surgical path, thereby reducing the overall efficiency of the operation. Summary of the invention

[0006] The content of this application is used to introduce concepts in a brief form, which will be described in detail in the detailed implementation section below. The content of this application is not intended to identify the key features or essential features of the technical solution claimed for protection, nor is it intended to limit the scope of the technical solution claimed for protection.

[0007] In order to solve the technical problem of low surgical efficiency, the present application provides a needle knife mirror surgery control system, including:

[0008] Interventional devices, used to construct surgical corridors and perform various surgical operations;

[0009] A collection device, used to collect video information of the operating end of the interventional device in real time;

[0010] a marking device that emits a laser into the surgical field to mark pre-marked lines;

[0011] An information processing device, which obtains video information when the marking device emits a linear laser, and extracts an image information set representing a surgical path;

[0012] A control device, used to display the video information collected by the acquisition device and control the intervention device to perform surgical operations;

[0013] The acupotomy surgery control system also includes:

[0014] An information fusion device, which fuses the marking information in the image information set into the video information collected in real time by the collection device to form the marking information of the preset marking line;

[0015] The laser emitted by the marking device flashes continuously at a preset frequency, and the frame rate of the video information collected by the collection device is an integer multiple of the preset frequency.

[0016] Compared with the traditional dye marking method, the technical solution proposed in this application has the significant advantages of being unaffected by liquids and having higher marking accuracy. Compared with the method of outlining marks on fixed images, this solution innovatively uses lasers to mark directly on solid tissues. This feature enables the marks to be flexibly adjusted with the movement of the operating end and accurately positioned to the desired position. Compared with the marks on two-dimensional images, this method provides more intuitive location information and reference information of the marking points, greatly improving the accuracy of the operation. In addition, the laser flashes at a preset frequency, and the frequency is exactly an integer multiple of the video frame rate. This design ensures that images containing laser marks and unmarked images appear alternately in the collected video information. This arrangement effectively avoids the interference of the laser on the actual features of the marked area and ensures the clarity and recognition of the image. Therefore, the surgeon can clearly and intuitively observe the surgical path previously marked by the laser based on the video information provided by the acquisition device, thereby significantly improving the efficiency of the operation.

[0017] In the process of setting the marking path, the doctor will constantly adjust the position of the operating end to ensure that the laser is accurately irradiated to the target position. However, if the image information at this time is directly collected, the special reflections generated by the laser on the surface of the patient's tissue may interfere with the accuracy of the subsequent matching of the virtual marking path in the video information, thereby affecting the consistency of the marking information with the surgical path planned by the doctor through the marking device. In response to this problem, the present application proposes the following technical solution: Further, the information processing device includes:

[0018] An image collection module obtains video information collected by the acquisition device when the marking device is working, and obtains a marked image group;

[0019] An image division module divides each image in the marked image group into a plurality of comparison groups, each comparison group includes at least two adjacent marked images and a blank image, and aligns the pixels of the marked image and the blank image one by one; wherein the marked image is an image captured when the marking device emits laser light, and the blank image is an image captured when the marking device does not emit laser light;

[0020] A feature extraction module extracts the laser irradiation area in the marking image to obtain the laser marking position;

[0021] The image set acquisition module acquires the laser marking position of the marked image in each comparison group, takes the laser marking position and the corresponding blank image as an information group, and takes all the information groups as an image information set.

[0022] In the technical solution provided by the present application, the video information is organized into comparison groups arranged in chronological order, each group contains two adjacent frames of images. Since these two frames of images are closely connected in time, the content of the area they present is basically consistent. In this way, after the mark position is extracted from the image containing the laser mark, the position of the laser irradiation area can be accurately found in the corresponding unmarked image. Therefore, by combining the unmarked image with the laser mark position, an image information set is constructed. This approach not only ensures the accuracy of the laser mark position, but also ensures that when the video information is subsequently matched with the virtual mark path, the laser mark area will not cause any interference due to its own characteristics.

[0023] The feature extraction module is responsible for accurately identifying the laser irradiation area in the video information. In order to ensure that the laser mark position accurately corresponds to the blank mark image, the frame rate of the video needs to be increased as much as possible, which means that the number of images that need to be processed will increase significantly. Therefore, the feature extraction module must have a high degree of accuracy and powerful data processing capabilities. In response to this demand, this application proposes the following technical solutions:

[0024] Furthermore, the feature extraction module includes:

[0025] An image preprocessing unit, used for preprocessing the input labeled image to obtain an input image X;

[0026] X∈R C×H×W , where C is the number of channels, H is the height, W is the width, and R represents a set of real numbers;

[0027] The pooling unit performs pooling operations on the input features in both the horizontal and vertical directions; ;

[0028] ;

[0029] ;

[0030] Where c represents the cth channel, h represents the height, Max(c, h) represents the maximum pooling of the cth channel and each row with height h, Avg(c, h) represents the average pooling of the cth channel and each row with height h, ⊕ represents or, X c (a, h) refers to the pixel value at the intersection of height h and width a on the cth channel, where a represents the width;

[0031] ;

[0032] ;

[0033] ;

[0034] Among them, c represents the cth channel, w represents the width, Max(c, w) represents the maximum pooling of the cth channel and each row with a width of w, Avg(c, w) represents the average pooling of the cth channel and each row with a width of w, ⊕ represents or, b is an index variable used to traverse all height positions under a given width w, X c (b, w) refers to the pixel value at the intersection of height b and width w on the cth channel;

[0035] The feature concatenation unit concatenates the horizontal pooling result and the vertical pooling result in the channel dimension to obtain the feature map Z.

[0036] Convolution unit, applies n×n convolution to the feature map to fuse the information between channels, and performs normalization to obtain the normalized feature map ;

[0037] Z`=Conv n×n (Z;W n×n ; B n×n );

[0038] Among them, Z' is the first convolution map after the convolution of feature map Z, Conv represents the convolution operation, n×n represents the size of the convolution kernel, and W n×n ∈R,W n×n Convolution n×n The first weight, B n×n Represents convolution Conv n×n The first bias term of ;

[0039] Activate the unit and normalize the feature map Use the activation function to activate and obtain the activation function graph Z``, and convolve Z`` to obtain the second convolution graph Z```;

[0040] ;

[0041] Among them, σ represents the activation function;

[0042] Z```=Conv m×m (Z``; W`; B`), Conv m×m represents a convolution operation with a convolution kernel of m×m, W` and B` represent Conv 1×1 The second weight and the second bias term, m and n are integers, m<n;

[0043] A feature map generation unit weights the second convolution map and the input image to generate a feature extraction map Y;

[0044] ;

[0045] Among them, Y c (a, b) represents the value of the cth channel of feature map Y at position (a, b), a is the width index, b is the height index, X c (a, b) represents the value of the cth channel of the input image at position (a, b), represents the horizontal weight of the cth channel, Represents the vertical weight of the cth channel;

[0046] The target extraction unit extracts the laser mark position from the feature extraction map Y. In the technical solution presented in the present application, the input image X is converted into the feature extraction map Y in advance, and then the laser position is accurately located from the feature extraction map Y. This process takes advantage of the fact that the information of the feature extraction map Y is concise but more prominent, which not only covers the global average information, but also highlights the local significant features, significantly improving the efficiency of the target extraction unit in identifying the laser mark position. In the specific implementation of this solution, the process of generating the feature extraction map involves two convolution operations. The first convolution uses a larger convolution kernel. This design expands the receptive field and is conducive to effectively capturing the feature area representing the laser mark position from the original input image X. The subsequent second convolution uses a smaller convolution kernel, which aims to more carefully depict the feature differences between the laser mark position and other areas, further enhancing the accuracy of feature extraction. In addition, the feature extraction map Y in this solution is presented through channel information c. This method focuses on reflecting special differences in color rather than texture differences, so that the laser features can be more accurately characterized.

[0047] Considering the differences in acupuncture image information of different patients, if supervised learning methods are used to locate laser markers, a huge data set will need to be prepared, which not only makes the marking work extremely cumbersome, but also increases the difficulty of model training. In response to this challenge, this application proposes the following technical solutions:

[0048] Furthermore, the target extraction unit extracts the laser marking position based on an unsupervised feature algorithm.

[0049] The technical solution provided in the present application extracts the laser marking position based on the unsupervised feature algorithm, and thus does not require additional marking work, so the difficulty of model training can be reduced. At the same time, because the feature extraction map Y finally input into the target extraction unit can better show the difference between the laser irradiation area and the remaining areas, the accuracy of the laser position extraction using the unsupervised feature algorithm can be guaranteed.

[0050] The information fusion device includes:

[0051] An image set preprocessing module collects image information sets, obtains the positions of laser marking points, and obtains the first feature point set of the laser marking path;

[0052] The video feature extraction module acquires video information in real time, extracts all feature points from the video information, and obtains a second feature point set;

[0053] The image feature matching module maps the first feature point set to the second feature point set to obtain the laser marking position in the video information, and generates marking information based on the laser marking position.

[0054] When emitting laser markings, as well as during the specific surgical process, the distance between the camera and the marking position is definitely inconsistent, which will result in the first feature point set not being accurately matched to the second feature point set. To this end, the present application provides the following technical solutions:

[0055] The first feature point set and the second feature point set are both feature points extracted based on the SIFT algorithm.

[0056] The feature points extracted by the SIFT algorithm are invariant to scale, rotation, and illumination. Therefore, the feature points collected at different angles, different illumination conditions, and different camera magnification distances are highly recognizable, allowing the first feature point set to be accurately matched to the second feature point set.

[0057] In order to ensure the continuity of surgical operations, it is relatively necessary to quickly and accurately extract feature points and complete the matching of feature points. Relatively speaking, the first feature point set collected in the image set preprocessing module has a certain waiting time because there is a process gap, and the video feature extraction module must continuously process the acquired video information to ensure the continuity of the marking information. To this end, the present application provides the following technical solutions:

[0058] The video feature extraction module includes:

[0059] An image filtering unit, used to obtain video information and remove duplicate images from the video information based on vector movement information of the operation end to obtain an information extraction image set;

[0060] A neural network extraction unit, each picture in the information extraction picture set is input into the neural network extraction unit to obtain a second feature point of each picture to obtain a second feature point set.

[0061] In the technical solution provided in the present application, based on the vector information of the operating terminal and the movement of the operating terminal, it is determined whether the image information is updated, thereby effectively reducing the number of images to be processed and improving the efficiency of image processing.

[0062] Furthermore, the neural network extraction unit includes:

[0063] Input end, used to input information extraction pictures whose features need to be extracted;

[0064] A feature point detector, connected to the input end, extracts a second feature point from the information extraction picture based on a convolutional neural network;

[0065] A descriptor decoder, connected to the input end, extracts a descriptor of a second feature point from the information extraction image based on a convolutional neural network;

[0066] A trainer, connected to the feature point detector and the descriptor decoder by signals, respectively, for training the second feature point and the descriptor of the second feature point generated by the feature point detector and the descriptor decoder;

[0067] Among them, the label data of the feature point detector and the descriptor decoder are feature points extracted based on the SIFT algorithm.

[0068] In the technical solution provided in the present application, a pre-trained neural network extraction unit is used to extract the second feature point. Therefore, compared with directly calculating the feature points through a large amount of calculations, in the technical solution provided in the present application, the feature point extraction efficiency is higher and the corresponding feature points can be generated more accurately.

[0069] Further, the control device comprises:

[0070] A display, used to display the video information collected by the operation end;

[0071] A control module, used to control the interventional device to perform various surgical operations;

[0072] The marker adjuster is used to adjust the color reduction ratio of the marker information in the video information.

[0073] In the solution provided in the present application, the marking adjustment period can adjust the color fading ratio of the marking information in the video information, so that the doctor can reasonably adjust the color of the marking information as needed to avoid the marking information affecting the surgeon's judgment of the operation.

[0074] Furthermore, the operating end of the interventional device includes:

[0075] A micro-lens, used to obtain video information during surgical operations;

[0076] Vector displacement sensor, used to obtain the movement rate of the operating end.

[0077] The beneficial effects of this application are:

[0078] The acupuncture mirror operating system provided in the present application can use the operating end to mark the cutting site. Compared with marking on the picture transmitted by the operating system, the marking position can be more accurate and contain more three-dimensional information. When the position of the operating end changes during the operation, the corresponding position can also be accurately matched. BRIEF DESCRIPTION OF THE DRAWINGS

[0079] Figure 1 This is a schematic diagram of the structure of the acupuncture knife mirror surgery control system.

[0080] Figure 2 This is the feature extraction graph after processing.

[0081] Figure 3 This is a schematic diagram of preset marking matching.

[0082] Figure 4 A schematic diagram of the neural network structure.

[0083] Figure 5 Schematic diagram of the distribution of the first feature points matched in the second feature point set. DETAILED DESCRIPTION

[0084] Embodiments of the present application will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present application are shown in the accompanying drawings, it should be understood that the present application can be implemented in various forms, and should not be construed as being limited to the embodiments set forth herein. On the contrary, these embodiments are provided to provide a more thorough and complete understanding of the present application. It should be understood that the drawings and embodiments of the present application are only for exemplary purposes and are not intended to limit the scope of protection of the present application.

[0085] It should also be noted that, for ease of description, only the parts related to the invention are shown in the drawings. In the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other.

[0086] The present application will be described in detail below with reference to the accompanying drawings and in combination with embodiments.

[0087] Example 1: Reference Figure 1 The acupotomoscope is a commonly used surgical tool for treating joint diseases and is widely used in gout treatment, joint replacement and joint repair surgery. In gout treatment, it is often used to remove tophi attached to the joints; in joint replacement and repair surgery, it is responsible for a number of delicate operations such as cutting, grasping, and flushing the joints. The end of the acupotomoscope is equipped with a camera and an operating instrument. The camera is responsible for collecting real-time video images in the joint cavity, providing the surgeon with a clear field of view so that the acupotomoscope can be accurately controlled to complete the operation. With the continuous advancement of surgical robot technology, the operation of the acupotomoscope has been fully systematized. Now, the surgeon only needs to place the acupotomoscope in the surgical area, and then he can easily control the acupotomoscope to perform surgery on the console that integrates display and control, which greatly improves the accuracy and efficiency of the operation.

[0088] Based on this technology, the present application provides a needle knife mirror surgery control system, including an intervention device, a collection device, a marking device, an information processing device, a control device and an information fusion device.

[0089] The interventional device is used to construct a surgical channel and perform various surgical operations. The end of the interventional device is the control end, which is at least matched with a micro lens and a control component. The micro lens is used to collect image information, and the control component is used to perform surgical operations such as cutting and flushing. The interventional device also includes a vector displacement sensor to obtain the movement rate of the operation end, that is, the vector displacement sensor can be used to determine in real time whether the operation end is moving.

[0090] The acquisition device is used to collect the video information of the operation end of the intervention device in real time. That is, the acquisition device will send the image information transmitted by the micro lens to the control device.

[0091] The control device is used to display the video information collected by the acquisition device and control the intervention device to perform surgical operations. Specifically, the control device includes a display and a control module. The display is used to display the video information collected by the operation end; the control module is used to control the intervention device to perform various surgical operations.

[0092] Specifically: The marking device emits laser to the surgical area to mark the pre-marked lines. The preset marking lines are the marking lines that the surgeon needs to draw in the surgical area according to the actual situation during the operation to assist the surgical process. The marking device is essentially a laser set at the operating end of the access device. After the laser emits laser light and irradiates the surface of the surgical area, a light spot is formed. The light spot is used as the tip of the virtual marker, and the path of movement on the surgical area can be used as a marking line.

[0093] The information processing device obtains the video information when the marking device emits the linear laser, and extracts the image information set representing the surgical path from it. That is, when the marking device emits the laser, the acquisition device will collect the video information containing the laser. The position of the laser irradiation in this video information is the path that the doctor needs to mark.

[0094] The information fusion device fuses the marking information in the image information set into the video information collected in real time by the collection device to form the marking information of the preset marking line.

[0095] Specifically, when the doctor needs to mark the preset marking line, the marking device will be started first, and the device will then emit a laser to illuminate the surgical area. The doctor moves the laser spot by manipulating the operating end, and the path of the spot moving on the surgical area becomes the preset marking line. After completing the preset marking line, the marking device will receive and store the relevant image information set. In subsequent surgical operations, the video information collected by the acquisition device will be fused with the marking information in these image information sets, so that the preset marking line is displayed in the video to provide surgical guidance for the doctor.

[0096] There are two key technical points involved in this process: one is how to accurately extract the position of the light spot formed by the laser emitted by the marking device in the image; the other is how to accurately match the area illuminated by the light spot with the video information.

[0097] Regarding the first technical point, the present application proposes the following solution: the laser emitted by the marking device flashes at a preset frequency, and the frame rate of the acquisition device is set to an integer multiple of the preset frequency, specifically 2 times. In this way, images with light spots and images without light spots will appear alternately in the video sequence.

[0098] The information processing device consists of four main modules: image collection module, image segmentation module, feature extraction module and image set acquisition module. The image collection module is responsible for acquiring the video information recorded by the acquisition device when the marking device is working to form a group of marked images. The image segmentation module groups these marked images, each group contains at least one marked image (collected when the laser is emitted) and one blank image (collected when the laser is not emitted), and ensures that the pixels of the two correspond one to one. If the number of marked images is less than the number of blank images, the system will delete the extra blank images and only keep the one that is closest to the marked image.

[0099] The task of the feature extraction module is to extract the laser irradiated area from the marking image, that is, to determine the laser marking position. This process is essentially to extract features from the marking image. The specific method is as follows:

[0100] The feature extraction module includes:

[0101] An image preprocessing unit, used for preprocessing the input labeled image to obtain an input image X;

[0102] X∈R C×H×W , where C is the number of channels, H is the height, W is the width, and R represents a set of real numbers;

[0103] The pooling unit performs pooling operations on the input features in both the horizontal and vertical directions; ;

[0104] ;

[0105] ;

[0106] Where c represents the cth channel, h represents the height, Max(c, h) represents the maximum pooling of the cth channel and each row with a height of h, Avg(c, h) represents the average pooling of the cth channel and each row with a height of h, and ⊕ represents or; X c (a, h) refers to the pixel value at the intersection of height h and width a on the cth channel;

[0107] ;

[0108] ;

[0109] ;

[0110] Among them, c represents the cth channel, w represents the width, Max(c, w) represents the maximum pooling of the cth channel and each row with a width of w, Avg(c, w) represents the average pooling of the cth channel and each row with a width of w, ⊕ represents or, b is an index variable used to traverse all height positions under a given width w, X c (b, w) refers to the pixel value at the intersection of height b and width w on the cth channel;

[0111] In the pooling unit, the selection basis of average pooling and maximum pooling is that when the local pooling mean is greater than the overall pooling mean, average pooling is selected to reduce the influence of local regional features on overall regional features; when the local pooling mean is less than the overall pooling mean, maximum pooling is selected to highlight local features;

[0112] The feature concatenation unit concatenates the horizontal pooling result and the vertical pooling result in the channel dimension to obtain the feature map Z.

[0113] Convolution unit, applies n×n convolution to the feature map to fuse the information between channels, and performs normalization to obtain the normalized feature map ;

[0114] Z`=Conv n×n (Z;W n×n ; B n×n );

[0115] Among them, Z' is the first convolution map after the convolution of feature map Z, Conv represents the convolution operation, n×n represents the size of the convolution kernel, and W n×n ∈R,W n×n Convolution n×n The first weight, B n×n Represents convolution Conv n×n The first bias term of ;

[0116] Activate the unit and normalize the feature map Use the activation function to activate and obtain the activation function graph Z``, and convolve Z`` to obtain the second convolution graph Z```;

[0117] ;

[0118] Among them, σ represents the activation function;

[0119] Z```=Conv m×m (Z``; W`; B`), Conv m×m represents a convolution operation with a convolution kernel of m×m, W` and B` represent Conv 1×1 The second weight and the second bias term, m and n are integers, m<n;

[0120] A feature map generation unit weights the second convolution map and the input image to generate a feature extraction map Y;

[0121] ;

[0122] Among them, Y c (a, b) represents the value of the cth channel of feature map Y at position (a, b), a is the width index, b is the height index, X c (a, b) represents the value of the cth channel of the input image at position (a, b), represents the horizontal weight of the cth channel, Represents the vertical weight of the cth channel;

[0123] refer to Figure 2 ,exist Figure 2 This is the feature extraction image Y obtained after processing. Because the spectrum of laser reflection will be plotted as much as possible when selecting the channel, the area with laser reflection has high brightness, while the rest of the area has basically the same background color.

[0124] The target extraction unit extracts the laser marking position from the feature extraction map Y.

[0125] Because the laser mark position in feature extraction figure Y is clearer, the laser mark position can be extracted using an unsupervised feature algorithm.

[0126] The specific plan is as follows:

[0127] Step 1: Set up an encoder and a decoder. The encoder maps the feature extraction map Y to a low-dimensional space (called latent space or encoding), while the decoder tries to reconstruct the original image from this low-dimensional space.

[0128] Step 2: Use mean squared error (MSE) as the loss function.

[0129] Step 3: Use the unlabeled image dataset to train the autoencoder. At this stage, the autoencoder will learn the general features of the image.

[0130] Of course, in order to increase accuracy, some supervision information can be introduced during the training process. For example, a classifier can be trained using pairs of images containing laser areas and images not containing laser areas, and the output of this classifier can be used as an additional loss term of the autoencoder. In this way, after the training is completed, the laser marking position can be accurately extracted.

[0131] The image set acquisition module acquires the laser marking position of the marked image in each comparison group, takes the laser marking position and the corresponding blank image as an information group, and takes all the information groups as an image information set.

[0132] In a comparison group, the marked image and the blank image are separated by a short time, so the corresponding areas of the two are basically the same. In this scheme, they are considered to be approximately the same. Of course, the registration algorithm can also be used for preliminary registration and calibration. Therefore, the laser marking position marked on the marked image is actually the laser marking position in the blank image.

[0133] The image information set that can be obtained must include all the locations where the laser is irradiated.

[0134] refer to Figure 3 , the information fusion device essentially needs to Figure 3 The virtual preset marking line is matched to the real-time collected video information. The information fusion device includes: an image set preprocessing module, a video feature extraction module, and an image feature matching module. Among them, the image set preprocessing module collects the image information set, obtains the laser marking point position to obtain the first feature point set of the laser marking path;

[0135] The video feature extraction module acquires video information in real time, extracts all feature points from the video information, and obtains a second feature point set;

[0136] To this end, it is necessary to quickly extract the first feature point set and the second feature point set from the image information machine and the real-time acquired video information. The first feature point set only needs to extract the feature points of the laser marking point position, while the second feature point set needs to extract the feature points of each picture in the video information. The feature point extraction method in the image set preprocessing module and the video feature extraction module is the same.

[0137] Of course, for the first feature point set, in practice, repeated first feature points can also be deleted according to the similarity algorithm to reduce the number of feature points in the first feature point set. It should be noted that the second feature point set in this solution is the set of all feature points of a picture in the video information. In fact, the video information collected by the acquisition device is also transmitted from pictures one by one, and each picture only needs to be used as the second feature point set.

[0138] Furthermore, both the first feature point set and the second feature set are feature points extracted based on the SIFT algorithm. Specifically:

[0139] The video feature extraction module includes:

[0140] An image filtering unit, used to obtain video information and remove duplicate images from the video information based on vector movement information of the operation end to obtain an information extraction image set;

[0141] A neural network extraction unit, each picture in the information extraction picture set is input into the neural network extraction unit to obtain a second feature point of each picture to obtain a second feature point set.

[0142] The neural network extraction unit includes:

[0143] Input end, used to input information extraction pictures whose features need to be extracted;

[0144] A feature point detector, connected to the input end, extracts a second feature point from the information extraction picture based on a convolutional neural network;

[0145] A descriptor decoder, connected to the input end, extracts a descriptor of a second feature point from the information extraction image based on a convolutional neural network;

[0146] A trainer, connected to the feature point detector and the descriptor decoder by signals, respectively, for training the second feature point and the descriptor of the second feature point generated by the feature point detector and the descriptor decoder;

[0147] Among them, the label data of the feature point detector and the descriptor decoder are feature points extracted based on the SIFT algorithm.

[0148] That is, in the solution provided by this application, the feature point detector and descriptor decoder are convolutional neural networks, which can automatically learn the characteristics of the feature points extracted by the SIFT algorithm and then automatically extract the corresponding feature points. In order to further increase the accuracy of the feature point detector and descriptor decoder, the loss function Loss is configured in the trainer. dete ;

[0149] ;

[0150] ;

[0151] ;

[0152] Among them, Loss dcc Represents the detector loss of the original image, Loss warp represents the detector loss after the image undergoes a homography transformation, Represents the value of the i-th pixel in the original information extraction picture, Represents the value of the i-th pixel in the label image, and Respectively represent the value of the i-th pixel in the original information extraction image and the label image after homography transformation; N represents the total number of pixels, i represents the index of the pixel, ψ represents the Sigmoid function; BCE represents the binary cross entropy loss function. The loss function designed in this application is a combined loss function, which can increase the accuracy of feature detection, find boundary artifacts, and avoid the adverse effects of edge-variable feature knots on feature point extraction. The loss function of the descriptor decoder is a regularized loss function. The descriptor is mainly generated based on feature points, and the accuracy of descriptor generation is affected by feature point extraction.

[0153] The image feature matching module maps the first feature point set to the second feature point set to obtain the laser marking position in the video information, and generates marking information based on the laser marking position.

[0154] Specifically, the image feature matching module matches the second feature point set with the first feature point set. If a feature point in the second feature point set matches a feature point in the first feature point set, it means that the feature point in the second feature point set belongs to the laser marking position. In this way, a marking information is generated when the feature point is displayed.

[0155] There are two ways to match the first feature point with the second feature point. The first way is to directly use a similarity calculation formula to determine whether the feature points match the second feature point according to a similarity threshold calculated from the feature points.

[0156] The second method is to use neural network model matching. In essence, the matching of the first feature point set and the second feature point set is to find the mutually aligned areas between the two pictures. In order to improve the matching efficiency, this solution constructs a graph neural network and uses the self-attention mechanism and cross-attention mechanism to construct the neural network model. Figure 4 , Figure 4 is the structure of the neural network model, Figure 4 The first descriptor in is the descriptor corresponding to the first feature point, which is a binary code generated by the descriptor decoder. The second descriptor is the descriptor corresponding to the second feature point, which is a binary code generated by the descriptor decoder. The position encoder is used to determine the position of the descriptor.

[0157] It will not be introduced here. However, in order to increase the accuracy of the model, this application introduces the following distribution loss function F when training the neural network model:

[0158] F=G op +M ut Among them, G op is the original loss function of the neural network model, which can be a cross entropy loss function. The setting of this loss function is not restricted here. ut is a probability distribution function of the matching position of the first feature point in the second feature point set, used for modifying the distribution of the first feature point in the second feature point set based on the original function;

[0159] M ut =j1(v)+j2(v); where v is the index of the first feature point matched to the second feature point set, j1(v) is the aggregation of all first feature points on all current second feature point sets, and j2(v) is the continuity of all first feature points on all current second feature point sets. Continuity can be directly calculated using regression analysis. Aggregation is calculated using the aggregation index. The above two parameters are common technical means in this field. Figure 5 As shown, the second feature point set is essentially a series of pictures. After the first feature point is matched to the second feature point set, the preset marking line can be restored. Therefore, the denser the matching point positions are and the higher the continuity between them, the higher the matching accuracy is.

[0160] Embodiment 2: Embodiment 2 differs from Embodiment 1 in that the control device further comprises a mark adjuster. The mark adjuster is used to adjust the color reduction ratio of the mark information in the video information.

[0161] In Example 1, the preset marking lines can be synthesized onto the real-time received video information and displayed, and the marking adjuster adjusts the color fading ratio of the marking information. The higher the color fading ratio, the more transparent the marking information is. In this way, the surgeon can flexibly adjust the obstruction of the marking information to the actual surgical field of view.

[0162] The above description is only some preferred embodiments of the present application and an explanation of the technical principles used. Those skilled in the art should understand that the scope of the invention involved in the embodiments of the present application is not limited to the technical solutions formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above invention concept. For example, the above features are replaced with (but not limited to) technical features with similar functions disclosed in the embodiments of the present application to form a technical solution.

Claims

1. A needle knife mirror surgery control system, characterized by: include: Interventional devices, used to construct surgical corridors and perform various surgical operations; A collection device, used to collect video information of the operating end of the interventional device in real time; a marking device that emits a laser into the surgical field to mark pre-marked lines; An information processing device, which obtains video information when the marking device emits a linear laser, and extracts an image information set representing a surgical path; A control device, used to display the video information collected by the acquisition device and control the intervention device to perform surgical operations; The acupotomy surgery control system also includes: An information fusion device, which fuses the marking information in the image information set into the video information collected in real time by the collection device to form the marking information of the preset marking line; The laser emitted by the marking device flashes continuously at a preset frequency, and the frame rate of the video information collected by the collection device is an integer multiple of the preset frequency.

2. The acupotomy mirror surgery control system according to claim 1, characterized in that: The information processing device includes: An image collection module obtains video information collected by the acquisition device when the marking device is working, and obtains a marked image group; An image division module divides each image in the marked image group into a plurality of comparison groups, each comparison group includes at least two adjacent marked images and a blank image, and aligns the pixels of the marked image and the blank image one by one; wherein the marked image is an image captured when the marking device emits laser, and the blank image is an image captured when the marking device does not emit laser; A feature extraction module extracts the laser irradiation area in the marking image to obtain the laser marking position; The image set acquisition module acquires the laser marking position of the marked image in each comparison group, takes the laser marking position and the corresponding blank image as an information group, and takes all the information groups as an image information set.

3. The acupotomy mirror surgery control system according to claim 2, characterized in that: An image preprocessing unit, used for preprocessing the input labeled image to obtain an input image X; X∈R C×H×W , where C is the number of channels, H is the height, W is the width, and R represents a set of real numbers; The pooling unit performs pooling operations on the input features in both the horizontal and vertical directions; ； Where c represents the cth channel, h represents the height, Max(c, h) represents the maximum pooling of the cth channel and each row with a height of h, Avg(c, h) represents the average pooling of the cth channel and each row with a height of h, and ⊕ represents or; ； Among them, c represents the c-th channel, w represents the width, Max(c, w) represents the maximum pooling of the c-th channel and each row with width w, Avg(c, w) represents the average pooling of the c-th channel and each row with width w, and ⊕ represents or; The feature concatenation unit concatenates the horizontal pooling result and the vertical pooling result in the channel dimension to obtain the feature map Z. Convolution unit, applies n×n convolution to the feature map to fuse the information between channels, and performs normalization to obtain the normalized feature map ; Z`=Conv n×n (Z;W n×n ;B n×n ); Among them, Z' is the first convolution map after the convolution of feature map Z, Conv represents the convolution operation, n×n represents the size of the convolution kernel, and W n×n ∈R,W n×n Convolution n×n The first weight, B n×n Represents convolution Conv n×n The first bias term of ; Activate the unit and normalize the feature map Use the activation function to activate and obtain the activation function graph Z``, and convolve Z`` to obtain the second convolution graph Z```; ； Among them, σ represents the activation function; Z```=Conv m×m (Z``; W`; B`), Conv m×m represents a convolution operation with a convolution kernel of m×m, W` and B` represent Conv 1×1 The second weight and the second bias term, m and n are integers, m<n; A feature map generation unit weights the second convolution map and the input image to generate a feature extraction map Y; ； Among them, Y c (a, b) represents the value of the cth channel of feature map Y at position (a, b), a is the width index, b is the height index, X c (a, b) represents the value of the cth channel of the input image at position (a, b), represents the horizontal weight of the cth channel, Represents the vertical weight of the cth channel; The target extraction unit extracts the laser marking position from the feature extraction map Y.

4. The acupotomy mirror surgery control system according to claim 3, characterized in that: The target extraction unit extracts the laser marking position based on an unsupervised feature algorithm.

5. The acupotomy mirror surgery control system according to claim 1, characterized in that: The information fusion device includes: An image set preprocessing module collects image information sets, obtains the positions of laser marking points, and obtains the first feature point set of the laser marking path; The video feature extraction module acquires video information in real time, extracts all feature points from the video information, and obtains a second feature point set; The image feature matching module maps the first feature point set to the second feature point set to obtain the laser marking position in the video information, and generates marking information based on the laser marking position.

6. The acupotomology surgical control system according to claim 5, characterized in that: The first feature point set and the second feature point set are both feature points extracted based on the SIFT algorithm.

7. The acupotomology surgical control system according to claim 5, characterized in that: The video feature extraction module includes: An image filtering unit, used to obtain video information and remove duplicate images from the video information based on vector movement information of the operation end to obtain an information extraction image set; A neural network extraction unit, each picture in the information extraction picture set is input into the neural network extraction unit to obtain a second feature point of each picture to obtain a second feature point set.

8. The acupotomology surgical control system according to claim 7, characterized in that: The neural network extraction unit includes: Input end, used to input information extraction pictures whose features need to be extracted; A feature point detector, connected to the input end, extracts a second feature point from the information extraction picture based on a convolutional neural network; A descriptor decoder, connected to the input end, extracts a descriptor of a second feature point from the information extraction image based on a convolutional neural network; A trainer, connected to the feature point detector and the descriptor decoder by signals, respectively, for training the second feature point and the descriptor of the second feature point generated by the feature point detector and the descriptor decoder; Among them, the label data of the feature point detector and the descriptor decoder are feature points extracted based on the SIFT algorithm.

9. The acupotomology surgical control system according to claim 1, characterized in that: The control device includes: A display, used to display the video information collected by the operation end; A control module, used to control the interventional device to perform various surgical operations; The marker adjuster is used to adjust the color reduction ratio of the marker information in the video information.

10. The acupotomy mirror surgery control system according to claim 1, characterized in that: The operating end of the interventional device includes: A micro-lens, used to obtain video information during surgical operations; Vector displacement sensor, used to obtain the movement rate of the operating end.

Images